Parallel processing-decoding, post-filtering and RDOQ of image regions using neural networks

Abstract

The present invention relates to image encoding and decoding of image regions in units of blocks. Specifically, a plurality of components of an input tensor, including a first component and a second component, are processed in a spatial dimension in a plurality of pipelines. The processing of the first component includes dividing the first component into a first plurality of tiles in the spatial dimension. Similarly, the processing of the second component includes dividing the second component into a second plurality of tiles in the spatial dimension. The respective first plurality of blocks and the second plurality of blocks are then processed, respectively. And at least two corresponding juxtaposed blocks with different sizes are arranged in the plurality of first blocks and the plurality of second blocks. In the case of compression, the processing of the first component and / or the second component includes image encoding, rate distortion optimized quantization, and image filtering. In the case of decompression, the processing includes image decoding and image filtering.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of image or video encoding and decoding, and specifically relate to encoding and decoding of bit streams based on neural networks. Background Art

[0002] Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray Discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] Even short videos require a large amount of video data to depict, which can create difficulties when the data is to be streamed or otherwise transmitted over a communication network with limited bandwidth capacity. Therefore, video data is typically compressed before being transmitted over modern telecommunication networks. Due to limited memory resources, the size of the video needs to be considered when storing the video in a storage device. Video compression devices typically encode the video data using software and / or hardware on the source side and then transmit or store the video data, thereby reducing the amount of data required to represent the digital video image. A video decompression device that decodes the video data then receives the compressed data on the destination side. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality.

[0004] Neural network (NN) and deep-learning (DL) technologies using artificial neural networks have been used for some time, and these technologies are also used in the field of encoding and decoding technology for videos, images (such as still images), etc.

[0005] It is desirable to further improve the efficiency of such image decoding (video image decoding or still image decoding) based on a trained network (e.g., a neural network (NN)) that takes into account limitations of available memory and / or processing speed of a decoder and / or encoder. Summary of the invention

[0006] Some embodiments of the present invention provide methods and apparatus for encoding and / or decoding images in an efficient manner, thereby reducing the memory footprint of a processing unit and the required operating frequency. Specifically, the present invention is able to strike a balance between memory resources and computational complexity within a NN-based video / image encoding-decoding framework applicable to both moving and still images.

[0007] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are obvious from the dependent claims, the description and the drawings.

[0008] According to one aspect of the present invention, a method for processing an input tensor representing image data is provided, the method comprising the following steps: processing multiple components of the input tensor in a spatial dimension, the multiple components including a first component and a second component, the processing comprising: processing the first component, comprising: dividing the first component into a first plurality of blocks in the spatial dimension and processing the blocks in the first plurality of blocks respectively; processing the second component, comprising: dividing the second component into a second plurality of blocks in the spatial dimension and processing the blocks in the second plurality of blocks respectively; wherein at least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks are of different sizes. Therefore, the input tensor representing the image data can be efficiently processed on a component basis by using blocks in a sample-aligned manner within multiple pipelines. Therefore, memory requirements are reduced while processing performance (e.g., compression and decompression) is improved without increasing computational complexity.

[0009] In some exemplary implementations, at least two of the first plurality of blocks are processed independently or in parallel; and / or at least two of the second plurality of blocks are processed independently or in parallel. Therefore, the components of the input tensor can be processed quickly, thereby improving processing efficiency.

[0010] In another implementation, the first component represents the brightness component of the image data; the second component represents the chrominance component of the image data. Therefore, the brightness and chrominance components can be processed by multiple pipelines within the same processing framework.

[0011] In one example, the blocks in the first plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap; and / or the blocks in the second plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap. Therefore, the quality of the reconstructed image can be improved, especially along the boundaries of the blocks. Therefore, image artifacts can be reduced.

[0012] According to one implementation, the division of the first component includes determining the size of the blocks in the first plurality of blocks according to a first predefined condition; and / or the division of the second component includes determining the size of the blocks in the second plurality of blocks according to a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data. Therefore, the block size can be adjusted and optimized based on the available decoder resources and / or motion, thereby achieving content-based block size. In another example, determining the size of the blocks in the second plurality of blocks includes scaling the blocks in the first plurality of blocks. Therefore, the block size in the second plurality of blocks can be quickly determined, thereby improving the processing efficiency of the blocks.

[0013] In an exemplary implementation, the determined indication of the size of a block in the first plurality of blocks and / or the second plurality of blocks is encoded into a bitstream. Thus, the indication of the block size is efficiently included in the bitstream, requiring less processing.

[0014] In another implementation, all of the first plurality of blocks are of the same size and / or all of the second plurality of blocks are of the same size. Thus, blocks can be processed efficiently without requiring additional processing to handle different block sizes, thereby speeding up block processing.

[0015] In a second example, the indication further includes a location of a tile in the first plurality of tiles and / or the second plurality of tiles.

[0016] According to one implementation, the first component is a luminance component, the code stream includes the indication of the size of the block in the first plurality of blocks; the second component is a chrominance component, the code stream includes the indication of a scaling factor, wherein the scaling factor is related to the size of the block in the first plurality of blocks and the size of the block in the second plurality of blocks. Therefore, by the fast operation of scaling the block size of the luminance component, the block size of the chrominance component can be quickly obtained. In addition, the overhead for indicating the block size of chrominance can be reduced by using the scaling factor as an indication.

[0017] In an exemplary implementation, the processing of the input tensor includes processing as part of image or moving image compression. For example, the processing of the first component and / or the second component includes one of the following steps: image encoding of a neural network; rate distortion optimization quantization (RDOQ); image filtering. Thus, the compression process can be performed in a flexible manner including various processes (encoding, RDOQ and filtering).

[0018] Another exemplary implementation includes generating the code stream by including the output of the processing of the first component and the second component into the code stream. Thus, the processing output can be included in the code stream quickly, requiring less processing.

[0019] In an exemplary implementation, the processing of the input tensor includes processing as part of image or moving image decompression. For example, the processing of the first component and / or the second component includes one of the following steps: image decoding of a neural network; image filtering. Therefore, the decompression process can be performed in a flexible manner including various processes (encoding and filtering). For example, the processing of the second component includes decoding the chrominance component of the image according to the representation of the luminance component of the image. Therefore, the luminance component can be used as auxiliary information for decoding the chrominance component. This can improve the quality of the decoded chrominance. In another example, the processing of the first component and / or the second component includes image post-filtering; for at least two of the first plurality of blocks, one or more parameters of the post-filtering are different and are extracted from the code stream; for at least two of the second plurality of blocks, one or more parameters of the post-filtering are different and are extracted from the code stream. Therefore, the filter parameters can be efficiently indicated by the code stream. In addition, the post-filtering can be performed using filter parameters suitable for the block size, thereby improving the quality of the reconstructed image data.

[0020] In an exemplary implementation, the input tensor is an image or an image sequence comprising one or more of the plurality of components, wherein at least one component is a color component.

[0021] According to one aspect of the present invention, there is provided a computer program stored in a non-transitory medium, the computer program comprising codes, which, when executed on one or more processors, implement the steps of any of the above aspects of the present invention.

[0022] According to one aspect of the present invention, there is provided a device for processing an input tensor representing image data, the device comprising a processing circuit, the processing circuit being used to: process multiple components of the input tensor in a spatial dimension, the multiple components comprising a first component and a second component, the processing comprising: processing the first component, comprising: dividing the first component into a first plurality of blocks in the spatial dimension and processing the blocks in the first plurality of blocks separately; processing the second component, comprising: dividing the second component into a second plurality of blocks in the spatial dimension and processing the blocks in the second plurality of blocks separately; wherein at least two corresponding juxtaposed blocks of the first plurality of blocks and the second plurality of blocks are of different sizes.

[0023] According to one aspect of the present invention, there is provided a device for processing an input tensor representing image data, the device comprising: one or more processors; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein, when the program is executed by the one or more processors, the program causes the processing device to perform a method according to any of the above-mentioned aspects of the present invention.

[0024] The present invention is applicable to both end-to-end AI codecs and hybrid AI codecs. For example, in a hybrid AI codec, filtering operations (filtering of reconstructed images) can be performed by a neural network (NN). The present invention is applied to such NN-based processing modules. Generally, if at least part of the processing includes a NN, and if the NN includes convolution or transposed convolution operations, the present invention can be applied to the entire or part of the video compression and decompression process. For example, the present invention is applicable to separate processing tasks performed as part of the processing performed by an encoder and / or decoder, including in-loop filtering and / or post-filtering and / or pre-filtering.

[0025] It should be noted that the present invention is not limited to a specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0026] The present invention can be implemented by hardware (HW) and / or software (SW). In addition, the implementation based on HW can be combined with the implementation based on SW.

[0027] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0028] One or more embodiments will be described in detail in the drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0030] Figure 1A FIG. 4 is a block diagram of an example of a video decoding system for implementing an embodiment of the present invention.

[0031] Figure 1B FIG. 4 is a block diagram of another example of a video decoding system for implementing an embodiment of the present invention.

[0032] Figure 2A block diagram of an example of an encoding device or a decoding device.

[0033] Figure 3 A block diagram of another example of an encoding device or a decoding device.

[0034] Figure 4 is a block diagram of an exemplary hybrid encoder for implementing an embodiment of the present invention.

[0035] Figure 5 is a block diagram of an exemplary hybrid decoder for implementing an embodiment of the present invention.

[0036] Fig. 6A Schematic diagram of the variational autoencoder architecture including a hyper-prior model.

[0037] Figure 6B To include similar Fig. 6A Schematic diagram of another example of a variational autoencoder architecture for a hyper-prior model.

[0038] Figure 7 A block diagram of the parts of an exemplary autoencoder.

[0039] Figure 8 Compression of input data by an encoder and decompression of data by a decoder are shown, where the compressed data is represented by a latent space.

[0040] Fig. 9 Block diagram of the encoder and decoder that conform to the VAE framework.

[0041] Fig. 9A Based on Fig. 9 Block diagram of an encoder with corresponding components.

[0042] Fig. 9B Based on Fig. 9 Block diagram of a decoder with corresponding components.

[0043] Fig.10 The total receptive field that includes all input samples required to generate an output sample is shown.

[0044] Fig.11 A subset of the total receptive field is shown, in which case the output sample is generated from a smaller number of samples (the subset) and is used as the number of samples in the total receptive field. Padding samples may be required.

[0045] Fig.12 Shown is the use of two convolutional layers to downsample an input sample to one output sample.

[0046] Fig.13 This example illustrates computing the total receptive field for a set of 2×2 output samples using two convolutional layers with 3×3 kernel size.

[0047] Fig.14 An example of parallel processing is shown, where the image is divided into two blocks, and the decoding and sample reconstruction of the corresponding code streams are performed independently.

[0048] Fig.15 An example of parallel processing is shown, where a coding tree block (CTB) is divided into slices (rows), where the codestream of each slice is (almost) independently decoded, but sample reconstruction of the slices is not performed independently.

[0049] Fig.16 A block diagram of an encoder and a decoder with corresponding modules, including a NN-based subnetwork for processing a first plurality of blocks and a second plurality of blocks according to a first embodiment.

[0050] Fig.17A An example of partitioning the first tensor and / or the second tensor into overlapping regions Li (ie, first and second partitions), subsequent cropping of samples in the overlapping regions, and concatenation of the cropped regions is shown. Each Li includes a total receptive field.

[0051] Fig. 17B Shows something like Fig.17A Another example of dividing the first tensor and / or the second tensor into the overlapping area Li (ie, the first block and the second block), except that the cropping is cancelled.

[0052] Fig.18 An example of partitioning the first tensor and / or the second tensor into overlapping regions Li (i.e., first partitions and second partitions), subsequent cropping of samples in the overlapping regions, and concatenation of the cropped regions is shown. Each Li includes a subset of the total receptive field.

[0053] Fig.19 An example of partitioning the first tensor and / or the second tensor into non-overlapping regions Li (i.e., a first partition and a second partition), subsequent cropping of samples, and concatenation of the cropped regions is shown. Each Li includes a subset of the total receptive field.

[0054] Fig. 20 Various parameters that can be included in the codestream (and parsed from the codestream) are shown, such as the size of the area Li, Ri and the overlap area of ​​the first block and / or the second block, etc. Any of the various parameters can be included in the codestream (and parsed from the codestream) as an indication.

[0055] Fig.21 A flow chart of a method for encoding an input tensor representing image data according to a first embodiment is shown.

[0056] Fig. 22A flow chart of a method for decoding a tensor representing image data according to a first embodiment is shown.

[0057] Fig.23 A block diagram of a processing device for encoding an input tensor representing image data is shown, the processing device comprising a processing circuit. The processing circuit may include a module that performs the processing of the encoding method according to the first embodiment.

[0058] Fig.24 A block diagram of a processing device for decoding a tensor representing image data is shown, the processing device comprising a processing circuit. The processing circuit may include a module that performs the processing of the decoding method according to the first embodiment.

[0059] Fig.25 is a block diagram of an encoder-decoder with corresponding modules according to a second embodiment, wherein the luminance component and the chrominance component of a tensor are processed separately in multiple pipelines, and each pipeline processes multiple blocks of the corresponding components respectively.

[0060] Fig.26 A flow chart of a method for processing an input tensor representing image data according to a second embodiment is shown.

[0061] Fig. 27 A block diagram of an apparatus for processing an input tensor representing image data according to a second embodiment is shown, the apparatus comprising corresponding processing circuits and modules. DETAILED DESCRIPTION

[0062] Some embodiments of the present invention are described below with reference to the accompanying drawings. Figure 3 The present invention relates to a video encoding system and method that can be used with more specific embodiments of the present invention described in other figures. Figure 3 The described embodiments may be used with encoding / decoding techniques described further below that utilize neural networks to encode a code stream and / or decode a code stream.

[0063] In the following description, reference is made to the accompanying drawings which form a part of the present invention, which illustrate specific aspects of the present invention or specific aspects in which embodiments of the present invention may be used by way of illustration. It should be understood that the embodiments may be used in other aspects and include structural or logical changes not shown in the accompanying drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present invention is defined by the appended claims.

[0064] For example, it should be understood that what is disclosed with reference to a described method may also be true for a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units for performing the described one or more method steps (e.g., one unit that performs one or more steps; or multiple units, each unit performing one or more of the multiple steps), even if the one or more units are not explicitly described or shown in the drawings. In addition, for example, if a specific device is described in terms of one or more units (e.g., functional units), the corresponding method may include a step for performing the function of the one or more units (e.g., one step that performs the function of the one or more units, or multiple steps, each step performing the function of one or more of the multiple units), even if the one or more steps are not explicitly described or shown in the drawings. In addition, it should be understood that, unless otherwise specified, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0065] Video decoding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video decoding, the terms "frame" and "picture / image" can be used as synonyms. Video decoding (or commonly referred to as decoding) includes two parts: video encoding and video decoding. Video encoding is performed at the source end and generally includes processing (for example, by compression) the original video image to reduce the amount of data required to represent the video image (to obtain more efficient storage and / or transmission). Video decoding is performed at the destination end and generally includes inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or the respective video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding).

[0066] In the case of lossless video decoding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video decoding, further compression is performed by quantization and other means to reduce the amount of data required to represent the video image, and the video image cannot be fully reconstructed at the decoding end, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0067] Several video coding standards belong to the group of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding for quantization in the transform domain). Each image in a video sequence is usually segmented into a set of non-overlapping blocks, and decoding is usually performed at the block level. In other words, at the encoding end, the video is usually processed (i.e., encoded) at the block (video block) level, for example, by using spatial (intra-frame image) prediction and / or temporal (inter-frame image) prediction to generate a prediction block, subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data to be sent (compressed), while at the decoding end, the inverse process relative to the encoder is used for the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder is processed according to the decoder processing loop so that both generate the same prediction (e.g., intra-frame and inter-frame prediction) and / or reconstruction for processing, i.e., decoding the subsequent block. Recently, part or the whole encoding / decoding chain has been implemented using neural networks, or in general, any machine learning or deep learning framework.

[0068] In the following embodiment of the video decoding system 10 , the video encoder 20 and the video decoder 30 are described with reference to FIG. 1 .

[0069] Figure 1A FIG. 1 is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can use the techniques of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform techniques according to various examples described in the present application.

[0070] like Figure 1A As shown, the decoding system 10 includes a source device 12 for providing the encoded image data 21 to a destination device 14 or the like for decoding the encoded image data 13 .

[0071] The source device 12 includes an encoder 20 and may additionally (i.e., optionally) include an image source 16, a preprocessor (preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22. Some embodiments of the present invention (e.g., involving initial rescaling or rescaling between two process layers) may be implemented by the encoder 20. Some embodiments (e.g., related to initial rescaling) may be implemented by the image preprocessor 18.

[0072] The image source 16 may include or may be: any type of image capture device, such as a camera for capturing real-world images; and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images; or any type of device for obtaining and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device that stores any of the above images.

[0073] In order to distinguish from the pre-processor 18 and the processing performed by the pre-processing unit 18 , the image or image data 17 may also be referred to as raw image or raw image data 17 .

[0074] The preprocessor 18 is used to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (for example, usually from RGB to YCbCr or from RGB to YUV), color correction or denoising, etc. It will be understood that the preprocessing unit 18 may be an optional component. In the following, color space components (such as R, G, B in RGB space and Y, U, V in YUV space) are also referred to as color channels. In addition, in YUV or YCbCr color space, Y represents brightness, and U, V, Cb, Cr represent chrominance channels (components).

[0075] The video encoder 20 is used to receive the pre-processed image data 19 and provide encoded image data 21 (for example, according to Figure 4 The encoder 20 may be implemented by the processing circuit 46 to implement the combination of Figure 4 The various modules discussed herein may include encoder 20 and / or any other encoder system or subsystem described herein.

[0076] The communication interface 22 in the source device 12 may be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0077] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ), and may additionally (ie, optionally) include a communication interface or communication unit 28 , a post-processor 32 (or a post-processing unit 32 ), and a display device 34 .

[0078] The communication interface 28 of the destination device 14 is used to receive the encoded image data 21 (or any other processed version), for example directly from the source device 12, or from any other source (such as a storage device, such as an encoded image data storage device), and provide the encoded image data 21 to the decoder 30.

[0079] The communication interface 22 and the communication interface 28 can be used to send or receive the encoded image data 21 or the encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between the source device 12 and the destination device 14, or via any type of network (e.g., a wired network, a wireless network, any combination of wired and wireless networks, any type of private and public networks, or any type of combination of private and public networks).

[0080] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or use any type of transmission coding or processing to process the encoded image data for transmission over a communication link or a communication network.

[0081] For example, the communication interface 28 (corresponding to the communication interface 22 ) may be used to receive the transmitted data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation process to obtain the encoded image data 21 .

[0082] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1A The unidirectional communication interface represented by the arrow of the communication channel 13 pointing from the source device 12 to the destination device 14, or configured as a bidirectional communication interface, and can be used to send and receive messages, establish connections, confirm and exchange any other information related to the communication link and / or data transmission (such as the transmission of encoded image data), etc.

[0083] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 (eg, as described below according to Figure 3 and Figure 5 The decoder 30 may be implemented by the processing circuit 46 to achieve the combination of Figure 5 The various modules discussed in connection with decoder 30 and / or any other decoder systems or subsystems described herein.

[0084] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, cropping or resampling, or any other processing to prepare the decoded image data 31 for display by a display device 34 or the like.

[0085] Some embodiments of the present invention may be implemented by the decoder 30 or the post-processor 32 .

[0086] The display device 34 of the destination device 14 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0087] although Figure 1A The source device 12 and the destination device 14 are shown as separate devices, but embodiments of the device may also include both devices or the functions of both devices, that is, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In such embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented by the same hardware and / or software, by separate hardware and / or software, or any combination thereof.

[0088] From the above description, it is obvious to those skilled in the art that Figure 1A The presence and the (precise) functional division of the different units or functions of the source device 12 and / or the destination device 14 shown in FIG. 1 may vary depending on the actual device and application.

[0089] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both the encoder 20 and the decoder 30 may be configured as follows: Figure 1BThe processing circuits shown are implemented, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by processing circuits 46 to implement the various modules and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented by processing circuits 46 to implement the various modules and / or any other decoder systems or subsystems described herein. The processing circuits may be used to perform the various operations described herein. Figure 3 As shown, if these techniques are partially implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium, and these instructions can be executed in hardware by one or more processors to perform the techniques of the present invention. Either of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1B shown.

[0090] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be used for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0091] In some cases, Figure 1A The video decoding system 10 shown is exemplary only, and the techniques of the present application may be applied to video decoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, the data can be retrieved from a local memory, streamed over a network, etc. The video encoding device can encode the data and store the data to a memory, and / or the video decoding device can retrieve and decode the data from the memory. In some examples, the encoding and decoding are performed by devices that do not communicate with each other but simply encode the data to the memory and / or retrieve and decode the data from the memory.

[0092] For ease of description, some embodiments are described herein with reference to high-efficiency video coding (HEVC) or versatile video coding (VVC) (next generation video coding standard) reference software developed by the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG) video coding joint collaboration team (JCT-VC) (next generation video coding standard). It should be understood by those skilled in the art that the embodiments of the present invention are not limited to HEVC or VVC.

[0093] Figure 2 Schematic diagram of a video decoding device 200 provided in an embodiment of the present invention. The video decoding device 200 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video decoding device 200 may be a decoder (e.g. Figure 1A A video decoder 30) or an encoder (such as Figure 1A video encoder 20).

[0094] The video decoding device 200 includes: an inlet port 210 (or input port 210) and a receiving unit (Rx) 220 for receiving data; a processor, a logic unit or a central processing unit (CPU) 230 for processing the data; a transmitting unit (Tx) 240 and an outlet port 250 (or output port 250) for transmitting the data; and a memory 260 for storing the data. The video decoding device 200 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the inlet port 210, the receiving unit 220, the transmitting unit 240 and the outlet port 250, which are used as an outlet or an inlet of an optical signal or an electrical signal.

[0095] The processor 230 is implemented by hardware and software. The processor 230 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 230 communicates with the input port 210, the receiving unit 220, the sending unit 240, the output port 250, and the memory 260. The processor 230 includes a decoding module 270. The decoding module 270 implements the disclosed embodiments described above. For example, the decoding module 270 performs, processes, prepares, or provides various decoding operations. Therefore, the decoding module 270 is included, which provides substantial improvements to the functions of the video decoding device 200 and affects the transformation of the video decoding device 200 to different states. Alternatively, the decoding module 270 is implemented with instructions stored in the memory 260 and executed by the processor 230.

[0096] Memory 260 may include one or more disks, tape drives, and solid-state hard disks, and may be used as an overflow data storage device to store programs when such programs are selected for execution, as well as to store instructions and data read during program execution. For example, memory 260 may be volatile and / or non-volatile, and may be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0097] Figure 3 A simplified block diagram of an apparatus 300 is provided for an exemplary embodiment, which may be used as either or both of the source device 12 and the destination device 14 in FIG. 1 .

[0098] The processor 302 in the apparatus 300 may be a central processing unit. Alternatively, the processor 302 may be any other type of device or devices capable of manipulating or processing information now existing or later developed. Although the disclosed implementations may be implemented by a single processor (e.g., the processor 302), more than one processor may be used to increase speed and efficiency.

[0099] The memory 304 in the apparatus 300 may be a read only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 304. The memory 304 may include code and data 306 accessed by the processor 302 using a bus 312. The memory 304 may also include an operating system 308 and an application 310, which includes at least one program that causes the processor 302 to perform the methods described herein. For example, the application 310 may include applications 1 to N, which also include a video decoding application that performs the methods described herein.

[0100] The apparatus 300 may also include one or more output devices, such as a display 318. In one example, the display 318 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 318 may be coupled to the processor 302 via the bus 312.

[0101] Although described here as a single bus, the bus 312 of the device 300 may be composed of multiple buses. In addition, the auxiliary storage device 314 may be directly coupled to other components of the device 300, or may be accessed through a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, the device 300 can be implemented in a variety of configurations.

[0102] Figure 4 FIG. 2 is a schematic block diagram of an exemplary video encoder 20 for implementing the techniques of the present application. Figure 4 In the example of , the video encoder 20 includes an input terminal 401 (or an input interface 401), a residual calculation unit 404, a transform processing unit 406, a quantization unit 408, an inverse quantization unit 410, an inverse transform processing unit 412, a reconstruction unit 414, a loop filter 420, a decoded picture buffer (DPB) 430, a mode selection unit 460, an entropy coding unit 470, and an output terminal 472 (or an output interface 472). The mode selection unit 460 may include an inter-frame prediction unit 444, an intra-frame prediction unit 454, and a segmentation unit 462. The inter-frame prediction unit 444 may include a motion estimation unit and a motion compensation unit (not shown in the figure). Figure 4 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

[0103] The encoder 20 may be configured to receive an image 17 (or image data 17) via an input 401 or the like, for example, an image in an image sequence forming a video or a video sequence. The received image or image data may also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, it is referred to as image 17 in the following description. The image block 17 may also be referred to as a current image or an image to be decoded (specifically, in video decoding, the current image is distinguished from other images, such as previously encoded and / or decoded images in the same video sequence (i.e., a video sequence also including the current image)).

[0104] A (digital) image is or can be considered as a two-dimensional array or matrix of samples with intensity values. The samples in the array can also be called pixels (pixel / pel) (short for picture elements). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the image. Three color components are usually used to represent colors, that is, an image can be represented by three sample arrays or an image can include three sample arrays. In the RGB format or color space, the image includes corresponding red, green and blue sample arrays. However, in video decoding, each pixel is usually represented in a brightness and chrominance format or color space, for example, the color space is YCbCr, which includes a brightness component represented by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness component Y represents the brightness or grayscale intensity (for example in a grayscale image), while the two chrominance components Cb and Cr represent the chrominance or color information components. Therefore, an image in the YCbCr format includes a brightness sample array (Y) of brightness sample values ​​and two chrominance sample arrays (Cb and Cr) of chrominance values. An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luma sample array. Thus, the image may be, for example, a luma sample array in black and white format or a luma sample array and two corresponding chroma sample arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0105] An embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2The image segmentation unit may be used to segment the image 17 into a plurality of (typically non-overlapping) image blocks 403 (not shown). These blocks may also be referred to as root blocks or macroblocks (H.264 / AVC standard) or as coding tree blocks (CTB) or coding tree units (CTU) (H.265 / HEVC and VVC standards). The image segmentation unit may be used to use the same block size for all images in a video sequence and a corresponding grid defining the block size, or to vary the block size between images or subsets or groups of images, and to segment each image into corresponding blocks. The abbreviation AVC stands for Advanced Video Coding.

[0106] In other embodiments, the video encoder may be configured to directly receive a block 403 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 403 may also be referred to as a current image block or an image block to be encoded.

[0107] Although the size of the image block 403 is smaller than that of the image 17, like the image 17, the image block 403 is also or can also be considered as a two-dimensional array or matrix of samples with intensity values ​​(sample values). In other words, the block 403 may include, for example, one sample array (e.g., a brightness array in the case of the black and white image 17), three sample arrays (e.g., one brightness array and two chrominance arrays in the case of the color image 17), or any other number and / or type of arrays, depending on the color format applied. The number of samples of the block 403 in the horizontal and vertical directions (or axes) defines the size of the block 403. Therefore, an image block may be an M×N (M columns×N rows) sample array, or an M×N transform coefficient array, etc.

[0108] exist Figure 4 In the illustrated embodiment of the video encoder 20 , the video encoder 20 may be configured to encode the image 17 block by block, for example, performing encoding and prediction on each block 403 .

[0109] Figure 4 The illustrated embodiment of the video encoder 20 may also partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (typically non-overlapping), and each slice may include one or more blocks (e.g., CTUs).

[0110] Figure 4The illustrated embodiment of the video encoder 20 may also segment and / or encode an image using tile groups (also referred to as video tile groups) and / or blocks (also referred to as video blocks), wherein one or more tile groups (usually non-overlapping) may be used to segment or encode an image, each tile group may include one or more blocks (e.g., CTU) or one or more blocks, etc., wherein each block may be a rectangular shape, etc., and may include one or more blocks (e.g., CTU), such as complete or partial blocks.

[0111] Figure 5 5 is an example of a video decoder 30 for implementing the technology of the present application. The video decoder 30 is used to receive the encoded image data 21 (e.g., the encoded code stream 21) encoded by the encoder 20, for example, to obtain a decoded image 531. The encoded image data or code stream includes information for decoding the encoded image data, such as data representing image blocks of encoded video slices (and / or partition groups or partitions) and related syntax elements.

[0112] The entropy decoding unit 504 is used to parse the bitstream 21 (or generally referred to as the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded decoding parameters ( Figure 3 The entropy decoding unit 504 may be used to perform a decoding algorithm or scheme corresponding to the encoding scheme described by the entropy encoding unit 470 of the encoder 20. The entropy decoding unit 504 may also be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or as an alternative to slices and corresponding syntax elements, block groups and / or blocks and corresponding syntax elements may also be received and / or used.

[0113] The reconstruction unit 514 (e.g., an adder or summer 514) can be used to add the reconstructed residual block 513 to the prediction block 565 to obtain the reconstructed block 515 in the sample domain, for example, by adding the sample values ​​of the reconstructed residual block 513 to the sample values ​​of the prediction block 565.

[0114] Figure 5The embodiment of the video decoder 30 shown in FIG. 1 may be used to segment and / or decode a picture using slices (also referred to as video slices). A picture may be segmented into or decoded using one or more (usually non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs).

[0115] In an embodiment, Figure 5 The video decoder 30 shown may be used to segment and / or decode an image using block groups (also referred to as video block groups) and / or blocks (also referred to as video blocks). An image may be segmented into one or more block groups (usually non-overlapping) or decoded using one or more block groups (usually non-overlapping); each block group may include one or more blocks (e.g., CTU) or one or more blocks, etc.; each block may be rectangular, etc., and may include one or more complete or partial blocks (e.g., CTU), etc.

[0116] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 520. For example, for certain blocks or frames, a non-transform based decoder 30 may directly quantize the residual signal without the inverse transform processing unit 512. In another implementation, the video decoder 30 may combine the inverse quantization unit 510 and the inverse transform processing unit 512 into a single unit.

[0117] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current link can be further processed and then output to the next link. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further clipped or shifted.

[0118] In the following, more specific, non-limiting and exemplary embodiments of the present invention are described. Some explanations and definitions are first made to help understand the present invention:

[0119] Image size

[0120] Image size refers to the width w or height h of the image, or a pair of width and height. The width and height of an image are usually measured in the number of brightness samples.

[0121] Downsampling

[0122] In the process of downsampling, the sampling rate (sampling interval) of the discrete input signal is reduced. For example, if the input signal is an image with sizes h and w, and the downsampled output size is h2 and w2, then at least one of the following is true:

[0123] ●h2 <h

[0124] w2 <w

[0125] In one exemplary implementation, downsampling may be achieved by retaining only every mth sample while discarding the rest of the input signal (eg, image).

[0126] Upsampling:

[0127] In the process of upsampling, the sampling rate (sampling interval) of the discrete input signal is increased. For example, if the input image has sizes h and w, and the downsampled output has sizes h2 and w2, then at least one of the following is true:

[0128] ●h2>h

[0129] w2>w

[0130] Resampling:

[0131] Both downsampling and upsampling processes are examples of resampling. In the process of resampling, the sampling rate (sampling interval) of the input signal is changed. Resampling is a method of adjusting (or rescaling) the input signal.

[0132] During the upsampling process or the downsampling process, filtering can be applied to improve the accuracy of the resampled signal and reduce aliasing effects. Interpolation filtering usually involves a weighted combination of sample values ​​at sample positions surrounding the resampled position. Interpolation filtering can be implemented as:

[0133]

[0134] f() refers to the resampled signal, (x r ,y r ) is the resampling coordinate (the coordinate of the resampled sample), C(k) is the interpolation filter coefficient, and s(x,y) is the input signal. The coordinates x and y are the coordinates of the input image sample. r ,y r ) r In other words, the new sample f(x r ,y r ) is obtained as a weighted sum of the input image samples s(x,y). The weighting is performed by the coefficient C(k), where k represents the position (index) of the filter coefficient in the filter mask. For example, in the case of a 1D filter, k takes values ​​from 1 to the order of the filter. In the case of a 2D filter that can be applied to a 2D image, k can be an index representing one of all possible (non-zero) filter coefficients. By convention, the index is associated with a specific position of the coefficient in the filter mask (filter kernel).

[0135] Crop:

[0136] Cropping (cutting) the outer edges of a digital image. Cropping can be used to make an image smaller (in terms of number of samples) and / or change the aspect ratio (length-to-width ratio) of an image. Cropping can be understood as removing samples from a signal, typically at the signal boundaries.

[0137] filling:

[0138] Padding refers to increasing the size of the input (i.e., input image) by generating new samples (e.g., at image boundaries) using predefined sample values ​​or using (e.g., duplicating or merging) sample values ​​at existing locations in the input image. The generated samples are approximations of actual sample values ​​that do not exist.

[0139] Resizing:

[0140] Resizing is a general term for changing the size of an input image. Resizing can be performed using one of the padding methods or the cropping method. Alternatively, resizing can be performed by resampling.

[0141] Integer division:

[0142] Integer division is division that discards the fractional part (the remainder).

[0143] convolution:

[0144] The one-dimensional convolution of the input signal f() and the filter g() can be defined as:

[0145]

[0146] Here, m is the index within the input signal and the filter. n represents the position (shift) of the filter relative to the input signal. Both n and m are integers. S-convolution in 2D can work similarly, as is well known in the art. For the sake of generality, m can be considered to have values ​​between negative infinity and positive infinity, as shown in the formula above. However, in practice, the filter f[] may have a finite length, in which case the filter coefficient f[m] is equal to zero for m that exceeds the filter size.

[0147] Artificial Neural Networks

[0148] An artificial neural network (ANN) or connectionist system is a computing system vaguely inspired by the biological neural networks that make up animal brains. These systems "learn" to perform tasks by considering examples, and are usually not programmed with task-specific rules. For example, in image recognition, these systems might learn to recognize images containing cats by analyzing example images that have been manually labeled as "cat" or "no cat" and using the results to recognize cats in other images. These systems do not have any prior knowledge about cats, such as whether these cats have fur, tails, whiskers, and cat-like faces. Instead, these systems automatically generate recognition features from the examples they are processed.

[0149] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely mimic neurons in biological brains. Each connection, like a synapse in a biological brain, can send signals to other neurons. An artificial neuron receives a signal, processes it, and can send signals to neurons connected to it. In the ANN implementation, the "signal" at the connection is a real number, and the output of each neuron can be calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges usually have weights that are adjusted as they learn. Weights can increase or decrease the strength of the signal on the connection. Neurons can have a threshold so that they only send a signal if the aggregate signal exceeds that threshold. Typically, neurons are grouped into layers. Different layers can perform different transformations on their inputs. Signals travel from the first layer (input layer) to the last layer (output layer), possibly traversing the layers many times during this time.

[0150] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, leading to a deviation from biology. ANNs have been used for a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even activities traditionally considered exclusive to humans, such as painting.

[0151] Downsampling layer:

[0152] A downsampling layer is a layer of a neural network that reduces the input by at least one dimension. Typically, the input may have 3 or more dimensions, where the dimensions may include the number of channels, width, and height. Downsampling layers typically involve reducing the width dimension and / or the height dimension. Downsampling layers can be implemented using operations such as convolution (possibly with strides), averaging, max pooling, etc.

[0153] Upsampling layer:

[0154] An upsampling layer is a layer of a neural network that is modified to increase the input by at least one dimension. Typically, the input may have 3 or more dimensions, where the dimensions may include the number of channels, width, and height. An upsampling layer typically involves increasing the width dimension and / or the height dimension. It can be implemented by operations such as deconvolution, replication, etc.

[0155] Feature map:

[0156] A feature map is generated by applying a filter (kernel) or feature detector to an input image or feature map output from a previous layer. Feature map visualization provides insight into the internal representation of a particular input to each convolutional layer in the model. Typically, a feature map is the output of a neural network layer. A feature map typically includes one or more feature elements.

[0157] Convolutional Neural Networks

[0158] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. Convolutional networks are simple neural networks that use convolution instead of general matrix multiplication in at least one of their multiple layers. Convolutional neural networks consist of an input layer, an output layer, and multiple hidden layers. The input layer is the layer that is provided with input for processing.

[0159] For example, Fig. 6A The neural network is CNN. The hidden layer of CNN is usually composed of a series of convolutional layers (for example, Fig. 6A The result of the layer is one or more feature maps, sometimes also called channels. Some or all layers may involve subsampling. As a result, the feature maps may become smaller. The activation function in a CNN can be a rectified linear unit (RELU) layer or a GDN layer as described above. The activation function can be followed by additional convolutions, such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because the inputs and outputs of these convolutions are masked by the activation function and the final convolution. By convention, layers are colloquially called convolutions. Mathematically speaking, a convolution is technically a sliding dot product or cross-correlation. This has important implications for the indexing in the matrix because the convolution affects the way the weights are determined at a specific index point.

[0160] When programming a CNN to process images, the input is a tensor (e.g., an input tensor such as Fig. 6AThen, after passing through the convolutional layer, the image is abstracted into a feature map (feature tensor) with a shape of (number of images) × (feature map width) × (feature map height) × (feature map channels). Fig. 6A In the example, the feature map is y. A convolutional layer in a neural network should have the following properties: A kernel defined by width and height (hyperparameters). Number of input channels and output channels (hyperparameters). The depth of the convolution filter (input channels) should be equal to the number of input feature map channels (depth). For example, Fig. 6A The con N×5×5 in the above formula refers to a kernel of size 5×5 and N channels, where N is an integer equal to or greater than 1.

[0161] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, these nodes suffer from high dimensionality and do not scale well with high-resolution images. An image of 1000×1000 pixels with RGB color channels has 3 million weights, which is too large to be efficiently processed at scale with full connectivity. Furthermore, this network architecture does not take into account the spatial structure of the data, thereby treating input pixels that are far apart in the same way as close pixels. This ignores the locality of reference in the image data both computationally and semantically. Therefore, full connectivity of neurons is wasteful for purposes such as image recognition dominated by spatially local input patterns.

[0162] Convolutional Neural Networks are biologically inspired variants of Multilayer Perceptrons specifically designed to mimic the behavior of the visual cortex. CNN models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of a CNN. The parameters of this layer consist of a set of learnable filters (the kernels described above) that have a small receptive field but extend to the entire depth of the input volume. During the forward pass, each filter is convolved over the width and height of the input volume, computing the dot product between the filter entries and the input, and generating a 2-dimensional activation map for that filter. Thus, the network learns filters that activate when certain specific types of features are detected at a certain spatial location in the input.

[0163] The activation maps of all filters stacked along the depth dimension form the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. It is called an activation map in some papers because it is a map of activations corresponding to different parts of the image, and it is also called a feature map because it is also a map of where a certain feature is found in the image. A high activation means that a certain feature is found.

[0164] Another important concept of CNN is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions that can implement pooling, of which max pooling is the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region. Intuitively, the exact location of a feature is not as important as the rough location of that feature relative to other features. This is the idea of ​​using pooling in convolutional neural networks. Pooling layers are used to gradually reduce the spatial size of the representation to reduce the number of parameters, memory footprint, and computation in the network, thereby also controlling overfitting. In CNN architectures, it is common to insert pooling layers regularly between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0165] Pooling layers operate independently on each depth slice of the input and spatially adjust these depth slices. The most common form is a pooling layer with filters of size 2×2, which applies a downsampling of stride 2 to each depth slice of the input, applying stride 2 along both width and height, discarding 75% of the activations. In this case, each max operation is over 4 numbers. The depth dimension remains unchanged.

[0166] In addition to maximum pooling, the pooling unit can also use other functions, such as average pooling or Pooling. Average pooling was historically often used, but has recently lost its edge over max pooling, which performs better in practice. There has been a recent trend to use smaller filters or to drop pooling layers entirely, due to the large reduction in representation size. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks, used for object detection based on the Fast R-CNN architecture.

[0167] The above ReLU is short for Rectified Linear Unit, and ReLU applies a non-saturated activation function. ReLU sets negative values ​​to zero, effectively removing negative values ​​from the activation map. ReLU increases the nonlinear nature of the decision function and the overall network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, such as saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because ReLU trains neural networks several times faster without significantly affecting generalization accuracy.

[0168] After several convolutional and max pooling layers, high-level reasoning in a neural network is done with fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as in a regular (non-convolutional) artificial neural network. Therefore, their activations can be calculated as an affine transformation, a matrix multiplication followed by a bias shift (a vector addition of a learned or fixed bias term).

[0169] The "loss layer" specifies how the training penalizes the deviation between the predicted label (output) and the true label, and is usually the last layer of the neural network. A variety of loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0170] Subnet

[0171] A neural network may include multiple subnetworks. A subnetwork consists of 1 or more layers. Different subnetworks have different input / output sizes, resulting in different memory requirements and / or computational complexity.

[0172] assembly line

[0173] A series of subnetworks that process specific components of an image. For example, an image component can be any of the R, G, or B components. A component can also be luma Y or one of the chroma components U or V. An example could be a system with 2 pipelines, where the first pipeline processes only the luma component and the second pipeline processes the chroma components. One pipeline processes only one component, while the second component (e.g. the luma component) can be used as auxiliary information to help the processing (chroma components, etc.). For example, a pipeline with chroma components as output can have potential representations of luma and chroma components as input (conditional decoding of chroma components).

[0174] Rate-distortion optimized quantization (RDOQ)

[0175] RDOQ is an encoder-only technique, meaning that it applies to processing performed by the encoder, not by the decoder. Before being written to the bitstream, the parameters are quantized (descaled, rounded, etc.) to a specified standard fixed precision. Among multiple rounding methods, the minimum RD cost variant can usually be selected, for example, for transform coefficient decoding in HEVC or VVC.

[0176] Conditional color separation (CCS)

[0177] In the NN architecture for image / video decoding / processing, CCS refers to the independent decoding / processing of the primary color component (e.g., luma component), while the secondary color component (e.g., chroma UV) is conditionally decoded / processed using the primary component as auxiliary input.

[0178] Autoencoders and Unsupervised Learning

[0179] An autoencoder is an artificial neural network that is used to learn efficient data encoding in an unsupervised manner. Figure 7 As shown, it can be regarded as Fig. 6A or Figure 6B A simplified representation of a CNN-based VAE (Variational Autoencoder) structure. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore the signal "noise", usually for dimensionality reduction. In addition to the dimensionality reduction side, a reconstruction side is also learned, where the autoencoder tries to generate a representation as close to its original input as possible from the simplified encoding, hence the name.

[0180] In the simplest case, given a single hidden layer, the encoder stage of the autoencoder takes the input x and maps it to h

[0181] h=σ(Wx+b).

[0182] This image h is often called a code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as a sigmoid function or a rectified linear unit. W is the weight matrix and b is the bias vector. The weights and biases are usually initialized randomly and then iteratively updated via backpropagation during training. Afterwards, the decoder stage of the autoencoder maps h to a reconstruction x′ of the same shape as x:

[0183] x′=σ′(W′h′+b′)

[0184] Here, σ′, W′, and b′ of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0185] Variational autoencoder models make strong assumptions about the distribution of latent variables. These variational autoencoder models use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm, called the Stochastic Gradient Variational Bayes (SGVB) estimator. It is assumed that the data is represented by a directed graphical model p θ (x|h) is generated, and the encoder is learning the posterior distribution p θ The approximate value q of (h|x) φ (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generation model), respectively. The probability distribution of the latent vector of a VAE is usually closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0186]

[0187] Here, D KL represents the KL divergence. The priors on the latent variables are usually set to centered isotropic multivariate Gaussian Typically, the shapes of the variational and likelihood distributions are chosen to be factorized Gaussians:

[0188]

[0189] Among them, ρ(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0190] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have led to researchers’ interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using networks based on variational autoencoders (VAEs) has been proposed. Data compression is therefore considered a fundamental and well-studied problem in engineering, usually with the goal of designing a code with minimum entropy for a given set of discrete data. This scheme relies heavily on the knowledge of the probabilistic structure of the data, and the problem is therefore closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors. In this case, i.e., the lossy compression problem, two competing costs must be balanced: the entropy of the discrete representation (rate) and the error (distortion) introduced by the quantization. Different compression applications (e.g., data storage or transmission over finite-capacity channels) require different trade-offs between rate and distortion. Joint optimization of rate and distortion is difficult. The general problem of optimal quantization in high-dimensional space is intractable without further constraints.

[0191] Therefore, most existing image compression methods perform the following operations: linearly transform the data vector into a suitable continuous-valued representation, independently quantize its elements, and then decode the resulting discrete representation using lossless entropy decoding. Due to the central role of the transform, this scheme is called transform coding. For example, JPEG uses discrete cosine transform on pixel blocks, and JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Usually, the three components of the transform decoding method (transform, quantizer and entropy decoding) are optimized separately (usually through manual parameter tuning). Modern video compression standards (such as HEVC, VVC and EVC) also use transform representations to decode the residual signal after prediction. Several transforms such as discrete cosine transform (DCT), discrete sine transform (DST) and low-frequency non-separable manually optimized transform (LFNST) are used for this purpose.

[0192] Potential Space:

[0193] The latent space refers to the feature map generated at the bottleneck layer of the NN. Figure 7 and Figure 8This is illustrated in the example shown. In the case of a NN topology, the purpose of the network is to reduce the dimensionality of the input signal (as in an autoencoder topology), and the bottleneck layer usually refers to the layer where the dimensionality of the input signal is reduced to the minimum. The purpose of reducing the dimensionality is usually to achieve a more compact representation of the input. Therefore, the bottleneck layer is a layer suitable for compression, so in the case of video coding applications, the bitstream is generated based on the bottleneck layer.

[0194] An autoencoder topology usually consists of an encoder and a decoder connected to each other at a bottleneck layer. The purpose of the encoder is to reduce the dimensionality of the input to make it more compact (or more intuitive). The purpose of the decoder is to reverse the operation of the encoder to reconstruct the input as best as possible based on the bottleneck layer.

[0195] Variational auto-encoder (VAE) framework

[0196] The VAE framework can be considered as a nonlinear transformation coding model. The transformation process can be divided into four parts: Fig. 9 An example is given in Figure 2 and the VAE framework is shown.

[0197] The transformation process can be divided into four parts. Fig. 9 The VAE framework is illustrated, including the encoder branch and the decoder branch. Fig. 9 In the example, encoder 901 maps the input image x to a latent representation (represented by y) by function y=f(x). Hereinafter, this latent representation may also be referred to as a portion or point of the "latent space". Function f() is a transformation function that transforms the input signal x into a more compressible representation y. Quantizer 902 Transform the latent representation y into a quantized latent representation with (discrete) values Where Q represents the quantizer function. The quantizer function may be RDOQ. The entropy model or super encoder / decoder (also called super prior) 903 estimates the quantized latent representation to obtain the minimum rate achievable by lossless entropy source coding.

[0198] Latent space can be understood as a representation of compressed data (e.g., image data) where similar data points are closer in the latent space. Latent space is very useful for learning data features and finding simpler representations of data for analysis.

[0199] Quantified latent representation T and hyper-prior 903 and side information Use arithmetic coding (AE) included in the code stream 2 (binarized), such as Fig. 9 shown.

[0200] Furthermore, a decoder 904 is provided which transforms the quantized latent representation into a reconstructed image Signal is an estimate of the input image x. We hope that x is as close as possible In other words, the reconstruction quality is as high as possible. However, The higher the similarity with x, the more side information is needed to send. This side information can include Fig. 9 The bitstream 1 and bitstream 2 shown are generated by the encoder and sent to the decoder. Generally, the larger the amount of side information, the higher the reconstruction quality. However, a large amount of side information indicates a low compression ratio. Therefore, Fig. 9 One goal of the system described in is to balance the quality of reconstruction and the amount of side information transmitted in the bitstream.

[0201] exist Fig. 9 , component AE 605 is an arithmetic coding module that converts the quantized latent representation Samples and side information The samples are converted into binary representation code stream 1. For example, Samples and The samples may include integers or floating point numbers. One purpose of the arithmetic coding module is to convert the sample values ​​into a binary digital string (through a binarization process) (which is then included in the codestream, which may include other parts corresponding to the encoded image or other side information).

[0202] Arithmetic decoding (AD) 906 is the process of restoring the binarization process, wherein the binarized numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 906 .

[0203] It should be noted that the present invention is not limited to this specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0204] exist Fig. 9 In this context, a network is a logical division between parts of the overall network. For example, in Fig. 9 In FIG. 1 , processing units (modules) 901, 902, 904, 905, and 906 are referred to as an autoencoder / autodecoder or simply an "encoder / decoder" network. In other words, the network can be defined in terms of processing units (modules) that are connected to enable functionality. Fig. 9The modules 901, 902, 904, 905 and 906 connected in the network perform the functions of encoding and decoding processing of the input image x (e.g., input tensor). Therefore, Fig. 9 The "encoder / decoder" network (first network) in the example is responsible for encoding (generating) and decoding (parsing) the first code stream "code stream 1". Accordingly, the connected processing units (modules) 903, 908, 909, 910 and 907 form another network (second network), which can be called a "super encoder / decoder" network. The second network is responsible for encoding (generating) and decoding (parsing) the second code stream "code stream 2". Fig. 9 In the example, the first code stream includes the encoded image data The second code stream includes side information Therefore, the two networks have different purposes. Any processing unit (module) of the first network and the second network can be a network itself, called a subnet, meaning that a particular module is part of a (larger) network. For example, Fig. 9 Any one of the modules 901, 902, 904, 905 and 906 in the first network is a subnet of the first network. Similarly, any one of the modules 903, 908, 909, 910 and 907 is a subnet of the second network. In the corresponding network, each processing unit (module) performs a specific function as required to respectively implement the processing of the entire first network and the second network. Fig. 9 In the example of , the functions are encoding-decoding processing of image data (first network) and encoding-decoding processing of side information (second subnet). In addition, the connected modules 901, 902 and 905 can be regarded as encoder subnets (i.e., subnets of the encoder-decoder network), while modules 904 and 906 can be regarded as decoder subnets. From the above discussion, it can be seen that, accordingly, the first network and the second network can each be interpreted as a subnet relative to the entire network including all processing units.

[0205] The first subnet is responsible for:

[0206] ● transform (901) the input image x to its latent representation y (which is easier to compress than x),

[0207] ● quantize (902) the latent representation y into a quantized latent representation

[0208] ● Arithmetic coding module 905 uses AE to compress the quantized latent representation To get the code stream "code stream 1",

[0209] ● Use the arithmetic decoding module 906 to parse the code stream 1 via AD,

[0210] ● Reconstruct the image using the parsed data (904)

[0211] The purpose of the second subnet is to obtain the statistical properties of the samples of "code stream 1" (such as the average value, variance and correlation between the samples of code stream 1) so that the first subnet can more efficiently compress code stream 1. The second subnet generates a second code stream "code stream 2", which includes the information (such as the average value, variance and correlation between the samples of code stream 1).

[0212] The second network includes an encoding part, which includes converting the quantized latent representation Transform (903) into side information z, and quantize the side information z into quantized side information And the quantized side information The encoding (e.g., binarization) (909) is performed into the bitstream 2. In this example, the binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes arithmetic decoding (AD) 910, which transforms the input bitstream 2 into decoded quantized side information. Possibly The same as arithmetic coding and decoding operations are lossless compression methods. Then, the decoded quantized side information is transformed (907) into decoded side information express The statistical properties of The average sample value, or the variance of the sample value, etc.). Then, the decoded potential representation is provided to the arithmetic encoder 905 and the arithmetic decoder 906 to control The probability model of .

[0213] Fig. 9 An example of a VAE (Variational Autoencoder) is described, the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain the statistical properties of the samples of bitstream 1. In one such implementation, there may be a context modeler whose goal is to extract the correlation information of bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 905 and arithmetic decoder (AD) 906 components.

[0214] Fig. 9 The encoder and decoder are shown in a single figure. As known to those skilled in the art, the encoder and decoder can and often are embedded in different devices, such as Fig. 9A and Fig. 9B Examples are given in the following sections.

[0215] Fig. 9A shows the encoder, Fig. 9B The decoder component of the VAE framework is shown. According to some embodiments, the encoder receives an image (image data) as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The output of the encoder (e.g. Fig. 9A ) are code stream 1 and code stream 2. Code stream 1 is the output of the first subnet of the encoder, and code stream 2 is the output of the second subnet of the encoder.

[0216] Similarly, in Fig. 9B In this example, two bitstreams (bitstream 1 and bitstream 2) are received as input and a reconstructed (decoded) image is generated at the output.

[0217] As mentioned above, VAE can be divided into different logical units that perform different operations, such as Fig. 9A and Fig. 9B Examples are given in the following sections. Fig. 9A Components involved in encoding a signal (such as a video) and provided with encoded information are shown. The encoded information is then, for example, Fig. 9B The decoder component shown in receives for encoding. Therefore, Fig. 9 , Fig. 9A and Fig. 9B The same reference numerals in the figures indicate that the corresponding processing units (modules) perform the same functions.

[0218] Specifically, Fig. 9A As shown in FIG. 1 , the encoder includes an encoder 901, which transforms an input x into a signal y and then provides the signal y to a quantizer 902. The quantizer 902 provides information to an arithmetic coding module 905 and a super encoder 903. The super encoder 903 provides the code stream 2 discussed above to a super decoder 907, and accordingly, the super decoder 907 sends signal information to the arithmetic coding module 605.

[0219] The output of the arithmetic coding module is code stream 1. Code stream 1 and code stream 2 are the outputs of signal encoding, and then code stream 1 and code stream 2 are provided (sent) to the decoding process.

[0220] Although unit 901 is referred to as an "encoder", it is also possible to Fig. 9A The complete subnetwork described in is called an "encoder". The encoding process usually refers to the unit (module) that converts the input into an encoded (e.g., compressed) output. Fig. 9AIt can be seen that unit 901 can actually be considered the core of the entire subnetwork, as it performs a transformation of the input x to y, which is a compressed version of x. Compression in encoder 901 can be achieved by applying a neural network or any processing network that generally has one or more layers. In such a network, compression can be performed by a cascade process (i.e., sequential processing) including downsampling, which reduces the size of the input and / or the number of channels. Therefore, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0221] The rest of the parts in the figure (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 901 through lossy compression. The AE 905 in conjunction with the super encoder 903 and super decoder 907 used to configure the AE 905 can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to Fig. 9A The entire subnetwork in is called an "encoder". A similar situation applies to Fig. 9B , where the entire subnetwork can be referred to as a "decoder".

[0222] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into a binary number (bit). For example, in the VAE framework, the encoder is a nonlinear transformation that maps the input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height and a smaller size, the dimension (size) of the signal is reduced, making it easier to compress the signal y. It should be noted that, in general, the encoder does not necessarily need to reduce the size in two (or usually all) dimensions. Instead, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or usually a subset).

[0223] The general principle of compression is Figure 8 The latent space is the output of the encoder and the input of the decoder, and the latent space represents the compressed data. It should be noted that the size of the latent space can be much smaller than the input signal size. Here, the term size can refer to the resolution, such as the number of samples of the feature map output by the encoder. The resolution can be given as the product of the number of samples in each dimension (e.g., the width × height × number of channels of the input image or feature map).

[0224] The reduction in input signal size is Figure 8 This is illustrated in , which shows a deep learning based encoder and decoder. Figure 8In , the input image x corresponds to the input data, which is the input to the encoder. The transformed signal y corresponds to a latent space that has a smaller dimension or size than the input signal in at least one dimension. Each column of circles represents a layer in the encoder or decoder processing chain. The number of circles in each layer represents the size or dimension of the signal at that layer.

[0225] from Figure 8 It can be seen that the encoding operation corresponds to a reduction in the size of the input signal, while the decoding operation corresponds to the reconstruction of the original size of the image.

[0226] One of the ways to reduce the size of a signal is by downsampling. As mentioned above, downsampling is the process of reducing the sampling rate of the input signal. For example, if the input image is of size h and w, and the downsampled output is h2 and w2, then at least one of the following is true:

[0227] ●h2 <h

[0228] w2 <w

[0229] The reduction in signal size usually occurs step by step along the chain of processing layers, rather than all at once. For example, if the input image x has dimensions (or dimensional sizes) of h and w (representing height and width, respectively), and the latent space y has dimensions of h / 16 and w / 16, then during encoding, the size reduction may occur in 4 layers, where each layer reduces the size of the signal by half in each dimension.

[0230] Some deep learning based video / image compression methods use multiple downsampling layers. For example, Fig. 6A The VAE framework shown uses 6 downsampling layers, which are labeled 601 to 606. In the layer descriptions, layers that include downsampling are indicated by downward arrows. The description of the layer "Conv N×5×5 / 2↓" means that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, 2↓ means that a downsampling factor of 2 is performed in this layer. Downsampling by a factor of 2 causes one of the dimensions of the input signal to be reduced by half at the output. Fig. 6A In the example, 2↓ indicates that the width and height of the input image are both reduced by half. Since there are 6 downsampling layers, if the width and height of the input image 814 (also represented by x) are w and h, the output signal The width and height of the grid are w / 64 and h / 64 respectively.

[0231] The modules denoted by AE and AD are the arithmetic encoder and arithmetic decoder, which have been referred to above. Fig. 9 , Fig. 9A and Fig. 9B These modules are explained. Arithmetic encoder and arithmetic decoder are specific implementations of entropy decoding. AE and AD (as Fig. 6A and Figure 6B Components 613 and 615 in FIG. 6 can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, which is a recoverable process. In addition, the "Q" in the figure corresponds to the above description of Fig. 6A and Figure 6B The quantization operation mentioned is explained in the above section "Quantization". In addition, the quantization operation and the corresponding quantization unit as part of the component 613 or 615 do not necessarily exist and / or can be replaced by another unit.

[0232] exist Fig. 6A and Figure 6B In the figure, a decoder including upsampling layers 607 to 612 is also shown. Another layer 620 is provided between the upsampling layers 611 and 610 in the order in which the input is processed, which layer 620 is implemented as a convolutional layer, but does not upsample the received input. A corresponding convolutional layer 620 for the decoder is also shown. Such a layer can be provided in the NN to perform operations on the input without changing the size of the input but changing specific features. However, it is not necessary to provide such a layer.

[0233] When the processing order of the code stream 2 through the decoder is followed, the order in which the upsampling layers are passed is opposite, that is, from the upsampling layer 612 to the upsampling layer 607. Each upsampling layer is shown here to provide an upsampling with an upsampling ratio of 2, and the upsampling is represented by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, and the upsampling layers can use other upsampling ratios such as 3, 4, 8, etc. Layers 607 to 612 are implemented as convolutional layers (conv). Specifically, since these convolutional layers are used to perform operations on the input opposite to the operations performed by the encoder, the upsampling layers can apply a deconvolution operation to the received input so that the size of the input is increased by a factor corresponding to the upsampling ratio. However, the present invention is generally not limited to deconvolution, and upsampling can be performed in any other manner, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample replication, etc.

[0234] In the first subnet, some convolutional layers (601 to 603) are followed by generalized divisive normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second subnet, the activation function applied is ReLU. It should be noted that the present invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLU.

[0235] Figure 6Bshows another example of a VAE-based encoder-decoder structure similar to Fig. 6A A structure in Figure 6B In the figure, it is shown that the encoder and decoder may include multiple downsampling layers and upsampling layers. Each layer applies a downsampling factor of 2 or an upsampling factor of 2. In addition, the encoder and decoder may include other components, such as a generalized divisive normalization (GDN) 650 on the encoder side and an inverse GDN (IGDN) 655 on the decoder side. In addition, both the encoder and decoder may include one or more ReLUs, specifically leaky ReLUs 660 and 665. A decomposition entropy model 670 may also be provided on the encoding side and a Gaussian entropy model 670 may be provided on the decoding side. In addition, multiple convolution masks 680 may be provided. In addition, in Figure 6B In an embodiment of the invention, the encoder includes a universal quantizer (UnivQuan) and the decoder includes an attention module.

[0236] The total number of downsampling operations and the stride size define the conditions on the input channel size, i.e. the size of the input to the neural network.

[0237] Here, if the input channel size is an integer multiple of 64 = 2×2×2×2×2×2, then after all downsampling operations, the channel size remains integer. By applying the corresponding upsampling operation in the decoder during upsampling, and by applying the same rescaling at the end of the input processing of the upsampling layer, the output size is again the same as the input size at the encoder.

[0238] Thereby, a reliable reconstruction of the original input is obtained.

[0239] Accepted domains:

[0240] In the context of neural networks, the receptive field is defined as the size of the region in the input that produces a sample at the output feature map. Basically, the receptive field is a measure of how well the output features (of any layer) are associated with a region (patch) of the input. It should be noted that the concept of receptive field applies to local operations (i.e. convolution, pooling, etc.). For example, a convolution operation with a kernel of size 3×3 has a receptive field of 3×3 samples in the input layer. In this example, the convolution node uses 9 input samples to obtain 1 output sample.

[0241] Total Receptive Domains:

[0242] The total receptive field (TRF) refers to the set of input samples used to obtain a specified set of output samples by applying one or more processing layers of a neural network.

[0243] The total receptive field can be Fig.10 Let's take an example. Fig.10 In Figure 1, an example of processing a one-dimensional input (7 samples on the left side of the figure) with 2 consecutive transposed convolutional (also called deconvolutional) layers is illustrated. The input is processed from left to right, that is, "Deconvolution Layer 1" processes the input first, and "Deconvolution Layer 2" processes the output of "Deconvolution Layer 1". In this example, the kernel size in both deconvolution layers is 3. This means that 3 input samples are required to obtain 1 output sample at each layer. In this example, the output sample set is marked within the dashed rectangle and includes 3 samples. Due to the size of the deconvolution kernel, 7 samples are required at the input to obtain an output sample set that includes 3 output samples. Therefore, the total receptive field of the 3 labeled output samples is 7 samples at the input.

[0244] exist Fig.10 In the example above, there are 7 input samples, 5 intermediate output samples, and 3 output samples. The reduction in the number of samples is due to the fact that the input signal is finite (not extending to infinity in every direction), and there are "missing samples" at the boundaries of the input. In other words, since the deconvolution operation requires 3 input samples for each output sample, if the number of input samples is 7, only 5 intermediate output samples can be generated. In fact, the number of output samples that can be generated is (k–1) samples less than the number of input samples, where k is the kernel size. Since in Fig.10 In , the number of input samples is 7, and after the first deconvolution with kernel size 3, the number of intermediate samples is 5. After the second deconvolution with kernel size 3, the number of output samples is 3.

[0245] like Fig.10 As shown, the total receptive field for the 3 output samples is 7 samples at the input. The size of the total receptive field is increased by successively applying processing layers with kernel size greater than 1. Typically, the total receptive field for a set of output samples is calculated by tracing the connections from the output layer to each node in the input layer, and then finding the union of all samples in the input that are directly or indirectly (through more than one processing layer) connected to the set of output samples. For example, in Fig.10 In , each output sample is connected to 3 samples in the previous layer. The union includes 5 samples in the middle output layer, which are connected to 7 samples in the input layer.

[0246] Sometimes, it is desirable to keep the number of samples the same after each operation (convolution or deconvolution or whatever). In such cases, padding can be applied at the boundaries of the input to compensate for the "missing samples". Fig.11 This is shown when the number of samples remains equal. It should be noted that the present invention is applicable to both cases, since padding is not a mandatory operation of convolution, deconvolution or any other processing layer.

[0247] This should not be confused with downsampling. During downsampling, for every M samples at the input, there are N samples at the output, where N < M. The difference is that M is typically much smaller than the number of inputs. In Fig.10 , there is no downsampling, and the reduction in the number of samples is due to the fact that the size of the input is not infinite and there are "missing samples" at the input. For example, if the number of input samples is 100 and the kernel size is k = 3, then when using two convolutional layers, the number of output samples is 100 – (k – 1) – (k – 1) = 96. Instead, if two transposed convolutional layers are performing downsampling (the downsampling ratio is the ratio of M and N, M = 2, N = 1), then the number of output samples is:

[0248]

[0249] Fig.12 An example of downsampling using 2 convolutional layers with a downsampling ratio of 2 (N = 1 and M = 2) is illustrated. In this example, due to the combined effect of downsampling at the boundaries and "missing samples", 7 input samples become 3. The number of output samples can be calculated using the following formula after each processing layer: where k is the kernel size and r is the downsampling ratio.

[0250] The convolutional operation and the transposed convolutional (i.e., deconvolution) operation are identical from the perspective of mathematical expressions. The difference stems from the fact that the deconvolution operation assumes a previous convolutional operation has occurred. In other words, deconvolution is the process of filtering a signal to compensate for a previously applied convolution. The goal of deconvolution is to recreate a signal that is the same as the one that existed before the convolution took place. The present invention is applicable to both convolutional operations and deconvolution operations (in fact, to any other operation with a kernel size greater than 1, as explained later).

[0251] Fig.13 Another example explaining how to calculate the total receptive field is shown. In Fig.13 , a two-dimensional array of input samples is processed by two convolutional layers, each with a kernel size of 3×3. After applying two transposed convolutional layers, an output array is obtained. The set of output samples (array) is marked with solid rectangles ("output samples") and includes 2×2 = 4 samples. The total receptive field of this set of output samples includes 6×6 = 36 samples. The total receptive field can be calculated as:

[0252] ● Each output sample is connected to 3×3 samples in the intermediate output. The union of all samples in the intermediate output that are connected to the set of output samples includes 4×4 = 16 samples.

[0253] Each of the 16 samples in the intermediate output is connected to 3×3 samples in the input. The union of all samples in the input that are connected to the 16 samples in the intermediate output includes 6×6=36 samples. Therefore, the total receptive field of the 2×2 output samples is 36 samples at the input.

[0254] In image and video compression systems, compression and decompression of very large input images are usually performed by dividing the input image into multiple parts. VVC and HEVC adopt this division method, for example, dividing the input image into blocks or wavefront processing units.

[0255] When partitioning is used in conventional video coding systems, an input image is usually divided into a plurality of rectangular shaped portions. Fig.14 An example of such a segmentation is given in Fig.14 In the MPEG-4 decoder, part 1 and part 2 can be processed independently of each other, and the code stream used to decode each part is encapsulated into independent decodable units. Therefore, the decoder can independently parse (obtain the syntax elements required for sample reconstruction) each code stream (corresponding to part 1 and part 2) and can independently reconstruct the samples of each part.

[0256] exist Fig.15 In the wavefront parallel processing shown, each part usually consists of 1 row of coding tree blocks (CTB). The difference between wavefront parallel processing and blocking is that in wavefront parallel processing, the bitstreams corresponding to each part can be decoded almost independently of each other. However, sample reconstruction cannot be performed independently because the sample reconstruction of each part still has dependencies between samples. In other words, wavefront parallel processing makes the parsing process independent while maintaining the dependency of sample reconstruction.

[0257] Wavefront and blocking are both techniques that can perform all or part of the decoding operation independently of each other. The benefits of independent processing are:

[0258] ●More than one identical processing core can be used to process the entire image, which can increase processing speed.

[0259] ● If the processing core is not powerful enough to process a large image, the image can be divided into multiple parts so that less resources are required for processing. In this case, the less powerful processing unit can process each part even if it cannot process the entire image due to resource limitations.

[0260] In order to meet the processing speed and / or memory requirements, HEVC / VVC uses processing memory large enough to handle the encoding / decoding of the entire frame. Top-of-the-line GPU cards are used to achieve this goal. In the case of traditional codecs such as HEVC / VVC, the memory requirement for processing the entire frame is usually not a big problem because the entire frame is divided into blocks and then each block is processed one by one. However, processing speed is a major issue. Therefore, if a single processing unit is used to process the entire frame, the speed of the processing unit must be very high, so the processing unit is usually very expensive.

[0261] On the other hand, NN-based video compression algorithms consider the entire frame in encoding / decoding, which is different from the block-based video compression algorithms used by traditional hybrid encoders. The memory requirement is too high to be handled by NN-based encoding / decoding modules.

[0262] In traditional hybrid video encoders and decoders, the amount of memory required is proportional to the maximum supported block size. For example, in VVC, the maximum block size is 128×128 samples.

[0263] However, the memory required for NN-based video compression is proportional to the size W×H, where W and H represent the width and height of the input / output image. It can be seen that the memory requirement can be very high compared to hybrid video encoders, as typical video resolutions include 3840×2160 image sizes (4K video). In image and video compression systems, compression and decompression of input images of very large size are usually performed by dividing the input image into multiple parts. To cope with memory limitations, NN-based video decoding algorithms can apply blocking in the latent space.

[0264] When patching is applied without overlap, boundary artifacts may be visible in the reconstructed image. This problem can be partially addressed by overlapping the patches in the latent space or signal domain, where the overlap is large enough to avoid these artifacts. If the overlap is larger than the size of the receptive field of the NN, the operation may be performed in a non-canonical manner and may incur some overhead due to computational complexity. Correspondingly, if the overlap is smaller than the size of the receptive field, the patching operation is not lossless / transparent and needs to be specified (canonicalized).

[0265] Furthermore, in the case of structures with multiple pipelines (e.g., pipelines for processing luma and / or chroma, or in general, multiple channels of an input tensor) or multiple subnetworks, a straightforward approach where the tiles are always the same size may result in performance losses. Fig. 9: Unit 908) is another computationally complex and memory intensive operation. Choosing the same tile size for all subnetworks in RDOQ may also not be optimal. In addition, if the scene representations in different pipelines are not sample aligned during tile segmentation (e.g., CCS), direct tile segmentation may not preserve information about the correlation along different components. Sample alignment refers to the mismatch between the size of luma and the size of chroma. In these cases, the performance of video processing may be poor because the spatial and / or temporal correlations are missing or at least not as obvious. As a result, the quality of the reconstructed image may be reduced.

[0266] Another problem is that in order to process large inputs using a single processing unit (such as a CPU or GPU), the processing unit must be very fast because the unit needs to perform a large number of operations per unit time. This requires the unit to have a high clock frequency and high memory bandwidth, which are costly design criteria for chip manufacturers. In particular, it is not easy to increase memory bandwidth and clock frequency due to physical limitations.

[0267] Although state-of-the-art deep learning-based image and video compression algorithms follow the variational auto-encoder (VAE) framework, NN-based video decoding algorithms for encoding and / or decoding are still in the early stages of development and no consumer devices include Fig. 9 , Fig. 9A and Fig. 9B The implementation of VAE shown. In addition, the cost of consumer devices is greatly affected by the implemented memory.

[0268] Therefore, in order for NN-based video coding algorithms to become cost-effective for implementation in consumer devices such as mobile phones, the memory footprint of the processing unit and the required operating frequency need to be reduced. This optimization has not yet been done.

[0269] The present invention is applicable to both end-to-end AI codecs and hybrid AI codecs. For example, in a hybrid AI codec, filtering operations (filtering of reconstructed images) can be performed by a neural network (NN). The present invention is applied to such NN-based processing modules. Generally, if at least part of the processing includes an NN, and if the NN includes a convolution or transposed convolution operation, the present invention can be applied to the entire or partial video compression and decompression process. For example, the present invention is applicable to separate processing tasks performed by an encoder and / or decoder as a processing part, including in-loop filtering, post-filtering and / or pre-filtering, and rate distortion optimization quantization (RDOQ) for encoders only.

[0270] Some embodiments of the present invention can achieve a balance between memory resources and computational complexity in a NN-based video codec framework, thereby providing a solution to the above problem. Specifically, the present invention provides the possibility to process parts of the input independently, and still have different image components aligned by samples. This reduces memory requirements while maintaining compression performance and with almost no increase in computational complexity.

[0271] The processing may be decoding or encoding. In the exemplary implementation discussed below, the neural network (NN) may be:

[0272] - A network comprising at least one processing layer, wherein more than one input sample is used to obtain an output sample (this is the general condition when the problem to be solved by the present invention arises).

[0273] ● A network comprising at least one convolutional (or transposed convolutional) layer. In one example, the kernel of the convolution is greater than 1.

[0274] ●A network that includes at least one pooling layer (max pooling, average pooling, etc.).

[0275] ●Decoding network, super-decoder network or encoding network.

[0276] ●A part of the above network (subnet).

[0277] The input can be:

[0278] ●Feature map.

[0279] ●The output of the hidden layer.

[0280] ●Latent space feature map: The latent space can be obtained based on the bitstream.

[0281] ●Input image.

[0282] First embodiment

[0283] The following describes a method and apparatus for image / video encoding-decoding (compression-decompression), wherein multiple subnetworks are used to process input tensors representing image data, such as Fig.16 shown.

[0284] In this exemplary and non-limiting embodiment, a method for encoding an input tensor representing image data is provided. The input tensor may have a matrix form with width=w, height=h in spatial dimensions, and a third dimension (e.g., the number of channels) of size equal to D. For example, the input tensor may be directly an input image having D components, which may include one or more color components, and possibly other channels, such as a depth channel or a motion channel, etc. However, the present invention is not limited to this input. In general, the input tensor may be a representation of image data, for example, a potential representation that may be a result of previous processing (e.g., preprocessing).

[0285] The input tensor is processed by a neural network including at least a first subnetwork and a second subnetwork. Fig.16 An example of a first subnet and / or a second subnet of a coding branch is shown, including an encoder 1601 and a rate distortion optimizing quantizer (RDOQ) 1602. The processing includes: applying the first subnet to a first tensor, including: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks by the first subnet. After applying the first subnet, applying the second subnet to a second tensor, including: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks by the second subnet.

[0286] exist Fig.16 In the example of , the output of the first subnet 1601 is provided as the input of the second subnet, and the second subnet is RDOQ 1602. In this case, the first tensor is the input image x representing image data, which can be the original image data. Correspondingly, the second tensor input to the second subnet is the feature tensor in the latent space. However, the present invention is not limited to the case where the first subnet and the second subnet are directly cascaded. Typically, the second subnet is after the first subnet, that is, the second subnet is applied after the first network is applied, but there may be some additional processing between the first subnet and the second subnet. Therefore, the above term "after" does not limit the above processing to after the output of the first subnet is directly input to the second subnet. Instead, for example, "after" refers to the first plurality of blocks and the second plurality of blocks being processed within the same processing path.

[0287] One or more channels of the first tensor and the second tensor are divided into blocks, which basically represent data obtained by splitting the input tensor in one spatial dimension or multiple spatial dimensions. As with current video decoding standards, blocks are used to provide the possibility of parallel (i.e., independent of each other) decoding. Blocks may include one or more examples. Blocks may have a rectangular shape, but are not limited to these regular shapes. The rectangular shape may be a square shape. Fig.14An example of partitioning into tiles of regular shapes is shown. However, the present invention is not limited to rectangles or specifically to square shapes. For example, the shape may be chess-like or irregular. The first input tensor and the second input tensor may be partitioned so that the tiles have triangular shapes or any other shapes, which may depend on the specific application and / or the type of processing performed by the subnet.

[0288] exist Fig.16 , the general processing of N components is shown in subnetworks 1601 and 1602. In this context, components may be input tensor channels (e.g., color components or latent space representations) that may be processed in parallel. Such processing may include partitioning the channels into tiles (including a first plurality of tiles) in the spatial domain.

[0289] but, Fig.16 The N components in may also correspond to corresponding N blocks (or groups of blocks) of one or more channels. The N blocks (or groups of blocks) may be processed in parallel. After such processing of the first plurality of blocks and / or the second plurality of blocks, the corresponding subnetwork may fuse the processed blocks into the output tensor. Fig.16 For the subnet encoder 1601, such a fused output tensor is y, and for the subnet RDOQ 1602, such a fused output tensor is It should be noted that in Fig.16 In the example, the number of components in the first subnet 1601 and the second subnet 1602 is the same (N). However, this is not necessarily the case, and there may be a different number of parallel processing paths in the first subnet than in the second subnet. For example, in the case of post-filtering, the first subnet is a post-filter, which processes each component (e.g., Y, U, and V) separately. Correspondingly, in the case of encoding or decoding processing, the encoder or decoder is the second subnet, respectively, where a distinction is made between Y and UV, i.e., UV is processed jointly.

[0290] In addition, at least two corresponding collocated chunks in the first plurality of chunks and the second plurality of chunks are of different sizes. Here, the term "collocated" refers to two chunks (i.e., one from the first plurality of chunks and one from the second plurality of chunks) being located at corresponding (e.g., at least partially overlapping) locations within the first input tensor and the second input tensor in terms of spatial dimensions. In other words, the manner in which each subnet is subdivided into chunks may be different, and thus each subnet may use different chunks (divided into chunks in different ways).

[0291] In an exemplary implementation, the input (in the spatial domain) of each subnet (i.e., the first tensor and the second tensor) is divided into a grid of tiles of the same size, except for the tiles at the bottom and right image boundaries, which can have a smaller size because the input tensor does not necessarily have a size that is an integer multiple of the tile size. Such a grid of tiles of the same size is advantageous because such a grid can be efficiently indicated in the codestream and has low processing complexity. On the other hand, a grid including tiles of different sizes can produce better performance and content adaptability.

[0292] In the above exemplary implementation, the blocks in the first plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap in at least one of the spatial dimensions. Additionally or alternatively, the blocks in the second plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap in at least one of the spatial dimensions. The term adjacent means that the corresponding blocks are adjacent. Adjacent block portion 1 and portion 2 are as follows Fig.14 As shown, they are adjacent but not overlapping. Fig.17A Partial overlap is shown, where, for example, the first tensor is divided into four blocks in 2D (i.e., regions L1, L2, L3, and L4) in the xy plane. Similar considerations apply to the second input tensor. The first block of the first plurality of blocks is L1 and the second block is L2. L1 and L2 are adjacent to each other in the x-axis direction and have overlapping boundaries along the y-axis. As shown, L1 and L2 partially overlap. Partial overlap means that blocks L1 and L2 include one or more of the same tensor elements. In some embodiments, the tensor elements may correspond to image samples of the first input tensor.

[0293] Fig. 17B Shown with Fig.17A The same x-axis and y-axis directions partially overlap the scene. Fig.17A and Fig. 17B In , L1 also overlaps with its neighboring block to the right (in the y dimension, with an overlapping border along the x dimension). L1 also overlaps slightly with its diagonally adjacent block (in both dimensions). Similarly, L2 overlaps with its directly and diagonally adjacent block to the right. Fig.18 Another example of partial overlap of blocks L1 and L2 is shown, where blocks L1 and L2 may be blocks in the first plurality of blocks. L1 and L2 overlap with each other in only one corresponding dimension. L1 partially overlaps with its neighboring block to the right, with the boundary along the x-axis (i.e., partial overlap in the y-dimension). Accordingly, L2 partially overlaps with the neighboring block L1 at the top. Fig.17A and Fig. 17B In the example of , the overlap between L1 and L2 means that L1 also includes samples from L2, and L2 also includes samples from L1. Fig.18In the example, L2 also includes samples from L1, but L1 does not include samples from L2. As known to those skilled in the art, other variations of overlap may exist. The present invention is not limited to any particular manner or extension of overlap. The present invention may also include block arrangements without overlap.

[0294] Fig.19 is another example of L1 and L2, which do not overlap with each other or with any other tiles. Fig. 20 Similar to Fig.17A and Fig. 17B , further details regarding the partial overlapping areas are further explained below.

[0295] In an exemplary implementation, the first subnet independently processes a block (e.g., L1 and L2) from the first plurality of blocks. Additionally or alternatively, the second subnet independently processes a block from the second plurality of blocks. In other words, the processing of the blocks is independent of each other and therefore mutually independent. Independent processing provides the possibility of parallelization. For example, in some implementations, the first subnet processes at least two blocks from the first plurality of blocks in parallel and / or the second subnet processes at least two blocks from the second plurality of blocks in parallel. Parallel processing is Fig.16 , involving processing 1 to processing N (processing paths 1 to N) in a subnetwork encoder 1601 or a quantizer RDOQ 1602. Using the encoder subnetwork 1601 as the first subnetwork, the encoder 1601 takes an input tensor x, which is divided into N blocks x1 to xN of the first input tensor. Then, the corresponding blocks process the corresponding blocks, i.e., processing 1 to processing N, and they do not need to interact with each other (e.g., wait for each other during processing). The result of each processing is the output tensor y1 to yN of the corresponding block, which can be a feature map in the latent space. The output tensors y1 to yN can also be combined into an output tensor y. The combination can (but does not necessarily) involve as shown in Figures 17 to Fig.19 The cropping shown in . Note that the combination into the tensor y does not need to be performed. It is conceivable that the second subnet reuses the tiles of the first subnet and merely modifies it (making the tiles smaller by further dividing them, or making them larger by merging multiple tiles into one). Fig.16 In the example of , processes 1 to N may perform processing based on blocks, that is, process i processes block i. Alternatively, process i may process component i of multiple components of the input tensor. In this case, process i divides component i into multiple blocks and processes the blocks separately or in parallel.

[0296] Above, for simplicity, an example has been given of processing all N input tensor blocks in parallel through corresponding N processing paths. However, the present invention is not limited to such parallel processing. There may be more than N blocks in the input tensor, divided into N groups of blocks, which are processed in parallel in corresponding N processing paths (instances of the first subnet and / or instances of the second subnet). As known to those skilled in the art, once the blocks are independent of each other, they can in principle be processed in parallel. The technician can design any number of parallel processing paths according to the corresponding performance requirements and / or hardware availability.

[0297] like Fig.17A / 17B to Fig. 20 As shown, the corresponding block has a certain size, which may have a certain width and a certain height. In the case of rectangular blocks, the width and height may be different, and in the case of square blocks, the width and height may be the same. In one implementation, the partitioning of the first tensor includes determining the size of the blocks in the first plurality of blocks based on a first predefined condition; and / or the partitioning of the second tensor includes determining the size of the blocks in the second plurality of blocks based on a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data. As an example of available hardware resources, the first predefined condition and / or the second predefined condition may be memory resources of a processing device (decoder or encoder). When the amount of available memory is less than a predefined value (amount of memory resources), the determined block size may be less than the block size when the available memory is equal to or greater than the predefined value. However, hardware resources are not limited to memory. The first condition and / or the second condition may be based on the availability of processing power, such as the number of processors and / or the processing speed of one or more processors.

[0298] Alternatively or in addition, the presence of motion may be used for the first condition and / or the second condition. For example, in the case where there is more motion in the portion of the input tensor corresponding to the tile, a smaller tile size may be determined compared to the case where the presence of the motion is less pronounced or there is no motion at all. Whether the motion is pronounced (i.e., fast motion and / or fast / frequent motion changes) may be determined by the corresponding motion vector based on the change in its amplitude and direction, and compared with corresponding predefined values ​​(thresholds) of amplitude and / or direction and / or frequency.

[0299] Another or additional condition may be a region of interest (ROI), wherein the tile size may be determined based on the presence of the ROI. For example, a ROI in the image data may be a detected object (e.g., a vehicle, a bicycle, a motorcycle, a pedestrian, an animal, etc.) that may have different sizes, move quickly or slowly, and / or change their direction of motion quickly or slowly and / or multiple times (i.e., more frequently relative to a predefined frequency value). In an exemplary implementation, the tile size in at least one dimension of the ROI may be smaller than the tile size of the rest of the input tensor.

[0300] Therefore, the tile size (including scene-specific tile size) can be adjusted or optimized to adapt to hardware resources or the content of the image data. The tile size can also be adjusted or optimized jointly to adapt to hardware resources and the content of the image data.

[0301] Fig.17A / 17B to Fig. 20 The example illustrates dividing the first (or second) tensor into a plurality of blocks, which partially overlap with their neighboring blocks. As a result of the overlapping and / or block processing, the processed blocks corresponding to the region Ri may be cropped. Fig.17A / 17B to Fig. 20 In , the indices of L and R are the same for the corresponding partitions in the input and output. For example, L4 corresponds to R4. The placement of Ri follows the same pattern as Li, which means that if L1 corresponds to the upper left corner of the input space, then R1 corresponds to the upper left corner of the output space. If L2 is to the right of L1, then R2 is to the right of R1. Fig.17A In the example, the first tensor is partitioned so that each region L i Including R i In addition, R i The union of constitutes the entire target image R.

[0302] The determination of the total receptive field depends on the kernel size of each processing layer. The total receptive field can be determined by tracing the input samples of the first tensor in the reverse direction of processing. The total receptive field consists of the union of the input samples that are all used to compute a set of output samples. Therefore, the total receptive field depends on the connections between each layer and can be determined by tracing all connections from the output in the direction of the input.

[0303] exist Fig.13 In the example of convolutional layers shown, the kernel sizes of convolutional layers 1 and 2 are K1×K1 and K2×K2, respectively, and the downsampling ratios are R1 and R2, respectively. Convolutional layers usually use regular input-output connections (e.g., each output always uses K×K input samples). In this case, the calculation of the total receptive field size can be shown as follows:

[0304] W=((wxR2)+K2-1)xR1+(K1-1),

[0305] H=((hxR2)+K2-1)xR1+(K1-1),

[0306] Among them, H and W represent the size of the total receptive field, and h and w represent the height and width of the output sample set, respectively.

[0307] In the above examples, the convolution operation is described in a two-dimensional space. When the dimensions of the space to which the convolution is applied are large, a 3D convolution operation can be applied. The 3D convolution operation is a direct extension of the 2D convolution operation, in which an additional dimension is added to all operations. For example, the kernel size can be expressed as K1×K1×N and K2×K2×N, and the total receptive field can be expressed as W×H×N, where N represents the size of the third dimension. Since the extension of the 2D convolution operation and the 3D convolution operation is simple, the present invention is applicable to the 2D convolution operation and the 3D convolution operation. In other words, the size of the third dimension (or even the fourth dimension) can be greater than one, and the present invention can be applied in the same manner.

[0308] The above formula is an example that shows how to determine the size of the total receptive field. The determination of the total receptive field depends on the actual input-output connections of each layer. The output of the encoding process is R i . R i The union of y1 to y2 constitutes the feature tensor y. N Fusion Fig.16 ). In this example, R i have overlapping areas, so first apply the cropping operation to get an R-crop with no overlapping areas i Finally, R-crop i Cascade to obtain the fused feature tensor y. In this embodiment, L i Including R as mentioned above i The total receptive field of .

[0309] R i and L i The determination of (i.e. the size of the first plurality of partitions and / or the second plurality of partitions) may be performed as follows:

[0310] ●First determine N non-overlapping regions R-crop i For example, R-crop i There can be N×M regions of equal size, where N×M is determined by the decoder according to memory limitations.

[0311] ●Determine R-crop i The total receptive field of L iSet separately for each R-crop i The total receptive field of .

[0312] ●Process each L i , to obtain R i This refers to R i is the size of the output sample set generated by the NN. It should be noted that no actual processing may be required. Once L is determined i The size of R can be determined based on the function i The size and position of NN are known, and the size L i and R i The relationship between is known. Therefore, according to L i Calculate R by function i , without actually performing the processing.

[0313] ●If the size R i Not equal to R-crop i , then cut R i To obtain R-crop i .

[0314] ■ If padding is applied during NN processing of input or intermediate output samples, R-crop i The size may not be equal to R i For example, padding may be applied to a NN if the input and intermediate output samples must meet certain size requirements. For example, a NN (e.g., due to its specific structure) may require that the input size must be a multiple of 16 samples. In this case, if L i is not a multiple of 16 samples, padding can be applied in that direction to make it a multiple of 16. In other words, the padding samples are dummy samples used to ensure that each Li has an integer multiple of the size requirement of the corresponding NN layer.

[0315] ■ The samples to be cropped may be obtained by determining the output samples including the padding samples in their calculation. This option may be applicable to the exemplary implementation discussed.

[0316] This is Fig.17A As shown in FIG. 1 , in which, through the first subnet (eg Fig.16 After the encoder 1601 in FIG. 1 processes the first plurality of blocks (i.e., regions Li), the size Ri output by the first subnet is too large and thus may not be suitable for subsequent subnets (e.g., Fig.16 Therefore, in this example, Fig.17A The processed blocks R1 and R2 are subjected to a cropping operation, after which the cropped blocks R-crop1 to R-crop4 are fused.

[0317] Fig. 17B An example of partial overlap of regions L1 to L4 (ie, the first plurality of tiles and / or the second plurality of tiles) is shown, wherein no cropping of the processed tiles (ie, regions R1 to R4) is involved. i and L i The determination of can be done as follows and can be called Fig. 17B The "simple" non-clipping case shown:

[0318] ●First, determine N non-overlapping regions R i For example, R i There can be N×M regions of equal size, where N×M is determined by the decoder according to memory limitations.

[0319] ●Determine R i The total receptive field of L i Set to be equal to each R i The total receptive field is obtained by i Each output sample is traced in the backward direction until L i Therefore, L i Used to calculate R i The sample consists of all samples with at least one sample in .

[0320] ●For each L i Process to obtain R i This refers to R i is the size of the output sample set generated by the NN.

[0321] The exemplary implementation addresses the total peak memory problem by partitioning the input space into multiple smaller independently processable regions.

[0322] In the above exemplary implementation, compared with not dividing the input into regions, L i The larger the overlap, the more additional processing is required. In particular, in some cases, R i The total receptive field of may be too large. In this case, the total number of computations to obtain the entire reconstructed image may increase too much.

[0323] Fig.18 Another example of partially overlapping partitions (i.e., regions L1 to L4) is shown, which also involves a similar Fig.17A The processed blocks R1 to R4 are cropped. Fig.17A and Fig. 17B Compared with the input area L i (i.e., chunking) is now Fig.18 The smaller ones are shown because they represent only the corresponding R iA subset of the total receptive domains of . The corresponding subnet (e.g. Fig.16 The encoder 1601 and / or RDOQ 1602) processes each region Li independently, thereby obtaining two regions R1 and R2 (i.e., output subsets). Since a subset of the total receptive field is used to obtain a set of output samples, a filling operation may be required to generate missing samples. In an exemplary implementation, the processing of the first plurality of blocks by the first subnet and / or the processing of the second plurality of blocks by the second subnet may include filling before processing with one or more layers. Therefore, the missing samples in the input subset can be added by the filling process, which improves the quality of the reconstructed output subset Ri. Therefore, after combining the output subset Ri, the quality of the reconstructed image is also improved.

[0324] To recap, padding refers to increasing the size of the input (i.e., input image) by either using predefined sample values ​​or generating new samples (at the image boundaries) using sample values ​​at locations in the input image. Fig.11 . The generated samples are approximations of actual sample values ​​that do not exist. Thus, for example, the filling samples can be obtained based on one or more nearest neighbor samples of the sample to be filled. For example, the samples are filled by copying the nearest neighbor samples. If there are more neighboring samples at the same distance, the neighboring samples to be used among these neighboring samples can be specified by a convention (such as a standard). Another possibility is to interpolate the filling samples from multiple neighboring samples. Alternatively, the filling can include using zero-valued samples. It may also be necessary to fill in intermediate samples generated by the processing. The intermediate samples can be generated based on samples of an input subset that includes one or more filling samples. The filling can be performed before the input of the neural network or within the neural network. However, the filling should be performed before processing one or more layers.

[0325] For completeness, Fig.19 Another example is shown, where after the output subset Ri is cropped, the corresponding cropped regions Ri-crop are seamlessly merged without any overlap of the cropped regions. Fig.17A , Fig. 17B and Fig.18 In contrast to the example of , the regions Li are non-overlapping. The corresponding implementation can be as follows:

[0326] 1. Determine N non-overlapping regions Li (i.e., the first plurality of blocks and the second plurality of blocks) in the first tensor and the second tensor respectively, wherein at least one of the regions includes R i A subset of the total receptive field of one of the i The union of constitutes the complete output of the first or second subnet (e.g., feature tensor y).

[0327] 2. Use NN to process each L independently i , to obtain the region Ri .

[0328] 3. Fusion R i to obtain the fused output tensor.

[0329] Fig.19 This case is shown, where Li is chosen to be a non-overlapping region. This is a special case where the total amount of computation required to obtain the entire reconstructed output (i.e., the output image) is minimized. However, the reconstruction quality may be affected.

[0330] As described above, the first subnet and the second subnet are part of a neural network. The subnet itself is a neural network, including at least one processing layer. In one implementation, the first subnet performs processing through one or more layers including at least one convolutional layer and at least one pooling layer; and / or the second subnet performs processing through one or more layers including at least one convolutional layer and at least one pooling layer.

[0331] In one implementation example, the first subnet and the second subnet perform respective processing as part of image or moving image compression.

[0332] In addition, the first subnet and / or the second subnet performs one of the following steps: image encoding of the convolutional subnet; rate distortion optimization quantization (RDOQ); image filtering. As described above, the first subnet and / or the second subnet may be Fig. 9 , wherein the input image x is first processed by the encoder 901 to perform image coding. The encoder 901 may be part of a VAE encoder-decoder framework, such as Fig. 6A and Figure 6B As shown, the corresponding encoder "ga" processes the input image through a series of convolutional layers 601 to 604 (including processing through the GDN layer) to perform corresponding image encoding. Similarly, Fig. 6A and Figure 6B The quantizer "Q" in can perform the function of quantization or RDOQ of the first subnet and / or the second subnet. The same applies to Fig. 9 Unit Q or RDOQ 902 and 908 in the image post-filtering Fig. 6A / Figure 6B and Fig. 9 Typically, some processing layers of the neural network at the decoding device may have post-filtering or general filtering functions.

[0333] For example, Fig.16An example of an encoding device is shown, which may correspond to a neural network and includes an encoder subnet 1601 as a first subnet and an RDOQ subnet 1602 as a second subnet. However, the present invention is not limited to such an embodiment. The neural network may be a network for decoding an image or a latent representation and includes a decoding subnet and a post-filtering subnet. Other examples of subnets are possible, including a preprocessing subnet or other subnets.

[0334] like Fig.16 As shown in , the encoding device (or generally, the encoding process) may also include a super encoder 1603 and a quantizer 1608 for the output z of the super encoder 1603, and a quantizer 1608 for treating the quantization information on the super prior included in the code stream 2 Arithmetic encoder 1609 performs encoding.

[0335] The decoding device (or decoding process) may also include an arithmetic decoder 1610 of the super a priori information, followed by a super decoder 1607. The super a priori part of the encoding and decoding may correspond to Fig. 6A and Figure 6B The VAE framework in or its variants.

[0336] Fig. 6A and Figure 6B An encoder within a NN-based VAE framework is shown, with corresponding convolutional layers 601 to 606, wherein the corresponding size of the input image (image data) is subsequently reduced by half. For example, the number of samples (i.e., samples of image data) used by the neural network (NN) may depend on the kernel size of the first input layer of the NN. One or more layers of the neural network (NN) may include one or more pooling layers and / or one or more subsampling layers. The NN may pool multiple samples through one or more pooling layers to generate one sample of an output subset. Alternatively or in addition, one output sample may be generated by the NN through one or more downsampling convolutional layers (e.g. Figure 6B The output sample is generated by subsampling (i.e., downsampling) of the convolutional layers 601 to 606 in FIG. Pooling and downsampling can be combined to generate an output sample. Fig.12 Shown is the downsampling of two convolutional layers, starting from 7 samples in the total receptive field and providing one sample as output.

[0337] exist Fig. 6A and Figure 6BIn , the input image 614 corresponds to image data. According to one implementation, the input tensor 614 is an image or image sequence comprising one or more components, wherein at least one component is a color component. Alternatively, the input tensor may be a latent space representation of an image, which may be an output of a preprocessing (e.g., an output tensor). One or more components are color components and / or depth and / or motion maps and / or other feature maps associated with an image sample.

[0338] It should be noted that the input tensor may represent other types of data (e.g., any type of multidimensional data) having one or more spatial components, which may be suitable for blocking and / or processing by subnetworks as described in the first embodiment.

[0339] In one implementation, the input tensor has at least two components, a first component and a second component; the first subnet divides the first component into a third plurality of blocks, and divides the second component into a fourth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the third plurality of blocks and the fourth plurality of blocks are of different sizes. In principle, the present invention is not limited to any particular number of spatial components. There may be one spatial component (e.g., a grayscale image). However, in this implementation, there are multiple spatial components in the input tensor. The first component and the second component may be color components. Then, the blocks may not only vary from subnet to subnet, but may also vary from component to component processed by the same subnet.

[0340] Thus, in addition or alternatively, the second subnetwork divides the first component into a fifth plurality of blocks and divides the second component into a sixth plurality of blocks, wherein at least two corresponding juxtaposed blocks of the fifth plurality of blocks and the sixth plurality of blocks are of different sizes. In other words, the block divisions of the spatial components of the second input tensor may be different. Further details of a second embodiment are provided below, wherein the block divisions are different for different spatial input components. The second embodiment may be combined with the first embodiment described so far, wherein the block divisions are different for different subnetworks of the neural network.

[0341] Fig.16 The encoding (similar to Fig. 6A and Figure 6B ) further comprises generating the code stream by including the output of the processing of the neural network into the code stream. The neural network may also include entropy decoding. This Fig. 6A, where, after encoding the input image 614 through the encoder neural network (convolutional layers 601 to 604), the output y of the encoder NN ga is quantized (e.g., RDOQ) and arithmetically coded (613) to provide bitstream 1. Similarly, the super prior neural network takes the (bottleneck) feature map y (the output of the encoder NN ga) and processes it through convolutional layers 605 and 606 and two ReLU layers and provides an output z containing the statistics of the input image. Quantization and arithmetic coding (615) are performed to generate bitstream 2. The corresponding bitstreams 1 and 2 are also obtained by Figure 6B , Fig. 9 and Fig. 9A The processing shown in generates .

[0342] For the sizes of the partitions determined as discussed above, the generation of the codestream further comprises including an indication of the sizes of the partitions in the first plurality of partitions and / or an indication of the sizes of the partitions in the second plurality of partitions in the codestream. The codestream may be a portion of a codestream comprising codestream 1 and codestream 2, i.e., a portion of a codestream generated by the entire encoding apparatus (encoding process).

[0343] Further details and implementation examples regarding the processing of components (including color components) and the indication of block sizes are discussed in the second embodiment. Fig. 20 , showing examples of various parameters (eg, the size of a block in the first plurality of blocks and / or the size of a block in the second plurality of blocks), corresponding indications being included in the codestream.

[0344] The encoding process of the input tensor has its decoding correspondence, thereby sharing the functional correspondence in the process. In this exemplary and non-limiting embodiment, a method for decoding a tensor representing image data is provided. The tensor may have a matrix form with width = w, height = h in two spatial dimensions and a third dimension (e.g., depth or number of channels) of size equal to D. The method comprises: a neural network including at least a first subnet and a second subnet processes an input tensor representing the image data. It should be noted that the width and height of the input tensor of the decoder may be different from the width and height of the input tensor processed by the encoder. In addition, as known to those skilled in the art, the first subnet and the second subnet of the decoder may fully or partially (e.g., functionally) perform the inverse function of the first subnet and the second subnet of the encoder. However, the inverse function cannot be interpreted in a strict mathematical way. Instead, the term "inverse" refers to the process used to decode the tensor in order to reconstruct the original image data. It will be appreciated by those skilled in the art that encoding compression and decoding decompression may include further processing that may not be required for decoding and / or encoding. For example, Fig.16The RDOQ shown in is a process for the encoder only. It should also be noted that the terms "first" and "second" subnetworks are merely labels to distinguish the subnetworks of the decoder (and for this purpose, the encoder discussed above).

[0345] Fig.16 An example of a first subnet and / or a second subnet of an encoding branch is shown, including a decoder 1604 and a post-filter 1611. In the method, the processing includes: applying the first subnet to a first tensor, including: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks through the first subnet; after applying the first subnet, applying the second subnet to a second tensor, including: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks through the second subnet. Fig.16 In the example of , the first subnet is the decoder 1604, whose output is used as the input of the second subnet, and the second subnet is the post-filter 1611. Here, the first tensor is the quantized feature tensor in the latent space It is previously decoded from code stream 1 by arithmetic decoder 1606. Accordingly, the second tensor input to the second subnet 1611 for post-filtering is is a feature tensor, such as a feature in a feature map or a feature map in a latent space. Therefore, similar to the processing of the encoder, the type of input tensor (e.g., the first and second tensors) can depend on the processing performed by the previous subnet. Fig.16 In the example of , the subnet is the encoder 1604. Therefore, the term "after" does not limit the decoding process to the output of the first subnet directly being input to the second subnet. Rather, "after" means that the first plurality of blocks and the second plurality of blocks are processed in a certain time sequence, for example, within the same path, which may not be immediate in time.

[0346] Furthermore, the type of input may also depend on the layer (e.g., a layer of a neural network NN or a layer of an untrained network) to which the processed input data may be branched to be used as the input of another (e.g., subsequent) subnetwork. For example, such a layer may be the output of a hidden layer. Similar to the encoding process, the first tensor and the second tensor are divided into blocks in the decoding process, which blocks have been defined above.

[0347] exist Fig.16 In the example, decoder 1604 may be a first subnet having processing 1 to processing N (processing pipelines 1 to N), such as Fig.16 As shown, decoder 1604 takes the input tensor The input tensor is divided into N blocks to (First multiple blocks). Then, the corresponding blocks process the corresponding tensor blocks, i.e., process 1 to process N, and they do not need to interact with each other (e.g., wait for each other during processing). The result of each processing provides N blocks. to (Tensor). After processing the first plurality of blocks, the decoder subnet 1604 can fuse the processed blocks into a first output tensor exist Fig.16 , the first output tensor may be a second tensor used as input by the post-filter 1611. The post-filter 1611 may be a second subnetwork having processing 1 to processing N (processing pipeline 1 to N), similar to the decoder 1604, the post-filter 1611 converts the input tensor Divide into second multiple blocks to These blocks are processed by the corresponding process 1 to process N of the post-filter 1611. Process 1 to process N provide the corresponding blocks to As output, these blocks can be fused into blocks In. Fig.16 In the example, the fusion is the decoded tensor representing the reconstructed image data. Fusion can (but does not necessarily) involve Fig.19 It should be noted that there is no need to perform fusion / combination into tensors or It is conceivable that the second subnet reuses the chunk of the first subnet and merely modifies it (making it smaller by further dividing it, or making it larger by merging multiple chunks into one). Fig.16 In the example of , processes 1 to N may perform processing based on blocks, that is, process i processes block i. Alternatively, process i may process component i of multiple components of the input tensor. In this case, process i divides component i into multiple blocks and processes the blocks separately or in parallel.

[0348] In addition, at least two corresponding juxtaposed tiles in the first plurality of tiles and the second plurality of tiles have different sizes. In other words, the way each subnet is subdivided into tiles may be different, so each subnet may use a different tile size. However, the input of each subnet (i.e., the first tensor and the second tensor) is divided into a grid of tiles of the same size, except for the tiles at the bottom and right image boundaries, which can have smaller sizes.

[0349] Otherwise, the properties and / or characteristics of the first plurality of partitions and the second plurality of partitions used in the decoding process are similar to one of the encoding processes discussed above. Specifically, in the above exemplary implementation, the partitions in the first plurality of partitions that are adjacent in at least one of the spatial dimensions partially overlap; and / or the partitions in the second plurality of partitions that are adjacent in at least one of the spatial dimensions partially overlap. Examples of adjacent partitions and partial overlap are described in Fig.17A , 17B , 18, 19 and 20 are shown.

[0350] In addition, in some exemplary implementations, the first subnet independently processes a block in the first plurality of blocks; and / or the second subnet independently processes a block in the second plurality of blocks. In other words, the processing of the blocks is independent of each other and is therefore parallelizable. In one example, in some implementations, the first subnet processes at least two blocks in the first plurality of blocks in parallel and / or the second subnet processes at least two blocks in the second plurality of blocks in parallel. Fig.16 16 shows parallel processing on the decoder side, where the decoder 1604 processes components (e.g., block and / or spatial components) of the first input tensor. to There is no interaction between the processing of components 1 to N. The result of each processing is the output tensor component to They can be further combined into output tensors The second subnet can be Fig.16 The post-filter 1611 in , which transforms the tensor from the decoder 1604 (first subnet) As input. In this example, the input of post-filter 1611 is directly connected to the output of decoder 1604. Alternatively, Fig.16 There can be further processing between the post-filter and the decoder in . Input tensor Divided into N blocks by the post-filter to The post-filters then process the corresponding blocks independently or in parallel. Fig.16 The parallel processing is shown in Figure 1, because the processing of components 1 to N does not affect each other. The result of the parallel processing of the postfilter is the reconstruction of the blocks to It can be fused into a tensor representing the reconstructed image data middle.

[0351] Furthermore, the partitioning of the first tensor comprises determining the size of the blocks in the first plurality of blocks based on a first predefined condition; and / or the partitioning of the second tensor comprises determining the size of the blocks in the second plurality of blocks based on a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or the presence of motion in the image data, or other features, as already described above with reference to encoding.

[0352] The first subnet and the second subnet for decoding processing may have a configuration similar to that of the subnet for encoding processing. Specifically, the first subnet performs processing through one or more layers including at least one convolutional layer and at least one pooling layer; and / or the second subnet performs processing through one or more layers including at least one convolutional layer and at least one pooling layer. Fig. 6A and Figure 6B The decoder within the NN-based VAE framework is shown, with corresponding convolutional layers 607 to 6012, where the input tensor (feature map) The corresponding size of is then enlarged by 2 (up-sampling). In addition, the first subnet and the second subnet perform corresponding processing as part of image or moving image decompression. Such processing can be performed by Fig. 6A and Figure 6B The VAE encoder shown in the figure provides a feature tensor As input, in order to decompress and reconstruct the output image as the output of the convolution layer 607, representing the reconstructed image data (i.e., the decoded tensor). For example, the first subnet and / or the second subnet performs one of the following steps: image decoding of the convolution subnet; image filtering. As described above, the first subnet and / or the second subnet for decoding processing can be Fig. 9 The decoder 904 processes the feature map tensor In order to generate reconstructed image data that approximates the original image data x like Fig. 9 As shown, the decoder 904 can be part of a VAE encoder-decoder framework, such as Fig. 6A and Figure 6B As shown, the corresponding decoder "gs" processes the feature tensor through a sequence of convolutional layers 610 to 607. (including through the reverse IGDN layer processing) to perform the corresponding image decoding. Image filtering (such as post-filter) is performed in Fig. 6A / Figure 6B and Fig. 9 Not further shown.

[0353] In one implementation, the input tensor is an image or image sequence comprising one or more components, at least one of which is a color component. Alternatively, the input tensor may be a latent space representation of an image, which may be an output of preprocessing (e.g., an output tensor). One or more components are color components and / or depth and / or motion maps and / or other feature maps associated with image samples. The input tensor has at least two components, namely a first component and a second component; the first subnet divides the first component into a third plurality of blocks, and divides the second component into a fourth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the third plurality of blocks and the fourth plurality of blocks have different sizes; and / or the second subnet divides the first component into a fifth plurality of blocks, and divides the second component into a sixth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the fifth plurality of blocks and the sixth plurality of blocks have different sizes.

[0354] The decoding method described above also includes extracting an input tensor from the bitstream for processing by the neural network. The neural network may also include entropy decoding. Fig. 6A and Figure 6B shows the input tensor of the decoder subnet gs decoded from code stream 1 by arithmetic decoding Entropy encoding is performed by the subnetwork hs, where the entropy tensor can be decoded from code stream 2 by arithmetic decoding, where is processed to obtain information about Statistical information of the decoding process of the encoded image data. Figure 6B Further details on entropy decoding are shown, where information about the distribution given by mean μ and variance σ1, σ2 is further input to a Gaussian entropy model 670 and used to As discussed above, the subnet hs( Fig. 6A ) (possibly including a leaky ReLU layer 660) can be Fig. 9 and Fig. 9B . In an exemplary implementation, the second subnetwork performs image post-filtering; one or more parameters of the post-filtering are different for at least two of the second plurality of partitions and are extracted from the bitstream. In addition, the decoding process also includes parsing an indication of the size of a partition in the first plurality of partitions and / or an indication of the size of a partition in the second plurality of partitions from the bitstream.

[0355] In this exemplary and non-limiting embodiment, a computer program stored in a non-transitory medium is provided, the computer program comprising code, when the code is executed on one or more processors, to implement the steps of any encoding and decoding method discussed above. The corresponding flow chart of encoding and encoding processing is shown in FIG. Fig.21 and Fig. 22 As shown. Fig.21 In step 2110, the first subnet processes the first tensor, including dividing the first tensor into a plurality of blocks, and then the first subnet processes the blocks. Fig.21 In the example, the image data (i.e., the input tensor representing the image data) is input to the first subnet shown by the dashed line. This illustrates that the original image data is not necessarily directly input to the first subnet, which may depend on the processing performed by the first subnet relative to the processing order. In other words, the first tensor that is the input to the first subnet comes from the image data. In the first subnet, Fig. 9 In the case of encoder 901 shown, the first tensor may be the input tensor x. In step S2120, the first subnet further processes the first plurality of blocks. After the first subnet processes, in step S2130, the second subnet processes the second tensor, including dividing the second tensor into the second plurality of blocks. Similarly, the second tensor of the second subnet indicated by the dashed line is not necessarily a direct output of the first subnet. Fig. 9 In the implementation example of , the feature tensor y of encoder 901 (first subnet) is directly input to RODQ 902 (second subnet). However, there may be additional processing between encoder 901 and RODQ 902. In step S2140, the second plurality of blocks are further processed. The output of the processing of the second subnet (which may include further processing ( Fig.21 )) may include generating a code stream (eg, Fig. 9 In addition, the process for determining the size of the blocks in the first plurality of blocks and the second plurality of blocks and / or including an indication of the size in the codestream may be a processing step before providing the codestream as an output of the neural network processing. Fig. 22 The flowchart describes the reverse processing steps, starting from the input tensor, which can be the direct or indirect input of the first subnetwork, as shown by the dotted line. Fig. 9 In an exemplary implementation of is used as direct input to the decoder 904, which in this case is the first subnet. In step S2220, the first subnet (e.g., decoder 1604) processes the first plurality of blocks, such as Fig.16After the first subnet processes, in step S2230, the second subnet processes the second tensor by dividing the second tensor into a plurality of blocks, and then in step S2240, the second subnet further processes the second tensor. Fig.16 In an exemplary implementation, the output of the first subnet of decoder 1604 is is the second tensor, corresponding to the second subnet's post-filter 1611 processing the second tensor. Fig.16 In the example of As output. It should be noted that Fig.16 Further processing may be performed between the decoder 1604 and the post-filter 1611, in which case the output of the decoder 1604 It may not be directly input to the post-filter 1611 .

[0356] In addition, as mentioned above, the present invention also provides a device (apparatus) for executing the steps of the above method.

[0357] In this exemplary and non-limiting embodiment, a processing apparatus is provided for encoding an input tensor representing image data. Fig.23A processing device 2300 having corresponding modules for performing encoding processing steps is shown, including a processing circuit 2310. The processing circuit is used to: process an input tensor through a neural network, the neural network comprising at least a first subnet and a second subnet, the first subnet and the second subnet being implemented by an NN processing module (subnet 1 2311) and an NN processing module (subnet 2 2312). The NN processing module (subnet 1 2311) may have a separate partitioning module 1 2313, which partitions the first tensor into a first plurality of blocks in a spatial dimension and processes the first plurality of blocks through the first subnet. Alternatively, the corresponding modules for processing the first input tensor and / or the first plurality of blocks may be implemented in a single module, which may be included in a single circuit or a separate circuit. Similarly, the NN processing module (subnet 2 2312) and the partitioning module 2314 apply the second subnet to the second tensor, including: partitioning the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks through the second subnet. It should be noted that after processing the first tensor, the processing of the second tensor can be achieved by wiring the corresponding modules so that the signals (in terms of their input signals and output signals) are input in a corresponding time (not necessarily immediately in time) order. Alternatively or in addition, the corresponding indication order can be achieved by software configuration modules. Modules 2313 and 2314 can also provide a function of determining the size of the blocks in the first plurality of blocks and / or the second plurality of blocks. The processing device may also have a code stream module 2315, which provides a function of generating a code stream, wherein the output of the neural network processing is included in the code stream. In addition, module 2315 provides a function of including an indication of the size of the blocks in the first plurality of blocks and / or the second plurality of blocks in the code stream.

[0358] In this exemplary and non-limiting embodiment, a processing device is provided for encoding an input tensor representing image data, the processing device comprising: one or more processors; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the encoder to perform a method according to the encoding method described above.

[0359] In this exemplary and non-limiting embodiment, a processing device is provided for decoding a tensor representing image data. Fig.24A processing device 2400 having corresponding modules for performing decoding processing steps is shown, including a processing circuit 2410. The processing circuit is used to: process an input tensor through a neural network, the neural network comprising at least a first subnet and a second subnet, the first subnet and the second subnet being implemented by an NN processing module (subnet 1 2411) and an NN processing module (subnet 2 2412). The NN processing module (subnet 1 2411) may have a separate partitioning module 1 2413, which partitions the first tensor into a first plurality of blocks in a spatial dimension and processes the first plurality of blocks through the first subnet. Alternatively, the corresponding modules for processing the first input tensor and / or the first plurality of blocks may be implemented in a single module, which may be included in a single circuit or a separate circuit. Similarly, the NN processing module (subnet 2 2412) and the partitioning module 2414 apply the second subnet to the second tensor, including: partitioning the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks through the second subnet. Modules 2413 and 2414 may also provide a function of determining the size of a block in the first plurality of blocks and / or the second plurality of blocks. It should also be noted that, after processing the first tensor, the processing of the second tensor may be achieved by wiring corresponding modules, etc., so that the signals (in terms of their input signals and output signals) are input in a corresponding time (not necessarily immediately in time) order. Alternatively or in addition, the corresponding indication sequence may be achieved by a software configuration module. The processing device may also have a parsing module 2415, which provides a function of parsing an indication of the size of a block in the first plurality of blocks and / or the second plurality of blocks from the code stream. Module 2415 may also provide a function of extracting an input tensor from the code stream.

[0360] In this exemplary and non-limiting embodiment, a processing device for decoding a tensor representing image data is provided, the processing device comprising: one or more processors; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the encoder to perform a method according to the decoding method described above.

[0361] The above described and Fig.16 The encoding and decoding processes performed by the VAE-encoder-decoder shown in Figure 1A Therefore, the source device 12 represents the encoding side, providing a decoding system 10. Fig.16 The input tensor x is a compression of the input image data 21. Specifically, Figure 1AThe encoder 20 may include modules for performing encoding processing according to the present invention, such as an encoder 1601, a quantizer or RDOQ 1602, and an arithmetic encoder 1605. The encoder 20 may also include a super-a priori module, such as a super-decoder 1603, a quantizer or RDOQ 1608, and an arithmetic encoder 1609. Similarly, Figure 1A The destination device 14 represents a decoding side, providing decompression of the input tensor representing the image data. Specifically, Figure 1A The decoder 30 may include a module for performing a decoding process according to the present invention, such as Fig.16 The decoder 1604 and the post filter 1611, and the arithmetic decoder 1606. In addition, the decoder 30 may also include a decoder 1604 and a post filter 1611, and an arithmetic decoder 1606. The super prior (arithmetic decoder 1610, super decoder 1607 and arithmetic decoder 1606). In other words, Figure 1A The encoder 20 and decoder 30 may be implemented and configured to include Fig.16 Any module to implement the encoding or decoding process according to the present invention, wherein the plurality of subnetworks process the input tensor divided into the first plurality of blocks and the second plurality of blocks, and then the first plurality of blocks and the second plurality of blocks are processed as described in the first embodiment. Figure 1A The encoder 20 and decoder 30 are shown separately, but they can be Figure 1B In other words, the processing circuit 46 can be implemented by Fig.16 The corresponding circuits of the corresponding modules are used to provide the encoding-decoding processing functions of the present invention.

[0362] Similarly, Figure 2 The video decoding device 200 and the decoding module 270 of the processor 230 can perform the encoding process or decoding process of the present invention. For example, the video decoding device 200 can be a Fig.16 The encoder or decoder of the corresponding module is used to perform the encoding or decoding process as described above.

[0363] Figure 3 The apparatus 300 may be implemented as an encoder and / or decoder having an encoder 1601, a quantizer or RDOQ 1602, a decoder 1604, a post filter 1611, a super encoder 1603 and a super decoder 1607, and an arithmetic encoder 1605, 1609 and an arithmetic decoder 1606, 1610, so as to perform the block processing as discussed according to the first embodiment. For example, Figure 3 The processor 302 may have corresponding circuits to perform encoding and / or decoding processing according to the method described above.

[0364] Figure 4The encoder 20 shown in FIG. Figure 5 The exemplary implementation of the decoder 30 shown in FIG. 1 can also implement the encoding and decoding functions of the present invention. For example, Figure 4 The segmentation unit 452 in the embodiment may divide the first tensor and / or the second tensor into a first plurality of blocks and / or a second plurality of blocks, such as Fig.16 1601. Thus, syntax element 465 may include an indication of the size and position of the partition, as well as an indication of the filter index, etc. Similarly, quantization unit 408 may perform the quantization or RDOQ of RDOQ module 1602, and entropy coding unit 470 may implement the functionality of super prior (i.e., modules 1603, 1605, 1607, 1608, 1609). Accordingly, Figure 5 The entropy decoding unit 504 may perform the decoding by dividing the encoded image data 21 (input tensor) into blocks, while parsing the indication of the block size or position, etc. from the code stream as the syntax element 566. Fig.16 The entropy decoding unit 504 may also implement a super-a priori module (ie, modules 1606, 1607, 1610). Fig.16 The post-filter 1611 can also be performed by the entropy decoding unit 504 or the like. Alternatively, the post-filter 1611 can be used as an additional unit ( Figure 5 ) is implemented within the mode application unit 560.

[0365] Second embodiment

[0366] In the previous exemplary implementation of the first embodiment, the blocks in the first plurality of blocks and / or the second plurality of blocks are processed by the first subnet and the second subnet, respectively, as discussed above. Here, the encoding and decoding processing of components in multiple pipelines is discussed, wherein each pipeline processes one or more images / image components, which can be color planes. It can be seen from the following discussion that the first embodiment and the second embodiment partially share similar or identical processing.

[0367] In this exemplary and non-limiting embodiment, a method for processing an input tensor representing image data is provided. The input tensor may have a matrix form with width=w, height=h, and a third dimension (such as depth or number of channels) of size equal to D. In addition, the input tensor may be processed by a neural network. In the method, multiple components of the input tensor are processed, the components including a first component and a second component in a spatial dimension. The input tensor is an image or an image sequence including one of the multiple components, an image of multiple components or at least one of which is a color component. In one implementation, the first component represents the brightness component of the image data; the second component represents the chrominance component of the image data. For example, the multiple components of the input tensor may be in YUV format, Y is the brightness component, and UV is the chrominance component. The brightness component may be referred to as the primary component, and the chrominance component may be referred to as the secondary component. It should be noted that the terms "primary" and "secondary" are labels used to apply more weight and / or importance to the primary component than to the secondary component. As known to those skilled in the art, even if the brightness component is generally a component of higher importance, there may be application instances where another component different from the brightness is prioritized. The prioritization is usually accompanied by using information about the main component (such as brightness) as auxiliary information to process secondary components (i.e., not as important as brightness). The multiple components can also be in RGB format or any other format suitable for processing components in the corresponding processing pipeline. The method includes: processing the first component, including: dividing the first component into a first plurality of blocks in the spatial dimension and processing the blocks in the first plurality of blocks separately. Processing the second component, including: dividing the second component into a second plurality of blocks in the spatial dimension and processing the blocks in the second plurality of blocks separately. It should be noted that separate does not mean independent, that is, auxiliary information can still be shared to process the first component and / or the second component. Examples of rectangular-shaped blocks are as follows: Fig.14 , Fig.17A , Fig. 17B and Figures 18 to 20As shown. In the method, at least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks have different sizes. In other words, the blocks in the first plurality of blocks and the second plurality of blocks have different sizes. As for the sizes of the blocks in the corresponding plurality of blocks, all blocks in the first plurality of blocks are the same size and / or all blocks in the second plurality of blocks are the same size. Therefore, the corresponding pipelines process blocks of the same size, and the blocks between pipelines are different. Specifically, Y and UV can have different block partitions, which is different from VVC. Blocks are introduced to solve memory problems and may include multiple blocks. Blocks are decoded independently of each other and can be decoded in parallel. A block is part of a block / image / strip, where each block uses its own decoding method and the decoding is sequential, so a coding tree (block structure) is required for better local adaptation. In other words, in this exemplary and non-limiting embodiment, a separate chroma coding tree can be used.

[0368] In one implementation, at least two of the first plurality of blocks are processed independently or in parallel; and / or at least two of the second plurality of blocks are processed independently or in parallel. Fig.25 An example of encoder-decoder processing is shown in which the first component processed in separate pipelines is luminance and the second component is one of chrominance U or V. It should be noted that the chrominance components U and V can be processed jointly as one chrominance component, such as Fig.25 In an exemplary implementation, the processing of the input tensor includes processing as part of image or moving image compression. For example, the processing of the first component and / or the second component includes one of the following steps: image encoding of a neural network; rate distortion optimization quantization (RDOQ); image filtering. The compression processing is Fig.25 2501, 2508 and RDOQ 2502, 2509. Encoders 2501 and 2508 perform image encoding of input image x on luminance Y and chrominance components UV by means of a neural network (NN). Modules RDOQ 2502 and 2509 quantize feature tensor y by optimizing rate distortion and provide quantized feature tensors for its corresponding components. The modules super encoder 2503, 2510 and RDOQ 2504, 2511 form part of the super prior network on the encoding side, which generates statistical information of the image data Furthermore, the code stream is generated by including the output of the processing of the first component and the second component into the code stream. Fig.25 In the example, the quantized feature tensors of the first (luminance) component and the second (chrominance) component are are arithmetically encoded (1605) and included in the code stream Y1 and code stream UV1 respectively. Similarly, the quantization statistics of luma and chroma are arithmetically encoded (1609) and included in the code stream Y2 and code stream UV2 respectively. Fig.25 In the example of , two code streams are generated, supporting decoupling of the encoded image data from the encoding statistics of the image data.

[0369] In another exemplary implementation, the processing of the input tensor includes processing as part of image or moving image decompression. For example, the processing of the first component and / or the second component includes one of the following steps: image decoding by a neural network; image filtering. The decompression processing is performed in Fig.25 The decoders 2506 and 2513 are described in detail by corresponding module decoders 2506, 2513 and post filters 2507, 2514. Decoders 2506 and 2513 decode the quantized feature tensors of the luminance and chrominance components from the corresponding code streams Y1 and UV1. Image decoding of the input image x is performed on the luminance component Y and the chrominance component UV by a neural network (NN). As described above, the super prior also requires a module for decoding, including super decoders 2505, 2512, which decode the quantized statistical information of the image data from the code stream Y2 and the code stream UV2 for the luminance component and the chrominance component. This information is input to the arithmetic decoder 1606, the output of which is provided to the decoders 2505 and 2513. In one implementation example, the processing of the first component and / or the second component includes image post-filtering. Fig.25 As shown, the decoder output is input to post filters 2507 and 2514, which perform post filtering of the component images and provide reconstructed brightness and brightness, respectively. and chromaticity Reconstructed image data as output. Fig.25 The functional blocks (modules) shown in FIG. 1 are similar to those of the first embodiment. Fig.16 The same structural arrangement of the corresponding modules of the VAE encoder-decoder shown in .

[0370] exist Fig.25In the example of , the input tensor is YUV, which has Y, U and V as multiple components. Specifically, the first component here is the luma Y, which is input to the encoder 2501 of the luma pipeline. Similarly, as described above, U and V can be collectively regarded as the second component, which is input to the encoder 2508 of the chroma pipeline. It should be noted that different reference numerals have been assigned to the corresponding functional modules (units) of the VAE encoder-decoder of the luma and chroma pipelines because the modules can be configured differently so that processing of corresponding blocks of different sizes of the luma and chroma pipelines can be supported. However, modules (such as encoder 2501 and encoder 2508) can perform the same / similar functions in encoding, RDOQ, decoding, post-filtering, etc. The functions are similar to Fig.16 The functions performed by the corresponding modules of the VAE encoder-decoder are identical and have been described above. In one implementation, the processing of the second component comprises decoding the chrominance components of the image based on a representation of the luma component of the image. Fig.25 As shown in the figure, the luminance component is the main component and is therefore considered to be more important than the chrominance component UV. As shown in the figure, in the decoding process of luminance and chrominance, the luminance feature tensor Obtained from the luminance code stream Y1 by arithmetic decoding (1606), the code stream Y1 is used as input to the decoder 2513 for decoding the UV component.

[0371] Similar to the first embodiment, the blocks in the first plurality of blocks adjacent in at least one dimension in the spatial dimensions partially overlap; and / or the blocks in the second plurality of blocks adjacent in at least one dimension in the spatial dimensions partially overlap. Fig.17A , Fig. 17B and Figures 18 to 20An example of adjacent partitions that may partially overlap is discussed. In an exemplary implementation, the partitioning of the first component includes determining the size of the partitions in the first plurality of partitions based on a first predefined condition; and / or the partitioning of the second component includes determining the size of the partitions in the second plurality of partitions based on a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data. The first and / or predefined conditions may be memory resources of a decoder and / or encoder. When the available memory is less than a predefined value of the memory resources, the determined partition size may be less than the partition size when the available memory is equal to or greater than the predefined value. Alternatively or in addition, when the presence of motion is significant, the partition size may be determined to be smaller than when the presence of motion is less significant. Whether the motion is significant (i.e., fast motion and / or fast / frequent motion changes) may be determined by the corresponding motion vector based on changes in its amplitude and direction, and compared with corresponding predefined values ​​(thresholds) of amplitude and / or direction and / or frequency. In addition, another predefined condition may be a region of interest (ROI), wherein the tile size may be determined to be smaller when the ROI size is smaller than the ROI reference size. For example, the ROI in the image data may be based on detected objects (e.g., vehicles, bicycles, motorcycles, pedestrians, animals, etc.), which may have different sizes, move quickly or slowly, and / or change their direction of motion quickly or slowly and / or multiple times (i.e., more frequently relative to a predefined frequency value). The ROI may also be based on the smoothness of the region within the image data. For example, for regions with greater smoothness (e.g., measured relative to a predefined smoothness value), the tile size may be larger, while a smaller tile size may be used for less smooth regions, i.e., regions with very obvious spatial variations, etc. In other words, the tile size may be determined based on the texture level of the region within the image data. In addition, a larger tile size may be determined for a primary component (e.g., brightness). Accordingly, a smaller tile size may be determined for a secondary component (e.g., chrominance). Therefore, the tile size (including a scene-specific tile size) may be optimized to accommodate hardware resources associated with the content of the image data. In another implementation, determining the size of the blocks in the second plurality of blocks includes scaling the blocks in the first plurality of blocks. In other words, the block size of the first component is used as a reference so that the block size of the second component is derived by scaling the block size of the first component. Scaling may include enlarging (expanding) so that the scaled blocks of the second plurality of blocks are larger than the size of the first plurality of blocks. Alternatively, scaling may include shrinking (shrinking) so that the scaled blocks of the second plurality of blocks are smaller than the size of the first plurality of blocks. Whether to use expansion or shrinking may depend on the importance of the first component and / or the second component.If at least one of the multiple components being processed has at least one color component, the use of expansion or contraction may also depend on a specific color (eg, a color component indicating "danger / warning", etc.).

[0372] In the following, information about block sizes may be indicated and decoded from a codestream, and various aspects of the indication and decoding of block sizes and / or other suitable parameters are discussed. In an exemplary implementation, an indication of the determined size of a block in the first plurality of blocks and / or the second plurality of blocks is encoded into the codestream. In addition, the indication also includes the position of the block in the first plurality of blocks and / or the second plurality of blocks. For example, the first component is a luminance component, and the codestream includes the indication of the size of the block in the first plurality of blocks; the second component is a chrominance component, and the codestream includes an indication of a scaling factor, wherein the scaling factor is related to the size of the block in the first plurality of blocks and the size of the block in the second plurality of blocks.

[0373] The following table shows examples of block sizes and / or positions of blocks indicating components, and extracts the code syntax for implementing the indication. The syntax table is an example and is not limited to this specific syntax. In particular, these syntax examples are for illustration, as they can be used to indicate corresponding indications of block sizes and / or block positions, etc., of luminance components and / or chrominance components included in the code stream. The first syntax table refers to the automatic decoder, while the second table refers to the post-filter. As shown in Tables 1 and 2, the same / similar indications can be used and included in the code stream, but can be included separately for the automatic decoder (Table 1) and the post-filter (Table 2). In one implementation, for at least two of the first plurality of blocks, one or more parameters of the post-filter are different and are extracted from the code stream; for at least two of the second plurality of blocks, one or more parameters of the post-filter are different and are extracted from the code stream. In other words, different block-based post-filtering parameters can be indicated in the code stream, so that post-filtering of each first plurality of blocks and / or second plurality of blocks can be performed with high precision.

[0374] In the following, for Fig.25 Some examples of the modules of the encoder-decoder VAE and the way these modules are used are discussed with reference to Tables 1 and 2 respectively.

[0375]

[0376]

[0377]

[0378]

[0379]

[0380] A.Decoder

[0381] In this implementation example, luma and chroma are decoded separately, as referenced above. Fig.25 As explained, Fig.25 Separate pipelines are shown for decoding the luma and chroma components. However, as shown by the dashed line pointing from the arithmetic decoder 1606 of the luma pipeline to the decoder 2513 of the chroma pipeline, decoding of the chroma components requires both chroma and luma latent spaces (ie, CCS).

[0382] Export of block diagram:

[0383] Several methods for obtaining block maps are shown in Table 1.

[0384] 1. Only the main component is explicitly indicated for the partition map. Other components use the same partition map (for YUV420 and CCS, the main component is luma / Y, and another component is chroma / UV). Using the same partition map includes using the same partition map in a literal way.

[0385] In one implementation, the same tile map (e.g., of the luma component) is used to derive the tile map of the secondary component (e.g., chroma UV) by scaling the tile size of the luma component. For this purpose, an indication of the scaling factor is included in the codestream. In Table 1, such a scaling factor is "chroma_tile_scaling_factor".

[0386] 2. Explicitly indicate the partitioning map for each component of the image.

[0387] In the indication example of Table 1, whether to indicate the tile of each component depends on the indications "tiles_enabled_for_chroma" and "tiles_enabled_for_luma." The indication may be a simple flag "0" or "1" indicating off or on.

[0388] When a tiled graph is explicitly indicated, it can be done in one of the following ways:

[0389] 1. Use a regular tile grid with the same size (except for tiles at the bottom and right borders). Indicate offset and size values ​​from which the tile map can be derived (see below). In this case, the tiles of the first component (luminance) have the same size, where the tile size is defined in terms of width and height when the tiles are rectangular. The corresponding indications for the tile sizes are "tile_width_luma" and "tile_height_luma" in Table 1. Similar indications can be used for the same chroma tile size, except that the chroma tile size is different from the luma tile size.

[0390] 2. Use a regular tile grid with the same size (except for tiles at the bottom and right borders). Use offset and size values, but do not indicate them directly. Instead, the size and offset values ​​are derived from the already decoded level definition. The tile map can be derived from the size and offset values ​​(see below).

[0391] 3. Use an arbitrary grid of blocks. First, indicate the number of blocks, then for each block, indicate its position and size (overlap is implicitly included in this indication). In Table 1, the arbitrary grid is reflected in that for a predefined number of luminance blocks

[0392] 'num_luma_tiles', each tile can have an individual size in terms of 'tile_width[i]' and 'tile_height[i]'.

[0393] In addition, the indication of the tile position (here the start position) is also included in the bitstream, and the indication of the tile position is "tile_start_x[i]"

[0394] and "tile_start_y[i]". In this example, the tiles are in 2D xy space.

[0395] If chroma indication is enabled and chroma components are processed independently from luma components (Table 1: "!use_dependent_chroma_tiles",

[0396] That is, the chrominance block does not depend on the luma block), a similar indication is used, wherein the corresponding indication is included in the bitstream.

[0397] A further indication of the overlap of the partitions of the luma component and / or the chroma components may be included in the codestream and thus indicated to the decoder.

[0398] If the tile map is indicated by the tile size (tile width equals height) and the overlap value (signal space size / coordinate), then the N overlap regions are derived as follows:

[0399]

[0400] Decoding of the luminance component:

[0401] There are N overlapping regions (tiles) in the image. Each tile (im_tile i ) in the latent space (lat_tile i ) has a corresponding block. In addition, im_tile i Overwrite only lat_tile i A subset of the total receptive field of . For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows. Here, alignment_size is a power of 2, which depends on the number of downsampling layers in the subnet:

[0402]

[0403]

[0404] Using lat_tile, the corresponding region of the latent space is extracted and processed by the decoder subnet. The decoder subnet also has im_tile as an auxiliary input, which is required for correct padding (specifically at image boundaries), where the size of the tile may not be a multiple of alignment_size. The output of the decoder is assigned to the region of the image specified by im_tile. This step may include a cropping operation to remove parts of the reconstruction that overlap with another tile reconstruction.

[0405] refer to Fig.17A and Fig.18 The necessity of cropping regions R1 to R4 of processed tiles L1 to L4 is discussed to ensure seamless fusion.

[0406] Decoding of chroma components:

[0407] There are N overlapping regions (blocks) in the image. Fig.17A and Fig.18 In the example shown, there are four tiles that partially overlap. Each tile (im_tile i ) in the latent space (lat_tile i ) has a corresponding block. In addition, im_tile i Overwrite only lat_tile i The N overlapping regions are derived from the indicated parameters of the patch size and patch overlap (patch width equals patch height). The size is indicated in signal space units (i.e., not latent space). The patch positions and sizes in signal space are then derived as follows:

[0408]

[0409] For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows. Here, alignment_size is a power of 2, which depends on the number of downsampling layers in the subnet:

[0410]

[0411] Using lat_tile, extract the corresponding area of ​​the chrominance latent space lat_UV. In addition, determine the corresponding area of ​​the luma latent space lat_Y. For the example of YUV420, one possible approach is to downsample the luma latent space by a factor of 2 and then extract lat_Y using the same lat_tile as for chrominance. Both lat_Y and lat_UV are then processed by the decoder subnet. The decoder subnet also has im_tile as an auxiliary input, which is required for correct padding (specifically at image boundaries), where the size of the tile may not be a multiple of alignment_size. The output of the decoder is assigned to the area of ​​the image specified by im_tile. This step may include a cropping operation to remove parts of the reconstruction that overlap with another tile reconstruction.

[0412] B. Post-processing filter:

[0413] Export of block diagram:

[0414] Several methods for obtaining block maps are shown in Table 2:

[0415] 1. Use the same blocking map as the decoder. For the example of YUV420, this can be implemented such that: for the case of filtering the Y component, the same blocking as for luma / Y is used in the decoder, while for U and V, the same blocking as the decoder uses for chroma (UV) is used. This behavior can be indicated with a single flag.

[0416] 2. The partitioning map is explicitly indicated only for the main component (e.g., the luma component). The same partitioning map is used for other components (e.g., the chroma components). As shown in Table 2, the indication of the partitioning map (i.e., the partitioning size, position, and overlap) can be done in a similar manner as in Table 1 for the automatic decoder.

[0417] 3. Explicitly indicate the partition map for each component of the picture. Similarly, the indication of the partition map for each component can be accomplished by using the same indication included in the codestream in Table 1.

[0418] It should be noted that when the filter selection uses multi-scale structural similarity (MS-SSIM) as the distortion criterion (encoder), the blocks should be large enough for MS-SSIM (including some downsampling steps). If the blocks at the bottom and right image boundaries are too small, they are enlarged by taking regions from their corresponding neighbors (i.e., adjacent regions). Since the decoder must perform the same processing, this will be normative. When using MSE and / or peak signal noise ratio (PSNR) as the distortion criterion, there is no problem that the block size may be too small.

[0419] When explicitly indicating a tiled graph this can be done in one of the following ways:

[0420] 1. Use a regular tile grid with equal sizes (except for tiles at the bottom and right borders). Indicate offset and size values ​​from which the tile map can be derived (see discussion below).

[0421] 2. Use a regular tile grid with the same size (except for tiles at the bottom and right borders). Use offset and size values, but do not indicate them directly. Instead, the size and offset values ​​are derived from the already decoded level definition. The tile map can be derived from the size and offset values ​​(see discussion below).

[0422] 3. Using an arbitrary grid of tiles: First, indicate the number of tiles, then for each tile, indicate its position and size (overlap is implicitly included in this indication).

[0423] Export of the filters used for each block:

[0424] In tile I In a specific image component, use the filter indexed by filter I In other words, block-based filters can be specified based on a parameter (i.e., filter index). I The selection can be indicated in one of the following ways, as shown in Table 2:

[0425] 1. All blocks of the image component use the same model / filter. This can be done with a single flag and filter index filter I Indicated together. In Table 2, such a flag is "same_model_for_all_luma" and the filter index is "model_idx_luma".

[0426] 2. The model of the filter is indicated for each block of image components.

[0427] a. Indicates the default filter index. In Table 2, the default index is "default_model_idx_luma". For each tile I (The number of blocks is already known from the block map), a flag is indicated to indicate whether the filter index is different from the default value. In Table 2, this flag for each block is "use_default_idx[i]", and the index "i" marks the corresponding block in the first or second plurality of blocks. If different, the filter index of the block is indicated. I In Table 2, the corresponding filter index is "model_idx_luma[i]". In other words, for at least two partitions of the first plurality of partitions, one or more parameters of the filter are different. A similar situation applies to one or more parameters regarding the second plurality of partitions.

[0428] b. For each tile I (The number of blocks is already known from the block map), indicating the filter index of the block I In Table 2, the corresponding filter index is “model_idx_luma[i]”.

[0429] The filter index for each component may also be explicitly indicated. Alternatively, the filter index for a component may be derived from those filter indices indicated for another component (eg, the luma component).

[0430] When the selected model of the filter is indicated, the post-filter can use the same block map as the decoder uses. This can reduce the overhead of indicating the block size.

[0431] Component filtering:

[0432] There are N overlapping regions (tiles) in the image. Each tile is processed independently (possibly in parallel). I , whose position, size, overlap and filter index used are known from the instructions described above. Reconstructing the region I is extracted from the reconstructed image based on the values ​​of position and size. Then, filtering based on the filter index is performed on this region. The output is cropped using the values ​​of position, size and overlap and assigned to the filtered output image at the location described by the values ​​of position and size.

[0433] C.RDOQ

[0434] In this implementation, RDOQ is applied to the luma component and the chroma component separately, i.e., the latent space of each luma and chroma component is optimized separately. Fig.25The corresponding RDOQ processing of luma Y and chroma UV by RDOQ modules 2502 and 2509 in their separate pipelines is shown. However, decoding of chroma components requires both chroma and luma latent spaces (i.e., CCS). Therefore, the processing of RDOQ 2502 is performed on luma first, thereby obtaining a new, optimized luma latent space This luma latent space is then kept fixed and used as an additional input (i.e. as side information) to the chroma latent space optimization. Fig.25 This is shown by the dashed line.

[0435] If the same partition map is used for luma and chroma during the RDOQ process, the optimization of the chroma component does not have to wait until the luma RDOQ optimization is completed for all luma partitions. Instead, once the optimization of a specific luma partition is completed, it can be directly used as input to optimize the corresponding chroma latent partition.

[0436] Export of block diagram:

[0437] Since RDOQ is an encoder-only process, the used partitioning does not have to be indicated. In this implementation, a regular partitioning grid is used for the luma and chroma components. Each grid is described by an offset and size value (encoder parameters). Using this information, a partitioning map of the component can be derived (see below).

[0438] Optimization of the brightness component:

[0439] RDOQ iteratively optimizes the cost of a chunk as follows:

[0440] cost=R+λD

[0441] Here, R is an estimate of the number of bits required to encode the tile, and D is a measure of the distortion of the (tile's) reconstruction compared to the original tile (e.g., peak-signal noise ratio (PSNR), multi-scale structural similarity (MS-SSIM), ...). λ is a parameter set according to the operating point of the encoder. im_tile i The R and D are obtained as follows:

[0442] There are N overlapping regions (tiles) in the image. Each tile (im_tile i ) in the latent space (lat_tile i ) has a corresponding block. In addition, im_tile i Overwrite only lat_tile iA subset of the total receptive field of . For each im_tile in the signal space, the corresponding matching lat_tile in the latent space is derived as follows. Here, alignment_size is a power of 2, which depends on the number of downsampling layers in the subnet:

[0443]

[0444] Using lat_tile, the corresponding region of the latent space is extracted and processed by the decoder subnet to obtain a reconstructed patch. The decoder subnet also has im_tile as an auxiliary input, which is necessary for correct padding (specifically at image boundaries), where the size of the patch may not be a multiple of alignment_size. Using im_tile, the corresponding region is extracted from the original image. Then, D can be calculated as a function of the original patch and the reconstructed patch (such as peak-signal noise ratio (PSNR), multi-scale structural similarity (MS-SSIM)...). R is obtained by calling the encoding function of the extracted latent patch and measuring / estimating the number of bits required to encode it. The RDOQ process is iterative, that is, the process is repeated for a certain number of iterations (usually 10 to 30 iterations).

[0445] Optimization of chroma components:

[0446] RDOQ iteratively optimizes the cost of a chunk as follows:

[0447] cost=R+λD

[0448] Here, R is an estimate of the number of bits required to encode the partition, and D is a measure of the distortion of the (partitioned) reconstruction compared to the original partition (e.g., peak-signal noise ratio (PSNR), multi-scale structural similarity (MS-SSIM), ...). λ is a parameter set according to the operating point of the encoder. In the provided implementation, chrominance is conditionally decoded according to luma. Therefore, here, R also includes the number of bits required to encode the corresponding luma partition:

[0449] R=R luma +R chroma

[0450] R luma Remains constant during chromaticity RDOQ optimization.

[0451] im_tilei R chroma and D are obtained as follows:

[0452] There are N overlapping regions (tiles) in the image. Each tile (im_tile i ) in the latent space (lat_tile i ) has a corresponding block. In addition, im_tile i Overwrite only lat_tile i A subset of the total receptive field of . For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows. Here, alignment_size is a power of 2, which depends on the number of downsampling layers in the subnet:

[0453]

[0454]

[0455] Using lat_tile, extract the corresponding area of ​​the chroma latent space lat_UV. In addition, determine the corresponding area of ​​the luma latent space lat_Y. For the example of YUV420, one possible approach is to downsample the luma latent space by a factor of 2 and then extract lat_Y using the same lat_tile as for chroma. The decoder subnet then processes lat_Y and lat_UV to obtain the reconstructed chroma block. The decoder subnet also has im_tile as an auxiliary input, which is required for correct padding (particularly at image boundaries), where the size of the block may not be a multiple of alignment_size. Using im_tile, extract the corresponding area from the original image. D can then be calculated as a function of the original chroma block and the reconstructed chroma block (e.g. peak-signal noise ratio (PSNR), multi-scale structural similarity (MS-SSIM)...). R chroma is obtained by calling the encoding function on the extracted chroma latent block and measuring / estimating the number of bits required to encode it.

[0456] The RDOQ process is iterative, that is, the process is repeated for a certain number of iterations (usually 10 to 30 times).

[0457] In this exemplary and non-limiting embodiment, a computer program stored in a non-transitory medium is provided, the computer program comprising code, when the code is executed on one or more processors, for the steps of the method for processing an input tensor representing image data discussed in the second embodiment. The corresponding flowchart is as follows Fig.26 As shown. In step S2610, a first component of the input tensor is processed, including dividing the first component into a first plurality of blocks. Similarly, in step S2630, a second component of the input tensor is processed, including dividing the second component into a second plurality of blocks. Then, in steps S2620 and S2640, each of the corresponding first plurality of blocks and second plurality of blocks is processed respectively. Fig.26 As shown in the flowchart of , steps S2610 and S2620 are performed separately from steps S2630 and S2640, reflecting the processing of the first component and the second component in two separate pipelines, as shown in FIG. Fig.25 As shown in the case of the luma Y component (first component) and the chroma component UV (second component). The processing steps S2610 and / or S2630 of dividing the first tensor and the second tensor into the first plurality of blocks and the second plurality of blocks can be part of the encoding process of the encoder 2501 and / or the encoder 2508. Further processing of the first plurality of blocks and the second plurality of blocks in a separate manner may include any one of encoding, quantization RDOQ, decoding and post-filtering, as shown by Fig.25 At the end of the separate processing of the first plurality of blocks and the second plurality of blocks, a reconstructed first component and a reconstructed second component are provided. The reconstructed component may be a luminance component and chrominance components The reconstructed image data, such as Fig.25 As shown. Fig.26 In the processing of , the dashed horizontal arrows indicate that the processing of the first plurality of partitions and the second plurality of partitions may affect each other, because, for example, auxiliary information from the processing of the first plurality of partitions / the second plurality of partitions may be used in the processing of the second plurality of partitions / the first plurality of partitions. Fig.25 As explained, the processing of the UV chrominance component in the chrominance pipeline can use the information of the luma component processed by the luma pipeline as auxiliary information.

[0458] In addition, as described above, the present invention also provides a device (apparatus) for executing the steps of the method described for the second embodiment.

[0459] In this exemplary and non-limiting embodiment, an apparatus for processing an input tensor representing image data is provided. Fig. 27 An apparatus 2700 is shown including a processing circuit 2710 having corresponding modules to perform Fig.26Modules 2711 and 2712 are used to perform processing of the first component and the second component, including processing of the first plurality of blocks and the second plurality of blocks. In addition, two modules 2713 and 2714 are used to divide the first component and the second component of the input tensor into the first plurality of blocks and the second plurality of blocks in the spatial dimension, respectively. It should be noted that although Fig. 27 Separate modules 2711 and 2712 are shown, and modules 2713 and 2714 are shown separately, but the modules can be combined into one module, but are used to process the first component and the second component separately, including separately processing the first plurality of blocks and the second plurality of blocks to achieve pipeline-based processing of the corresponding components. Specifically, modules 2711 and 2712 can provide corresponding pipelines Fig.25 The functions of the various modules shown in , including encoders 2501 and 2508, quantizers RDOQ 2502 and 2509, decoders 2506 and 2513, and post-filters 2507 and 2514. In addition, the functions of the encoder-decoder super-prior can be performed by the corresponding modules 2711 and 2712 of the corresponding pipeline. Fig. 27 , module 2715 performs processing including generating code stream Y / UV 1 and code stream Y / UV 2, wherein indications of block sizes of the first plurality of blocks and / or the second plurality of blocks may also be included in the corresponding code streams. Parsing module 2716 parses code stream Y / UV 1 and code stream Y / UV 2, including extracting indications of block sizes and / or positions from code stream Y / UV 1.

[0460] In this exemplary and non-limiting embodiment, a device for processing an input tensor representing image data is provided, the device comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the device to perform the method discussed in the second embodiment.

[0461] More details about the instructions

[0462] As discussed above, indications of the sizes of the tiles in the first plurality of tiles and / or the second plurality of tiles are included in the codestream Y / UV 1. This also applies to indications of tile positions and / or tile overlaps, and / or scaling and / or filter indices, etc. Alternatively, all or part of the indications may be included in the side information. The side information may then be included in the codestream 1, and the decoder parses the side information from the codestream 1 to determine the tile sizes, positions, etc. required for the decoding process (decompression). For example, indications related to tiles (e.g., tile sizes, positions, and / or overlaps) may be included in the first side information. Accordingly, indications related to scaling may be included in the second side information, and indications about filter models (e.g., filter indices, etc.) may be included in the third side information. In other words, indications may be grouped and included in group-specific side information (e.g., first to third side information). Therefore, the group here refers to a "tile" group, a "filter" group, etc.

[0463] In the following, reference is made to some examples of instructions on the decoder side. Fig. 20 , provides further details on indicating the instructions for the example of block segmentation. It should be noted that the same applies to the encoder side. Fig. 20 Various parameters that can be included in and parsed from the codestream are shown, such as the size of regions Li, Ri, and overlap regions, etc. Region Li refers to a block of the first tensor and / or the second tensor being processed (e.g., Fig.25 The region Ri refers to the corresponding block generated as output after processing the block Li. For example, the block Ri can be Fig.25 The chunks of output y after processing the input tensor x.

[0464] For example, the side information includes an indication of one or more of the following:

[0465] – the number of said input subsets,

[0466] – the size of the input set, i.e. the number of blocks (e.g. of luma and / or chroma components),

[0467] - the size of each of the two or more input subsets (h1, w1), i.e. the size of the blocks to be processed,

[0468] – the size (H, W) of the reconstructed image (R),

[0469] - the size of each of the two or more output subsets (H1, W1), i.e. the size of the blocks after processing,

[0470] - the amount of overlap between the two or more input subsets (L1, L2), i.e. the overlap between the blocks to be processed,

[0471] – The amount of overlap between two or more output subsets (R1, R2), ie the overlap between the blocks after processing.

[0472] Thus, various parameters may be indicated in the side information in a flexible manner. Thus, the indication overhead may be adjusted based on which of the above parameters are indicated in the side information, while other parameters are derived based on those indicated. Each of the two or more input subsets may be of different size. Alternatively, the input subsets may have a common size.

[0473] In one example, the position and / or amount of the sample to be cropped is determined based on the size of the input subset indicated in the side information and the neural network tuning parameter of the neural network, wherein the neural network tuning parameter represents the relationship between the size of the input of the network and the size of the output of the network. Therefore, the position and / or the cropping amount can be more accurately determined by considering the size of the input subset and the characteristics of the neural network (i.e., its tuning parameters). Therefore, the cropping amount and / or position can be adapted to the properties of the neural network, which further improves the quality of the reconstructed image data.

[0474] The adjustment parameter may be an additive term that is subtracted from the input size to obtain the output size. In other words, the output size of the output subset is related to its corresponding input subset by only an integer. Alternatively, the adjustment parameter may be a ratio. In this case, the size of the output subset is related to the size of the input subset, and the size of the output subset is obtained by multiplying the size of the input subset by the ratio.

[0475] As described above, the determination of Li, Ri and the cropping amount can be obtained from the code stream according to a predefined rule or a combination of the two.

[0476] • An indication indicating the amount of cropping may be included in (and parsed from) the codestream. In this case, the amount of cropping (ie the amount of overlap) is included in the side information.

[0477] The cropping amount can be a fixed amount. For example, the amount can be predefined by a standard, or it can be fixed once the relationship between the size (dimension) of the input and output is known.

[0478] • The amount of cropping can relate to cropping in the horizontal direction, the vertical direction, or both directions.

[0479] You can perform cropping based on pre-configured rules. After obtaining the cropping amount, the cropping rules can be as follows:

[0480] ● Depending on the position of Ri in the output space (top left corner, center, etc.). If one side of Ri does not coincide with the output boundary, clipping can be applied on that side (top, left, bottom or right).

[0481] The size and / or coordinates of the Li (ie, partition) may be included in the codestream. Alternatively, the number of partitions may be indicated in the codestream, and the size of each Li may be calculated based on the input size and the number of partitions.

[0482] The overlap amount of each input subset Li can be:

[0483] - An indication indicating the amount of overlap may be included in the codestream (and parsed or derived from the codestream).

[0484] The overlap amount can be a fixed quantity. As mentioned above, in this context, "fixed" means that the quantity is known, such as by convention such as a standard or proprietary configuration, or pre-configured as part of the encoding parameters or neural network parameters.

[0485] • The amount of overlap can be related to cropping in the horizontal direction, the vertical direction, or both directions.

[0486] ●The overlap amount can be calculated based on the cut amount.

[0487] Some numerical examples are provided below to illustrate which parameters can be indicated by (and parsed from) the side information included in the codestream, and how these indicated parameters can then be used to derive the remaining parameters. These embodiments are merely exemplary and do not limit the present invention.

[0488] For example, the following information related to the indication of Li may be included in the code stream:

[0489] • Number of segments along the vertical axis = 2. This corresponds to Fig. 20 An example in which the middle space L is divided vertically into 2 parts.

[0490] • Number of segments along the horizontal axis = 2. This corresponds to Fig. 20 An example in which the middle space L is divided horizontally into 2 parts.

[0491] • Equal size split part flag = True. This is illustrated in the figure by showing L1, L2, L3 and L4 as having the same size.

[0492] • The size of the input space L (wL=200, hL=200). In these examples, the width w and height h are measured in units of number of samples.

[0493] • Overlap = 10. In this example, the overlap is measured in number of samples.

[0494] According to the above information, since the information indicates that the overlap amount is 10 and the sizes of the divided parts are equal, the sizes of the divided parts can be obtained as w=(200 / 2+10)=110 and h=(200 / 2+10)=110.

[0495] Furthermore, since the size of the segment is (110, 110) and the number of segments along each axis is 2, the coordinates of the upper left corner of the segment can be obtained as:

[0496] ● The upper left corner coordinates associated with the first segment, L1 (x=0, y=0),

[0497] The coordinates of the upper left corner associated with the second segment, L2 (x=90, y=0),

[0498] ● The upper left corner coordinates associated with the third segment, L3 (x=0, y=90),

[0499] ●The upper left corner coordinates associated with the fourth segment, L4 (x=90, y=90).

[0500] The following examples illustrate different options for indicating all or part of the above parameters, which are Fig. 20 Shown in. Fig. 20 It is shown how the various parameters related to the input subset Li, the output subset Ri, the input image and the reconstructed image are linked.

[0501] It should be noted that the above signal parameters do not limit the present invention. As described below, there are many ways to indicate information, based on which the size of the input and output spaces and subspaces, clipping or padding can be derived. Some further examples are described below.

[0502] Example of first instruction:

[0503] Fig. 20 A first example is shown, in which the following information is included in the codestream:

[0504] ●The number of regions in the latent space (corresponding to the input space on the decoder side), which is equal to 4.

[0505] The total size (height and width) of the latent space, which is equal to (h, w). (referred to as wL, hL above).

[0506] ●h1 and w1, used to derive the size of the region (here is the size of the four Li), that is, the input subset.

[0507] ●The total size of the reconstruction output R (H, W).

[0508] ●H1 and W1. H1 and W1 represent the size of the output subset.

[0509] Accordingly, the following information is predefined or predetermined:

[0510] • The amount of overlap X between regions Ri. X is also used to determine the amount of cropping, for example.

[0511] ●The amount of overlap y between regions Li.

[0512] According to the information included in the code stream and the predetermined information, the size of Li and Ri can be determined as follows:

[0513] L1=(h1+y,w1+y)

[0514] L2=((h–h1)+y,w1+y)

[0515] L3=(h1+y,(w–w1)+y)

[0516] ●L4=((h–h1)+y, (w–w1)+y)

[0517] R1=(H1+X,W1+X)

[0518] R2=((H–H1)+X,W1+X)

[0519] R3=(H1+X,(W–W1)+X)

[0520] ●R4=((H–H1)+X, (W–W1)+X).

[0521] As can be seen from the first indicative example, the size (h1, w1) of the input subset L1 is used to derive the corresponding sizes of all remaining input subsets L2 to L4. This is possible because the input subsets L1 to L4 use the same overlap amount y, as Fig. 20 In this case, only a few parameters need to be indicated. Similar parameters also apply to the output subsets R1 to R4, where only an indication of the size (H1, W1) of the output subset R1 is required to derive the size of the output subsets R2 to R4.

[0522] In the above, h1 and w1, H1 and W1 are the coordinates of the middle of the input space and the output space, respectively. Therefore, in this first indication example, the input space and the output space are divided into 4 parts using a single coordinate (h1, w1) and (H1, W1), respectively. Alternatively, the size of more than one input subset and / or output subset may be indicated.

[0523] In another example, if the structure of the NN that processes Li is known, Ri can be calculated from Li, i.e., if the size of the input is Li, then Ri is the size of the output. In this case, the size (Hi, Wi) of the output subset Ri may not be indicated by the side information. However, in some other implementations, since the size Ri may sometimes not be determined before the NN operation is actually performed, it may be desirable (as in this case) to indicate the size Ri in the codestream.

[0524] Second instruction example:

[0525] In the second example indicated, H1 and W1 are determined according to the formula together with h1 and w1. For example, the formula can be changed to:

[0526] H1 = (h1 + y) * scalar – X

[0527] W1 = (w1 + y) * scalar – X

[0528] Wherein, a scalar is a positive number. A scalar is related to an adjustment ratio of an encoder and / or decoder network. For example, for a decoder, a scalar may be an integer such as 16, and for an encoder, a scalar may be a fraction such as 1 / 16. Therefore, in the second indication example, H1 and W1 are not indicated in the bitstream, but are derived from the indicated size of the corresponding input subset L1. In addition, a scalar is an example of an adjustment parameter.

[0529] Example of third instruction:

[0530] In the third indication example, the overlap amount y between the regions Li is not predetermined but indicated in the code stream. Then, according to the cropping amount y of the input subset, the cropping amount X of the output subset is determined according to the following formula:

[0531] X = y * scalar

[0532] Wherein, the scalar is a positive number. The scalar is related to the adjustment ratio of the encoder and / or decoder network. For example, for the decoder, the scalar can be an integer such as 16, and for the encoder, the scalar can be a fraction such as 1 / 16.

[0533] It should be noted that the present invention is not limited to a specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0534] The present invention can be implemented by hardware (HW) and / or software (SW). In addition, the implementation based on HW can be combined with the implementation based on SW.

[0535] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0536] The above described and Fig.25 The encoding and decoding processes performed by the VAE-encoder-decoder shown in Figure 1A Therefore, the source device 12 represents the encoding side, providing a decoding system 10. Fig.25 The compression of the input image data 21 of the input tensor x, which may be the corresponding components Y and UV respectively. Specifically, Figure 1A The encoder 20 may include modules for processing (eg, compressing and / or decompressing) according to the present invention for independently processing multiple components. For example, Figure 1A The encoder 20 may include an encoder 2501 for processing the luminance component Y, a quantizer or RDOQ 2502, and an arithmetic encoder 1605. The encoder 20 may also include a super-a priori module, such as a super-decoder 2503, a quantizer or RDOQ 2504, and an arithmetic encoder 1609. In addition, Figure 1A The encoder 20 may include an encoder 2508 for processing a chrominance component UV, a quantizer or RDOQ 2509, and an arithmetic encoder 1605. The encoder 20 may also include a super-a priori module, such as a super-decoder 2510, a quantizer or RDOQ 2511, and an arithmetic encoder 1609.

[0537] Similarly, Figure 1A The destination device 14 represents a decoding side, providing decompression of the input tensor representing the image data. Specifically, Figure 1A The decoder 30 may include a module for decompression processing of the luminance component Y, such as Fig.25 The decoder 2506 and the post filter 2507, and the arithmetic decoder 1606. In addition, the decoder 30 may also include a decoder for decoding The decoder 30 may also include a decoder for decoding. The super prior of, for example, arithmetic decoder 1610, super decoder 2505 and arithmetic decoder 1606. In order to process the chrominance component UV, the decoder 30 may include Fig.25 The decoder 2513 and the post filter 2514, and the arithmetic decoder 1606. In addition, the decoder 30 may also include a decoder for decoding chrominance. of the super prior, such as arithmetic decoder 1606, super decoder 2512 and arithmetic decoder 1610. In other words, Figure 1A The encoder 20 and decoder 30 may be implemented and configured to include Fig.25 Any module of to implement the encoding or decoding processing of multiple components in the corresponding pipeline (such as the luminance and chrominance pipeline) according to the present invention, wherein the input tensor has multiple components, which are divided into multiple blocks and processed as described in the second embodiment. Figure 1A The encoder 20 and decoder 30 are shown separately, but they can be Figure 1B In other words, the processing circuit 46 can be implemented by Fig.25 The functionality of the encoding-decoding process of the present invention is provided by circuits of modules of corresponding pipelines or two pipelines (luminance and / or chrominance).

[0538] Similarly, Figure 2 The video decoding device 200 and the decoding module 270 of the processor 230 can perform the processing (compression and decompression) functions of the present invention. For example, the video decoding device 200 can be a Fig.25 The encoder or decoder of the corresponding module is used to perform the encoding or decoding process as described above.

[0539] Figure 3 The apparatus 300 may be implemented as an encoder and / or decoder having encoders 2501, 2508, quantizers or RDOQs 2502, 2509, decoders 2506, 2513, post filters 2507, 2514, super encoders 2503, 2510 and super decoders 2505, 2512, and arithmetic encoders 2505, 2509 and arithmetic decoders 2506, 2510 to perform the processing of the blocks of each component as discussed according to the second embodiment. For example, Figure 3 The processor 302 may have corresponding circuits to perform compression and / or decompression processing according to the method described above.

[0540] Figure 4 The encoder 20 shown in FIG. Figure 5 The exemplary implementation of the decoder 30 shown in FIG. 1 can also implement the encoding and decoding functions of the present invention. For example, Figure 4 The segmentation unit 452 in the embodiment may divide the first tensor and / or the second tensor into a first plurality of blocks and / or a second plurality of blocks of the first component and the second component, respectively. Fig.25 4. Thus, the syntax element 465 may include an indication of the size and position of the partitions, as well as an indication of the filter index, etc. Similarly, the quantization unit 408 may perform Fig.252502 and 2509 of the RDOQ module 2504, and the entropy coding unit 470 can implement the functions of the super prior (i.e., modules 2503, 2505, 2510, 2511, 1605, 1608, 1609). Figure 5 The entropy decoding unit 504 may perform the decoding by dividing the encoded image data 21 (input tensor) into blocks, while parsing the indication of the block size or position, etc. from the code stream as the syntax element 566. Fig.25 The entropy decoding unit 504 can also implement the super-prior modules (ie, modules 2505, 2512, 1610). Fig.25 The post-filtering 2507 and 2514 can also be performed by the entropy decoding unit 504, etc. Alternatively, the post-filters 2507 and 2514 can be used as additional units ( Figure 5 ) is implemented within the mode application unit 560.

[0541] Further embodiments

[0542] According to one aspect of the present invention, a method for encoding an input tensor representing image data is provided, the method comprising the following steps: a neural network including at least a first subnet and a second subnet processes the input tensor, the processing comprising: applying the first subnet to the first tensor, comprising: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks by the first subnet; after applying the first subnet, applying the second subnet to the second tensor, comprising: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks by the second subnet; wherein at least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks have different sizes. Therefore, by changing the block sizes of different subnets, the input tensor representing image data can be encoded more efficiently. In addition, hardware limitations and requirements can also be considered.

[0543] In some exemplary implementations, the blocks in the first plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap; and / or the blocks in the second plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap. Therefore, the quality of the reconstructed image can be improved, especially along the boundaries of the blocks. Therefore, image artifacts can be reduced.

[0544] In another implementation, the first subnet processes a block in the first plurality of blocks independently; and / or the second subnet processes a block in the second plurality of blocks independently. For example, the first subnet processes at least two blocks in the first plurality of blocks in parallel; and / or the second subnet processes at least two blocks in the second plurality of blocks in parallel. Therefore, encoding of the image data can be performed faster.

[0545] According to an implementation example, the partitioning of the first tensor comprises determining the size of the blocks in the first plurality of blocks according to a first predefined condition; and / or the partitioning of the second tensor comprises determining the size of the blocks in the second plurality of blocks according to a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data. Thus, the block size can be adjusted and optimized according to available encoder and / or decoder resources and / or according to image content.

[0546] In one example, the first subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer; and / or the second subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer. Therefore, input tensor data can be processed efficiently because convolutional networks are particularly suitable for processing data in spatial dimensions.

[0547] In another example, the first subnet and the second subnet perform corresponding processing as part of image or moving image compression. For example, the first subnet and / or the second subnet perform one of the following steps: image encoding of the convolutional subnet; rate distortion optimization quantization (RDOQ); image filtering. Therefore, the encoding of image data can involve subnet processing at multiple related stages and improve decoding efficiency.

[0548] According to an implementation example, the input tensor is an image or image sequence including one or more components, wherein at least one component is a color component. This can support encoding of color components. In one example, the input tensor has at least two components, namely a first component and a second component; the first subnet divides the first component into a third plurality of blocks, and divides the second component into a fourth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the third plurality of blocks and the fourth plurality of blocks have different sizes; and / or the second subnet divides the first component into a fifth plurality of blocks, and divides the second component into a sixth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the fifth plurality of blocks and the sixth plurality of blocks have different sizes. Therefore, multiple components may be encoded in units of blocks, wherein the block sizes of the components are different, which can further improve decoding efficiency and / or improve hardware implementation.

[0549] Another implementation example includes generating the bitstream by including the output of the processing of the neural network into the bitstream. The implementation also includes an indication of the size of the blocks in the first plurality of blocks and / or an indication of the size of the blocks in the second plurality of blocks in the bitstream. Therefore, by providing the indication, the encoder and the decoder can set the block size in a corresponding and adaptive manner.

[0550] According to one aspect of the present invention, a method is provided for decoding a tensor representing image data, the method comprising the following steps: a neural network including at least a first subnet and a second subnet processes an input tensor representing the image data, the processing comprising: applying the first subnet to the first tensor, comprising: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks by the first subnet; after applying the first subnet, applying the second subnet to the second tensor, comprising: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks by the second subnet; wherein at least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks have different sizes. Therefore, by changing the block sizes of different subnets, the input tensor representing the image data can be decoded more efficiently. In addition, hardware limitations and requirements can also be considered.

[0551] In some exemplary implementations, the blocks in the first plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap; and / or the blocks in the second plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap. Therefore, the quality of the reconstructed image can be improved, especially along the boundaries of the blocks. Therefore, image artifacts can be reduced.

[0552] In another implementation, the first subnet processes a block in the first plurality of blocks independently; and / or the second subnet processes a block in the second plurality of blocks independently. For example, the first subnet processes at least two blocks in the first plurality of blocks in parallel; and / or the second subnet processes at least two blocks in the second plurality of blocks in parallel. Therefore, encoding of the image data can be performed faster.

[0553] According to an implementation example, the partitioning of the first tensor comprises determining the size of the blocks in the first plurality of blocks according to a first predefined condition; and / or the partitioning of the second tensor comprises determining the size of the blocks in the second plurality of blocks according to a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data. Thus, the block size can be adjusted and optimized according to available encoder and / or decoder resources and / or according to image content.

[0554] In one example, the first subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer; and / or the second subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer. Therefore, input tensor data can be processed efficiently because convolutional networks are particularly suitable for processing data in spatial dimensions.

[0555] In another example, the first subnet and the second subnet perform corresponding processing as part of image or moving image decompression. For example, the first subnet and / or the second subnet perform one of the following steps: image decoding of the convolutional subnet; image filtering. Therefore, decoding of image data can involve subnet processing of multiple related stages and improve decoding efficiency.

[0556] According to an implementation example, the input tensor is an image or image sequence including one or more components, wherein at least one component is a color component. This can support decoding of color components. In one example, the input tensor has at least two components, namely a first component and a second component; the first subnet divides the first component into a third plurality of blocks, and divides the second component into a fourth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the third plurality of blocks and the fourth plurality of blocks have different sizes; and / or the second subnet divides the first component into a fifth plurality of blocks, and divides the second component into a sixth plurality of blocks, wherein at least two corresponding juxtaposed blocks in the fifth plurality of blocks and the sixth plurality of blocks have different sizes. Therefore, multiple components may be decoded in units of blocks, wherein the block sizes of the components are different, which can further improve decoding efficiency and / or improve hardware implementation.

[0557] Another implementation example includes extracting the input tensor from the bitstream for processing by the neural network. Thus, the input tensor can be quickly extracted.

[0558] According to one implementation, the second subnet performs image post-filtering; for at least two of the second plurality of blocks, one or more parameters of the post-filtering are different and are extracted from the bitstream. Thus, decoding of the image data may involve subnet processing at multiple related stages and improve decoding efficiency. In addition, the post-filtering may be performed using filter parameters suitable for the block size, thereby improving the quality of the reconstructed image data.

[0559] In one example, the code stream further includes an indication of the size of a block in the first plurality of blocks and / or an indication of the size of a block in the second plurality of blocks. Thus, by providing the indication, the encoder and the decoder can set the block size in a corresponding and adaptive manner.

[0560] According to one aspect of the present invention, there is provided a computer program stored in a non-transitory medium, the computer program comprising codes, which when executed on one or more processors, perform the steps of any of the above aspects of the present invention.

[0561] According to one aspect of the present invention, there is provided a processing device for encoding an input tensor representing image data, the processing device comprising a processing circuit, the processing circuit being used for: a neural network including at least a first subnet and a second subnet processes the input tensor, the processing comprising: applying the first subnet to the first tensor, comprising: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks by the first subnet; after applying the first subnet, applying the second subnet to the second tensor, comprising: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks by the second subnet; wherein at least two corresponding juxtaposed blocks of the first plurality of blocks and the second plurality of blocks have different sizes.

[0562] According to one aspect of the present invention, a processing device for encoding an input tensor representing image data is provided, the processing device comprising: one or more processors; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the encoder to perform a method related to encoding an input tensor representing image data.

[0563] According to one aspect of the present invention, a processing device is provided for decoding a tensor representing image data, the processing device comprising a processing circuit, the processing circuit being used for: a neural network including at least a first subnet and a second subnet processes an input tensor representing the image data, the processing comprising: applying the first subnet to the first tensor, comprising: dividing the first tensor into a first plurality of blocks in a spatial dimension and processing the first plurality of blocks through the first subnet; after applying the first subnet, applying the second subnet to the second tensor, comprising: dividing the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks through the second subnet; wherein at least two corresponding juxtaposed blocks of the first plurality of blocks and the second plurality of blocks have different sizes.

[0564] According to one aspect of the present invention, a processing device is provided for decoding a tensor representing image data, the processing device comprising: one or more processors; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the decoder to perform a method related to decoding a tensor representing image data.

[0565] In summary, the present invention relates to image encoding and decoding of image regions in units of blocks based on a neural network. The neural network processes an input tensor representing image data, and the neural network includes at least a first subnet and a second subnet. The first subnet is applied to a first tensor, wherein the first tensor is divided into a first plurality of blocks in a spatial dimension. Then, the first subnet further processes the first plurality of blocks. After applying the first subnet, the second subnet is applied to a second tensor, wherein the second tensor is divided into a second plurality of blocks in the spatial dimension. Then, the second subnet further processes the second plurality of blocks. Among the first plurality of blocks and the second plurality of blocks, there are at least two corresponding juxtaposed blocks of different sizes. In the case of encoding, the first subnet and the second subnet perform partial compression, including image encoding, rate-distortion optimized quantization, and image filtering. In the case of decoding, the first subnet and the second subnet perform partial decompression, including image decoding and image filtering.

[0566] In addition, the present invention relates to image encoding and decoding of image areas in units of blocks. Specifically, multiple components of an input tensor, including a first component and a second component, are processed in a spatial dimension in multiple pipelines. The processing of the first component includes dividing the first component into a first plurality of blocks in the spatial dimension. Similarly, the processing of the second component includes dividing the second component into a second plurality of blocks in the spatial dimension. Then the corresponding first plurality of blocks and the second plurality of blocks are processed separately. Among the first plurality of blocks and the second plurality of blocks, there are at least two corresponding juxtaposed blocks of different sizes. In the case of compression, the processing of the first component and / or the second component includes image encoding, rate-distortion optimized quantization and image filtering. In the case of decompression, the processing includes image decoding and image filtering.

Claims

1. A method for processing an input tensor representing image data, characterized in that The method comprises the following steps: Processing a plurality of components of the input tensor in a spatial dimension, the plurality of components comprising a first component and a second component, the processing comprising: Processing the first component, comprising: dividing the first component into a first plurality of blocks in the spatial dimension and processing the blocks in the first plurality of blocks respectively; Processing the second component, comprising: dividing the second component into a second plurality of blocks in the spatial dimension and processing the blocks in the second plurality of blocks respectively; At least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks have different sizes.

2. The method according to claim 1, characterized in that At least two of the first plurality of blocks are processed independently or in parallel; and / or At least two of the second plurality of blocks are processed independently or in parallel.

3. The method according to claim 1 or 2, characterized in that: The first component represents a brightness component of the image data; The second component represents a chrominance component of the image data.

4. The method according to any one of claims 1 to 3, characterized in that The blocks in the first plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap; and / or Blocks in the second plurality of blocks that are adjacent in at least one of the spatial dimensions partially overlap.

5. The method according to any one of claims 1 to 4, characterized in that: The dividing of the first component comprises determining the size of a block in the first plurality of blocks according to a first predefined condition; and / or The dividing of the second component comprises determining the size of a block in the second plurality of blocks according to a second predefined condition.

6. The method according to claim 5, characterized in that The first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the image data.

7. The method according to claim 5 or 6, characterized in that: Determining the size of the tiles in the second plurality of tiles includes scaling the tiles in the first plurality of tiles.

8. The method according to any one of claims 5 to 7, characterized in that An indication of the determined size of a partition of the first plurality of partitions and / or the second plurality of partitions is encoded into a codestream.

9. The method according to any one of claims 1 to 8, characterized in that All the blocks in the first plurality of blocks have the same size and / or all the blocks in the second plurality of blocks have the same size.

10. The method according to claim 8 or 9, characterized in that: The indication further comprises positions of tiles in the first plurality of tiles and / or the second plurality of tiles.

11. The method according to any one of claims 8 to 10, characterized in that The first component is a luminance component, and the code stream includes the indication of the size of a block in the first plurality of blocks; The second component is a chrominance component, and the code stream includes an indication of a scaling factor, wherein the scaling factor is related to the size of the block in the first plurality of blocks and the size of the block in the second plurality of blocks.

12. The method according to any one of claims 8 to 11, characterized in that The processing of the input tensor includes processing as part of image or moving image compression.

13. The method according to claim 12, characterized in that The processing of the first component and / or the second component comprises one of the following steps: Image encoding with neural networks; Rate distortion optimization quantization (RDOQ); Image filtering.

14. The method according to claim 12 or 13, characterized in that Also includes: The code stream is generated by including the output of the processing of the first component and the second component into the code stream.

15. The method according to any one of claims 8 to 11, characterized in that The processing of the input tensor includes processing as part of image or moving image decompression.

16. The method according to claim 15, characterized in that The processing of the first component and / or the second component comprises one of the following steps: Image decoding with neural networks; Image filtering.

17. The method according to claim 16, characterized in that The processing of the second component comprises decoding chrominance components of the image from a representation of a luma component of the image.

18. The method according to any one of claims 12 to 17, characterized in that said processing of said first component and / or said second component comprises image post filtering; For at least two partitions of the first plurality of partitions, one or more parameters of post-filtering are different and are extracted from the bitstream; For at least two partitions of the second plurality of partitions, one or more parameters of post-filtering are different and are extracted from the codestream.

19. The method according to any one of claims 1 to 18, characterized in that The input tensor is an image or an image sequence comprising one or more of the plurality of components, wherein at least one component is a color component.

20. A computer program stored in a non-transitory medium, characterized in that The computer program comprises codes which, when executed on one or more processors, implement the steps of the method according to any one of claims 1 to 19.

21. An apparatus for processing an input tensor representing image data, characterized in that The apparatus comprises a processing circuit configured to: Processing a plurality of components of the input tensor in a spatial dimension, the plurality of components comprising a first component and a second component, the processing comprising: Processing the first component, comprising: dividing the first component into a first plurality of blocks in the spatial dimension and processing the blocks in the first plurality of blocks respectively; Processing the second component, comprising: dividing the second component into a second plurality of blocks in the spatial dimension and processing the blocks in the second plurality of blocks respectively; At least two corresponding juxtaposed blocks in the first plurality of blocks and the second plurality of blocks have different sizes.

22. An apparatus for processing an input tensor representing image data, characterized in that The device comprises: one or more processors; A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program causes the processing device to perform the method according to any one of claims 1 to 19.

Images