Method and apparatus for encoding a picture and decoding a bitstream

Abstract

This disclosure relates to a method, encoder, and decoder for processing a picture in the presence of a skip algorithm, which allows the encoder and decoder to skip encoding and decoding non-essential tensor elements based on standard deviation information available to both. As a result of the skip algorithm, elements are neither encoded nor decoded, so the minimum standard deviation in the arithmetic coder design should be aligned with the skip threshold. This reduces the size of the table in the arithmetic coder and improves coding efficiency.

Description

[Technical Field]

[0001] This disclosure relates to artificial intelligence (AI) coding. In particular, this disclosure relates to distributed quantization in entropy coding. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVDs, Blu-ray discs, video content acquisition and editing systems, and camcorders in security applications.

[0003] Even to depict relatively short videos, the amount of video data required can be substantial, which can pose difficulties when the data is streamed or otherwise transmitted over a communication network with limited bandwidth. Therefore, video data is generally compressed before being transmitted over today's telecommunications networks. Video size can also be a problem when video is stored on a storage device, as memory resources can be limited. Video compression devices often use software and / or hardware at the source to encode video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is received at the destination by a video decompression device that decodes the video data. Given limited network resources and the increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little to no sacrifice of image quality are desired.

[0004] Entropy coding is used to encode and decode data based on probability distributions, which requires memory to store probabilistic information. The memory used by entropy encoders and decoders is proportional to the number of probability distribution information entries stored. Fewer probability distribution information entries consume less memory, but this can lead to decreased coding efficiency. How to reduce the memory size required to store probability distribution information while simultaneously achieving higher compression efficiency is an urgent issue that needs to be addressed. [Overview of the project]

[0005] Embodiments of this disclosure provide apparatus and methods for entropy coding of data into a bitstream and for entropy decoding of data from a bitstream. Embodiments of this disclosure can also enable a reduction in the memory size for storing probability distribution information and an improvement in coding efficiency.

[0006] One embodiment of the present disclosure relates to a method for decoding a bitstream, the method being: Step to obtain the first bitstream; A step of obtaining a second bitstream, wherein the second bitstream contains hyperpre-information of the first bitstream; A step of obtaining a first sigma tensor based on a second bitstream, wherein the first sigma tensor contains several sigma values, where the sigma values ​​are variances or standard deviations, and the variances or standard deviations are related to the probability distribution information of the first bitstream; A step of obtaining a sigma index for a first sigma value of a first sigma tensor based on a quantization process, wherein the sigma index represents one of the probability distributions in a probability distribution table, the minimum sigma value of the quantization process is set to be equal to a skip threshold, and the skip threshold is used in the skip process to obtain a skip mask; The steps include entropy decoding the first bitstream based on the sigma index and obtaining the sequence of decoded symbols; The process includes the step of processing a sequence of decoded symbols based on a skip mask in order to obtain a residual tensor, wherein the residual tensor is used to obtain a reconstructed image.

[0007] The quantization process quantizes the sigma value as a sigma value among several discrete values ​​within a range of sigma values. The upper limit of the range is the maximum sigma value σ. max The lower limit of the range is the minimum sigma value σ min The number of discrete values ​​is the sigma number N. σ That is the case.

[0008] In other words, a quantized sigma value cannot be smaller than the minimum sigma value, nor can it be larger than the maximum sigma value.

[0009] Entropy coding is used to encode and decode data based on probability distributions and requires memory to store probabilistic information such as the cumulative distribution function (CDF) / probability mass function (PMF) for range coders. Generally, a zero-mean Gaussian distribution is used for latent features r ^Used to encode or decode, the variance of latent features is derived from the output (σ) of the hyperscale decoder. The variance is quantized to a finite number of quantized sigma values. Each quantized sigma value represents a specific distribution described by a predefined table. These tables are used occasionally during entropy coding, and are therefore typically stored in the read-only memory (ROM) of the ASIC codec or the L1 data cache of the software codec (typically 32 KB per CPU core). Smaller table sizes help reduce the cost of ASIC design and minimize cache misses in software codecs. However, a small number of sampled distributions can lead to a loss of compression efficiency due to the use of inaccurate probabilities in entropy coding.

[0010] The memory used by entropy encoders and decoders (e.g., CDF / PMF tables in range coders) is proportional to the number of quantized sigma values ​​from the hyperscale decoder. Fewer quantization levels consume less memory, but can lead to decreased coding efficiency. The present invention proposes optimizing a sigma (dispersive) quantization method to reduce the number of quantization levels, saving entropy coding-related table size and achieving higher coding efficiency.

[0011] Skip is widely used to improve the throughput of arithmetic coders. Skip logic results in not encoding some elements that have a variance / standard deviation lower than the skip threshold. (σ) ” If it is incorrect (for example, too small), you can understand that the skip process will be disabled based on cube_flag. If the skip process is enabled, then due to the skip process, σ ”All residual elements that are (threshold_skip) are not coded and are set to 0, this means that σ ” Sigma_Idx for (threshold_skip) is never used. Therefore, when σ min is set to be less than (threshold_skip), σ min between σ and (threshold_skip) ” sigma_index for is never used, which causes waste of codec resources and reduction in coding efficiency, and particularly leads to inappropriate occupation of memory resources. Therefore, the present application proposes that σ, which is the minimum value of variance / standard deviation in the sigma quantization process, min is modified to be equal to threshold_skip in skip logic (such as the threshold used in the skip mask process). In one specific embodiment, σ min = (threshold_skip). In one specific embodiment, threshold_skip is set as a default value of 0.2 (equal to the value 382 in an implementation that is integerized), and as a result, σ min can be modified as 0.2. When threshold_skip is set as another default value, σ min should be modified to be equal to the default value of threshold_skip. In the prior art, σ min is equal to 0.11, and the modified σ min is larger than the prior art value of 0.11. On the one hand, the sigma_index table size is reduced, which reduces consumed memory; on the other hand, when the skip process is disabled by cube_flag, indicating that sigma (variance) is incorrectly set to an excessively small value by mistake, σ min is used.

[0012] Furthermore, the present invention relates to the table size / the number of quantized variances / the number of quantization levels / the number of probability distributions (N σWe propose that ) is set as a power of 2. In one particular embodiment, N σ is, 2 5 = Equivalent to 32. N in prior art σ This is equal to 35, and the corrected N σ The value is less than 35, and as a result, this modification helps reduce memory consumption and achieve a simpler search of the distributed index.

[0013] Furthermore, this invention relates to σ, which is the maximum variance / standard deviation value in the sigma quantization process. max However, it is modified to a value within the range of 30 to 64, and in one particular embodiment, σ max It is set to be equal to 30, and σ in the prior art max is equal to 100, corrected σ max This value is smaller than the value of 100 in the prior art. Therefore, the bitstream size can be reduced without changing the quality or affecting coding efficiency, and the dynamic range in entropy will be reduced.

[0014] In one embodiment, a skip mask is used to indicate which elements of a second tensor are present in the second bitstream, and the second tensor is based on the sigma values ​​derived from the first sigma tensor.

[0015] It is possible to understand that some elements of the second sigma tensor are skipped based on the sigma mask.

[0016] In one embodiment, the minimum sigma value of the quantization process is set to be equal to 0.2.

[0017] In one embodiment, the minimum sigma value of the quantization process is set to be equal to threshold_skip, where threshold_skip indicates a skip threshold.

[0018] In one embodiment, the maximum sigma value of the quantization process is set to a value within the range of 30 to 64.

[0019] In one embodiment, the maximum sigma value of the quantization process is set to be equal to 30.

[0020] In one embodiment, the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within the range from the minimum sigma value to the maximum sigma value.

[0021] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

[0022] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within that range.

[0023] In one embodiment, the sigma number of the quantization process is set to be equal to 32.

[0024] In one embodiment, the step of obtaining a first sigma tensor based on a second bitstream is: The second bitstream is entropy-decoded to obtain the decoded hyperpre-tensor; To obtain a second sigma tensor, the process involves processing the decoded hyperpre-tensor based on a hyperscale decoder; The process includes the step of scaling the second sigma tensor in order to obtain the first sigma tensor.

[0025] In one embodiment, the output of the hyperscale decoder is a log-domain sigma tensor, and the first sigma tensor is a scaled log-domain sigma tensor.

[0026] In one embodiment, the first sigma value is the standard deviation of the linear domain or the variance of the linear domain.

[0027] In one embodiment, the skip mask indicates which samples of the sequence of symbols to be decoded are included in the bitstream. All other samples of the sequence of symbols to be decoded are inferred to be equal to zero.

[0028] In one embodiment, the first sigma tensor includes sigma values ​​or standard deviation values ​​in the logarithmic domain.

[0029] In one embodiment, the first sigma value is the standard deviation or variance of the log domain.

[0030] In one embodiment, the input to the sigma quantization process is the standard deviation in the linear or logarithmic domain, or in other words, the input to the sigma quantization process is the logarithmic domain standard deviation tensor I ” σ Furthermore, the sigma quantization process converts the logarithmic domain standard deviation tensor into a sigma index.

[0031] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0032] This includes the step of obtaining the sigma index based on the following formula:

[0033]

number

[0034] Here, sigma_idx represents the sigma index, and I σ ” represents the first sigma value in the logarithmic domain, or I σ ”represents the standard deviation in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ It is equal to 32.

[0035] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0036] This includes the step of obtaining the sigma index based on the following formula:

[0037]

number

[0038]

number

[0039] Here, sigma_idx represents the sigma index, σ ” represents the first sigma value in the linear domain, σ min σ is the minimum sigma value, max This is the maximum sigma value, and N σ This is a sigma number.

[0040] In one embodiment, N σ It is equal to 32.

[0041] In one embodiment, σ min It is equal to 0.2.

[0042] In one embodiment, σ max This value is within the range of 30 to 64.

[0043] One embodiment of the present invention discloses a method for encoding a bitstream, the method being: Steps to obtain hyperprior information for image data; A step of obtaining a first sigma tensor based on hyperprior information, wherein the first sigma tensor contains several sigma values, where the sigma values ​​are variances or standard deviations, and the variances or standard deviations are related to the probability distribution information of the image data; A step of obtaining a sigma index for a first sigma value of a first sigma tensor based on a quantization process, wherein the sigma index represents one of the probability distributions in a probability distribution table, the minimum sigma value of the quantization process is set to be equal to a skip threshold, and the skip threshold is used in the skip process to obtain a skip mask; A step of processing the sequence of symbols based on a skip mask in order to obtain the sequence of encoded symbols; To obtain a first bitstream, the process includes the step of entropy encoding a sequence of encoded symbols based on a sigma index.

[0044] In one embodiment, a skip mask is used to indicate which elements of a second tensor are present in the second bitstream, and the second tensor is based on the sigma value derived from the first sigma tensor.

[0045] In one embodiment, the minimum sigma value of the quantization process is set to be equal to 0.2.

[0046] In one embodiment, the minimum sigma value of the quantization process is set to be equal to threshold_skip, where threshold_skip indicates a skip threshold.

[0047] In one embodiment, the maximum sigma value of the quantization process is set to a value within the range of 30 to 64.

[0048] In one embodiment, the maximum sigma value of the quantization process is set to be equal to 30.

[0049] In one embodiment, the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within a sigma range from the minimum sigma value to the maximum sigma value.

[0050] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

[0051] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within that range.

[0052] In one embodiment, the sigma number of the quantization process is set to be equal to 32.

[0053] In one embodiment, the first sigma tensor includes sigma values ​​or standard deviation values ​​in the logarithmic domain.

[0054] In one embodiment, the output of the hyperscale decoder is a log-domain sigma tensor, and the first sigma tensor is a scaled log-domain sigma tensor.

[0055] In one embodiment, the skip mask indicates which samples of the decoded symbol sequence are included in the bitstream. All other samples of the decoded symbol sequence are inferred to be equal to zero.

[0056] In one embodiment, the first sigma value is the standard deviation or variance of the log domain.

[0057] In one embodiment, the first sigma value is the standard deviation of the linear domain or the variance of the linear domain.

[0058] In one embodiment, the input to the sigma quantization process is the standard deviation in the linear or logarithmic domain, or in other words, the input to the sigma quantization process is the logarithmic domain standard deviation tensor I” σ Furthermore, the sigma quantization process converts the logarithmic domain standard deviation tensor into a sigma index.

[0059] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0060] This includes the step of obtaining the sigma index based on the following formula:

[0061]

number

[0062] Here, sigma_idx represents the sigma index, and I σ ” represents the first sigma value in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ It is equal to 32.

[0063] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0064] This includes the step of obtaining the sigma index based on the following formula:

[0065]

number

[0066]

number

[0067] sigma_idx represents the sigma index, σ ” represents the first sigma value in the linear domain, σ min σ is the minimum sigma value, max This is the maximum sigma value, and Nσ This is a sigma number.

[0068] In one embodiment, N σ It is equal to 32.

[0069] In one embodiment, σ min It is equal to 0.2.

[0070] In one embodiment, σ max This value is within the range of 30 to 64.

[0071] One embodiment of the present invention discloses a decoder for processing a bitstream, comprising a storage medium and one or more processors, wherein the storage medium is configured to store computer executable instructions, and the one or more processors are configured to perform a method, such that the encoder is adapted to perform a method according to any one of the above embodiments.

[0072] One embodiment of the present invention discloses an encoder for processing a bitstream, comprising a storage medium and one or more processors, wherein the storage medium is configured to store computer executable instructions, and the one or more processors are configured to perform a method according to any one of the embodiments described above.

[0073] One embodiment of the present invention discloses a computer-readable storage medium equipped with computer-executable instructions, which, when executed in a computer or processor, cause the computer or processor to perform a method according to any one of the embodiments described above.

[0074] One embodiment of the present invention discloses an encoder for encoding a picture, the encoder comprising a receiver for receiving a picture, a transmitter for outputting a bitstream, and one or more processors configured to perform a method according to any one of the embodiments described above.

[0075] One embodiment of the present invention discloses a decoder for decoding a bitstream representing a picture, the decoder comprising a receiver that receives a first bitstream and a second bitstream, a transmitter that outputs a decoded picture, and one or more processors configured to perform a method according to any one of the embodiments described above.

[0076] Embodiments of the present invention disclose a computer program product that, when executed on a computer system, includes computer-executable instructions that cause the computer system to perform a method according to any one of the embodiments described above.

[0077] One embodiment of the present invention discloses a computer program stored in a storage medium, which, when executed on a computer or processor, includes computer-executable instructions that cause the computer or processor to perform a method according to any one of the embodiments described above.

[0078] Furthermore, a computer-readable storage medium is provided for storing computer-executable instructions, and when the computer-executable instructions are executed on the arithmetic system, the arithmetic system is instructed to perform a method according to any one of the embodiments described above. [Brief explanation of the drawing]

[0079] [Figure 1A] Figure 1A is a block diagram showing an example of a video coding system configured to implement an embodiment of the present disclosure.

[0080] [Figure 1B] Figure 1B is a block diagram showing another example of a video coding system configured to implement some embodiments of the present disclosure.

[0081] [Figure 2]FIG. 2 is a block diagram illustrating an example of an encoding apparatus or a decoding apparatus.

[0082] [Figure 3] FIG. 3 is a block diagram illustrating another example of an encoding apparatus or a decoding apparatus.

[0083] [Figure 4] FIG. 4 collectively shows an encoder and a decoder according to an embodiment.

[0084] [Figure 5] FIG. 5 shows a schematic diagram of input encoding and decoding.

[0085] [Figure 6] FIG. 6 shows an encoder and a decoder in accordance with the VAE framework.

[0086] [Figure 7] FIG. 7 shows components of the encoder according to FIG. 4 according to an embodiment.

[0087] [Figure 8] FIG. 8 shows components of the decoder according to FIG. 4 according to an embodiment.

[0088] [Figure 9] FIG. 9 shows an example of a JPEG AI encoder architecture.

[0089] [Figure 10] FIG. 10 shows an example of a JPEG AI decoder structure.

[0090] [Figure 11] FIG. 11 shows an example of a decoder structure for one component.

[0091] [Figure 12] FIG. 12 shows an example illustrating a hyper-scale decoder.

[0092] [Figure 13] FIG. 13 shows an example illustrating a synthetic transformation network.

[0093] [Figure 14] FIG. 14 shows an example of a decoder structure.

[0094] [Figure 15] FIG. 15 shows an example of an encoder structure.

[0095] [Figure 16] FIG. 16 is a flowchart of a method for decoding a bitstream according to an embodiment.

[0096] [Figure 17] FIG. 17 is a flowchart of a method for encoding a bitstream according to an embodiment.

[0097] [Figure 18] FIG. 18 shows an exemplary block diagram of a decoder structure.

[0098] [Figure 19] FIG. 19 shows an exemplary block diagram of an encoder structure.

[0099] [Figure 20] FIG. 20 shows an exemplary block diagram of a decoder structure.

[0100] [Figure 21] FIG. 21 shows an exemplary block diagram of an encoder structure.

[0101] [Figure 22A] FIG. 22A is a schematic diagram of a residual activation unit.

[0102] [Figure 22B] FIG. 22B is a schematic diagram of a residual activation function.

[0103] [Figure 22C] Figure 22C shows a schematic diagram of residual nonlocal attention block (RBAN).

[0104] [Figure 22D] Figure 22D shows a schematic diagram of the residual block (RB).

[0105] [Figure 22E] Figure 22E shows a schematic diagram of a lightweight residual block (LRB). [Modes for carrying out the invention]

[0106] Several embodiments are described below with reference to the drawings. Figures 1A to 3 show video coding systems and methods that may be used in conjunction with more specific embodiments of the present invention described in further drawings. Specifically, embodiments described in reference to Figures 1A to 3 can be used in conjunction with coding / decoding techniques, which are described further below, that utilize a neural network to code and / or decode a bitstream.

[0107] The following description refers to accompanying drawings that form part of the Disclosure and illustrate specific aspects of the Disclosure or specific aspects in which embodiments of the Disclosure may be used. It is understood that embodiments may be used in other aspects and may include structural or logical modifications not shown in the drawings. Therefore, the following detailed description should not be constrained, and the scope of the Disclosure is defined by the accompanying claims.

[0108] For example, disclosures relating to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, e.g., functional units, for performing the described method steps, even if one or more such units are not explicitly described or shown in the drawings (e.g., one unit may perform one or more steps, or each of the multiple units may perform one or more of the steps). On the other hand, if a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step for performing the function of one or more units, even if one or more such steps are not explicitly described or shown in the drawings (e.g., one step may perform the function of one or more units, or multiple steps may each perform one or more of the functions of the multiple units). Furthermore, it should be understood that the various exemplary embodiments and / or features described herein may be combined with each other unless otherwise specified.

[0109] Video coding generally refers to the processing of a sequence of pictures that make up a video or video sequence. The terms “frame” or “image” may be used synonymously in the field of video coding instead of “picture.” Video coding (or coding in general) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves the reverse processing compared to the encoder to reconstruct the video picture. Embodiments referring to “coding” a video picture (or picture in general) should be understood to relate to the “encoding” or “decoding” of the video picture or the respective video sequence. The combination of the encoding and decoding parts is also called a CODEC (Coding and Decoding).

[0110] In lossless video coding, it is possible to reconstruct the original video picture; that is, the reconstructed video picture will have the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example by quantization, reducing the amount of data representing the video picture, which cannot be fully reconstructed by the decoder; that is, the quality of the reconstructed video picture will be lower or worse than the quality of the original video picture.

[0111] Several video coding standards belong to the group of “lossy hybrid video codecs” (i.e., those that combine spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each picture in a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. In other words, in the encoder, video is typically processed, i.e., encoded, at the block (video block) level by generating predicted blocks using, for example, spatial (intra-picture) prediction and / or temporal (inter-picture) prediction, subtracting the predicted blocks from the current block (the block currently being processed / to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce (compress) the amount of data to be transmitted. In the decoder, the reverse process compared to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Furthermore, the encoder replicates the decoder processing loop, resulting in both producing matched predictions (e.g., intra-predictions and inter-predictions) and / or reconstructions for processing subsequent blocks, i.e., for coding. Recently, some or all of the encoding and decoding chain has been implemented using neural networks, or generally any machine learning or deep learning framework.

[0112] In the following embodiments of the video coding system 10, a video encoder 20 and a video decoder 30 are described.

[0113] Figure 1A is a schematic block diagram showing an exemplary coding system 10, for example, a video coding system 10 (or simply coding system 10), on which the techniques of the present application can be utilized. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video coding system 10 represent examples of devices that can be configured to perform the techniques described in the various examples in the present application.

[0114] As shown in Figure 1A, the coding system 10 includes a source device 12 configured to provide the encoded picture data 21 to a destination device 14 for decoding the encoded picture data 13, for example.

[0115] The source device 12 includes an encoder 20 and may further include, optionally, a picture source 16, a preprocessor (or preprocessing unit) 18, such as a picture preprocessor 18, and a communication interface or communication unit 22. Some embodiments of the present disclosure (e.g., relating to initial rescaling or rescaling between two preceding layers) can be performed by the encoder 20. Some embodiments (e.g., relating to initial rescaling) may be performed by the picture preprocessor 18.

[0116] The picture source 16 may include any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device that acquires and / or provides real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also be any type of memory or storage for storing any of the aforementioned pictures.

[0117] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 is sometimes referred to as a raw picture or raw picture data 17.

[0118] The preprocessor 18 is configured to receive (raw) picture data 17, perform preprocessing on the picture data 17, and obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is possible to understand that the preprocessing unit 18 may be an optional component.

[0119] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0120] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or a further processed version) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0121] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0122] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or a further processed version) from, for example, the source device 12 directly, or from any other source, such as a storage device, such as an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0123] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network or any combination thereof, or any type of private and public network or any combination thereof.

[0124] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.

[0125] A corresponding communication interface 28 to communication interface 22 can be configured, for example, to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0126] Both communication interfaces 22 and 28 may be configured as one-way or two-way communication interfaces, as indicated by the arrows on communication channel 13 in Figure 1A pointing from source device 12 to destination device 14, and may be configured to acknowledge and exchange any other information relating to communication links and / or data transmission, such as encoded picture data transmission, for example, to send and receive messages, to set up connections, etc.

[0127] The decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31 (further details are described below, for example, with reference to Figure 3).

[0128] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, to obtain post-processed picture data 33, for example, the post-processed picture 33. Post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, and may include, for example, preparing the decoded picture data 31 for display by the display device 34.

[0129] Some embodiments of this disclosure may be implemented by the decoder 30 or by the post-processor 32.

[0130] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to, for example, a user or viewer. The display device 34 may be any type of display for representing the reconstructed picture, for example, an integrated or external display or monitor, or may include one. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0131] Figure 1A depicts the source device 12 and the destination device 14 as separate devices, but the device embodiment may also include the functionality of both the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality. In such embodiments, the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0132] As will be apparent to those skilled in the art based on this description, the presence and (precise) division of functions or functions of different units within the source device 12 and / or destination device 14 as shown in Figure 1A may vary depending on the actual device and application.

[0133] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented through processing circuits such as those shown in Figure 1B, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding components, or any combination thereof. The encoder 20 is implemented through the processing circuit 46 and can implement various modules and / or any other encoder system or subsystem as described herein. The decoder 30 is implemented through the processing circuit 46 and can implement various modules and / or any other decoder system or subsystem as described herein. The processing circuit can be configured to perform various operations, as will be described later. When the technique is partially implemented in software, as shown in Figure 3, the device can store instructions for the software in a suitable non-temporary computer-readable storage medium, and the instructions can be executed in hardware using one or more processors to perform the technique of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a composite encoder / decoder (codec) within a single device, for example, as shown in Figure 1B.

[0134] The source device 12 and destination device 14 are a wide range of devices and may include any type of handheld or stationary device, such as a notebook or laptop computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video gaming console, video streaming device (such as a content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may not use an operating system or may use some kind of operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Thus, the source device 12 and destination device 14 can be wireless communication devices.

[0135] In some cases, the video coding system 10 shown in Figure 1A is merely an example, and the techniques of the present application may be applicable to video coding configurations (e.g., video coding or video decoding) that do not necessarily involve any data communication between the coding device and the decoding device. In other examples, the data may be retrieved from local memory, streamed over a network, or similar. The video coding device is capable of coding the data and storing it in memory, and / or the video decoding device is capable of retrieving the data from memory and decoding it. In some examples, coding and decoding are performed by devices that do not communicate with each other but simply code the data into memory and / or retrieve the data from memory and decode it.

[0136] For the sake of explanation, some embodiments described herein are illustrated by reference software for, for example, High Efficiency Video Coding (HEVC) or Multipurpose Video Coding (VVC), a next-generation video coding standard developed by the ITU-T Video Coding Expert Group (VCEG) and the ISO / IEC Motion Picture Expert Group (MPEG) Joint Working Group on Video Coding (JCT-VC). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0137] Figure 2 is a schematic diagram of a video coding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for carrying out the disclosed embodiments as described herein. In embodiments, the video coding device 400 may be a decoder, such as the video decoder 30 in Figure 1A, or an encoder, such as the video encoder 20 in Figure 1A.

[0138] The video coding device 400 includes an inlet port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an exit port 450 (or output port 450) for transmitting data; and memory 460 for storing data. The video coding device 400 may also include optoelectronic (OE) elements and electro-optical (EO) elements coupled to the inlet port 410, receiver unit 420, transmitter unit 440, and exit port 450 for optical or electrical signal input or output.

[0139] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., such as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiver unit 420, the transmitter unit 440, the output port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the embodiments disclosed above. For example, the coding module 470 performs, processes, prepares, or provides various coding operations. Thus, including the coding module 470 results in a substantial improvement to the functionality of the video coding device 400, resulting in the conversion of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0140] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store a program when such a program is selected for execution, and to store instructions and data read during program execution. Memory 460 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random-access memory (RAM), tri-associative memory (TCAM), and / or static random-access memory (SRAM).

[0141] Figure 3 is a simplified block diagram of an exemplary embodiment of a device 500 that can be used as either or both of the source device 12 and destination device 14 in Figure 1.

[0142] The processor 502 within the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or a set of devices capable of manipulating or processing existing or future information. The disclosed implementation can be carried out using a single processor, e.g., processor 502, as illustrated, but speed and efficiency advantages can be achieved using more than one processor.

[0143] The memory 504 within the device 500 can be, in one implementation, a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device can be used as memory 504. Memory 504 can contain code and data 506 accessed by the processor 502 using the bus 512. Memory 504 can further contain an operating system 508 and an application program 510, the application program 510 containing at least one program that enables the processor 502 to perform the method described herein. For example, the application program 510 can further include applications 1 through N, which are video coding applications that perform the method described herein.

[0144] The device 500 may also include one or more output devices, such as a display 518. The display 518 may, in one example, be a touch-sensitive display with touch-sensitive elements that can operate to sense touch input. The display 518 can be coupled to the processor 502 via the bus 512.

[0145] Although shown as a single bus in this case, the bus 512 of device 500 can be composed of multiple buses. Furthermore, the secondary storage 514 can be directly coupled to other components of device 500, or it can be accessed via a network, and it can include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, device 500 can be implemented in a wide variety of configurations.

[0146] The following describes more specific and non-limiting exemplary embodiments of the present invention. Before that, some explanations will be given to help understand this disclosure.

[0147] Artificial neural networks (ANNs), or connectionist systems, are computational systems inspired in part by the biological neural networks that make up the brains of animals. ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in a biological brain. Each connection can transmit signals to other neurons, much like synapses in a biological brain. The receiving artificial neuron can then process the signal and signal to the neurons it connects to. In ANN implementations, the "signals" in a connection are real numbers, and the output of each neuron can be calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. Weights increase or decrease the intensity of the signal in a connection. Neurons can have thresholds such that they transmit a signal only when the aggregated signal exceeds a threshold. Typically, neurons are aggregated into layers. Different layers can perform different transformations on their inputs. The signal travels from the first layer (input layer), traversing multiple layers in some cases, to the final layer (output layer).

[0148] The original goal of the ANN approach was to solve problems in the same way the human brain does. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a variety of tasks, including computer vision.

[0149] The name "Convolutional Neural Network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is simply a neural network that uses convolution instead of general matrix multiplication in at least one of its layers. A convolutional neural network consists of an input layer, an output layer, and several hidden layers. The input layer is the layer to which the input is provided for processing. For example, the neural network in Figure 6 is a CNN. The hidden layers of a CNN typically consist of a series of convolutional layers that perform multiplication or other dot product convolutions. The result of the layers is often one or more feature maps called channels. Subsampling may be performed on some or all of the layers. As a result, the feature maps may become smaller. The activation function in a CNN can be a RELU (Normalized Linear Unit) layer or a GDN layer, as exemplified above, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers. These layers are referred to as hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially referred to as convolutions, this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index in the matrix in that it affects how the weights are determined at a particular index point.

[0150] When programming a CNN to process a picture or image, the input is: (Number of images) × (Image width) × (Image height) × (Image depth) This is a tensor having the shape of . Next, after passing through the convolutional layer, the image is (Number of images) × (Feature map width) × (Feature map height) × (Feature map channels) It is abstracted into a feature map having the shape of . Convolutional layers in a neural network should have the following attributes: The convolutional kernel is defined by its width and height (hyperparameters), and the number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0151] Traditionally, multilayer perceptron (MLP) models have been used for image recognition. However, due to their fully connected nature, they suffer from high dimensionality and have not scaled well to higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too many to process at a scale that can be fully connected. Furthermore, such network architectures do not consider the spatial structure of the data, treating distant input pixels in the same way as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Therefore, for purposes such as image recognition, which are governed by spatially local input patterns, the fully connected neurons are redundant. Convolutional neural network (CNN) models mitigate the problems of MLP architectures by leveraging the strong spatially local correlations present in natural images. The convolutional layer is the core building block of a CNN. The layer parameters consist of a set of learnable filters (the kernels mentioned above), which have a small receptive field but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, calculating the dot product between the filter entry and the input to generate a two-dimensional activation map of that filter. As a result, the network can learn which filters are activated when a particular type of feature is detected at a given spatial location within the input.

[0152] The entire output volume of the convolutional layer is formed by stacking the activation maps of all filters along the depth dimension. Therefore, all entries in the output volume can also be interpreted as the output of neurons that observe small regions in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping that corresponds to the activation of different parts of an image, and also called a feature map because it is a mapping that shows where certain features are found in the image. High activation means that a certain feature has been found.

[0153] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. Several nonlinear functions exist for performing pooling, with max pooling being the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region. Intuitively, the precise location of a feature is less important than its approximate location relative to other features. This is the idea behind utilizing pooling in convolutional neural networks. The pooling layer works to gradually reduce the spatial size of the representation, thereby reducing the number of parameters in the network, the memory footprint, and the computational load, and consequently, also to control overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0154] The above ReLU stands for Normalized Linear Unit and applies a non-saturated activation function. This effectively removes negative values ​​from the activation map by setting them to zero. This increases the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions, such as the saturated hyperbolic tangent and sigmoid functions, are also used to increase nonlinearity. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalization accuracy.

[0155] After some convolutional and max-pooling layers, high-level inference in neural networks is performed via fully connected layers. Neurons in fully connected layers have connections to all activations in the previous layer, as seen in typical (non-convolutional) artificial neural networks. Thus, their activations can be computed as affine transformations, followed by matrix multiplication and then bias offsets (vector addition of learned or fixed bias terms).

[0156] An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. The purpose of an autoencoder is to learn (encode) a representation of a set of data by training the network to ignore signal "noise," typically for dimensionality reduction. Along with the reduction side, the reconstruction side is learned, and the autoencoder attempts to produce a representation from the reduced encoding that is as close as possible to its original input, as its name suggests.

[0157] Picture size: Refers to the width or height of a picture, or the width-height pair. The width and height of an image are usually measured in lumens.

[0158] Downsampling: Downsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is reduced. For example, if the input signal is an image having a size of height h and width w (hereinafter denoted as H and W in the same manner), and the output of downsampling has height h2 and width w2, at least one of the following is true:

[0159] h2<h

[0160] w2<w

[0161] In one implementation example, downsampling can be performed by retaining only each m-th sample and discarding the rest of the input signal (which is basically a picture in the context of the present invention).

[0162] Upsampling: Upsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is increased. For example, if the input image has a size of h and w (hereinafter denoted as H and W in the same manner), and the output of downsampling is h2 and w2, at least one of the following is true:

[0163] h<h2

[0164] w<w2

[0165] Resampling: Both the downsampling and upsampling processes are examples of resampling. Resampling is a process of changing the sampling rate (sampling interval) of an input signal.

[0166] Interpolation filtering: During an upsampling or downsampling process, filtering can be applied to improve the accuracy of the resampled signal and reduce the influence of aliasing. An interpolation filter generally involves weighted combination of sample values at sample positions around the resampling position. This can be implemented as follows:

[0167]

number

[0168] Here, f() is the resampled signal, and (x r ,y r ) is the resampling coordinate, C(k) is the interpolation filter coefficient, and s(x,y) is the input signal. The summation operation is: (x r ,y r This is executed for (x,y) in the neighborhood of ).

[0169] Cropping: Trimming the outer edges of a digital image. Cropping can be used to make an image smaller (in terms of the number of samples) and / or to change the aspect ratio (length-to-width) of the image.

[0170] Padding: Padding refers to increasing the size of an input image (or image) by generating new samples at the boundaries of the image. This can be done, for example, by using predefined sample values ​​or by using sample values ​​at that location within the input image.

[0171] Resizing: Resizing is a general term for changing the size of an input image. This may be done using either padding or cropping. It can also be done by resizing processes that use interpolation. Hereafter, resizing may also be referred to as rescaling.

[0172] Integer division: Integer division is a type of division in which the decimal part (remainder) is truncated.

[0173] Convolution: Convolution is given by the following general formula. In the following, f() can be defined as an input signal and g() can be defined as a filter:

number

[0174] Downsampling layer: A processing layer, such as a layer in a neural network, that results in a reduction of at least one of the dimensions of the input. Generally, the input may have three or more dimensions, which may include the number of channels, width, and height. However, this disclosure is not limited to such signals. Rather, signals that may have one or two dimensions (such as audio signals or audio signals with multiple channels) may be processed. A downsampling layer typically refers to a reduction in the width and / or height dimension. This can be achieved through processes such as convolution, averaging, and max pooling. Other downsampling methods are also possible, and the present invention is not limited in this respect.

[0175] Upsampling layer: A processing layer, such as a layer in a neural network, that results in an increase in one of the dimensions of the input. Generally, the input may have three or more dimensions, which may include the number of channels, width, and height. An upsampling layer typically refers to an increase in the width and / or height dimension. This can be achieved through processes such as deconvolution and replication. Other upsampling methods are also possible, and the present invention is not limited in this respect.

[0176] Several deep learning-based image and video compression algorithms follow the Variational Autoencoder Framework (VAE), such as G-VAE: A Continuously Variable Rate Deep Image Compression Framework, (Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, Yihui Feng) And, https: / / arxiv.org / abs / 2003.02012 It is available in [location].

[0177] The VAE framework can be counted as a nonlinear transformation coding model.

[0178] The transformation process can be divided into four main parts: Figure 4 illustrates the VAE framework. In Figure 4, the encoder 601 maps the input image x to a latent representation (denoted by y) using the function y=f(x). This latent representation may hereafter be referred to as a part of the “latent space” or a point within it. The function f() is a transformation function that converts the input signal x to a more compressible representation y. The quantizer 602 converts the latent representation y to a quantized latent representation y ^ For those which have (discrete) values, y ^ The transformation is performed by =Q(y), where Q represents the quantization function. The entropy model, or hyperencoder / decoder (also known as hyperplier) 603, is a quantized latent representation y ^ The distribution is estimated to obtain the minimum rate achievable with lossless entropy source coding.

[0179] The latent space can be understood as a compressed representation of data where similar data points are closer to each other within the latent space. The latent space is useful for learning data features and finding a simpler representation of the data for analysis.

[0180] Quantized latent representations T,y ^ and Hyperplier 3 side information z ^ This is included in bitstream 2 using arithmetic coding (AE) (it is binarized).

[0181] Furthermore, the quantized latent representation is reconstructed in image x ^ A decoder 604 is provided to convert to x ^ =g(y ^) is the signal x ^ This is an estimation of the input image x. x is x ^ It is desirable that the reconstruction quality be as close to the original as possible, that is, as high as possible. However, x ^ The higher the similarity between x and x, the greater the amount of side information that needs to be transmitted. This side information includes bitstream 1 and bitstream 2, shown in Figure 4, which are generated by the encoder and transmitted to the decoder. Generally, a larger amount of side information results in higher reconstruction quality. However, a larger amount of side information means lower compression. Therefore, one objective of the system described in Figure 4 is to balance reconstruction quality with the amount of side information transmitted in the bitstream.

[0182] In Figure 4, component AE605 is an arithmetic coding module, and the quantized latent representation y ^ and side information z ^ Convert this to a binary representation bitstream 1. ^ and z ^ The samples may include, for example, integers or floating-point numbers. One purpose of the arithmetic coding module is to convert the sample values ​​into a sequence of binary digits (through the process of binarization) (which are then included in a bitstream that may contain further parts corresponding to the encoded image or further side information).

[0183] Arithmetic decoding (AD) 606 is the process that reverses the binarization process, converting the binary numbers back to sample values. Arithmetic decoding is performed by the arithmetic decoding module 606.

[0184] Please note that this disclosure is not limited to this specific framework. Furthermore, this disclosure can be applied not only to image and video compression, but also to object detection, image generation, and recognition systems.

[0185] Figure 4 shows two interconnected subnetworks. In this context, a subnetwork is a logical division between parts of a larger network. For example, in Figure 4, modules 601, 602, 604, 605, and 606 are called the "encoder / decoder" subnetwork. The "encoder / decoder" subnetwork is responsible for encoding (generating) and decoding (analyzing) the first bitstream, "Bitstream 1". The second network in Figure 4, including modules 603, 608, 609, 610, and 607, is called the "hyper-encoder / decoder" subnetwork. The second subnetwork is responsible for generating the second bitstream, "Bitstream 2". The two subnetworks have different purposes. The first subnetwork has the following roles:

[0186] ●Conversion of input image x to latent representation y 601 (easier than compressing x),

[0187] ● Latent representation y, quantized latent representation y ^ Quantizing to 602,

[0188] ● Quantized latent expression y ^ This is compressed using AE by the arithmetic coding module 605 to obtain the bitstream "bitstream 1".

[0189] ● Analyze bitstream 1 using AD with arithmetic decoding module 606, and

[0190] ● Using the analyzed data, the reconstructed image (x ^ Reconstructing )

[0191] The purpose of the second subnetwork is to obtain the statistical properties of the samples in "bitstream 1" (e.g., mean, variance, and correlation between samples in bitstream 1) in a way that makes the compression of bitstream 1 by the first subnetwork more efficient. The second subnetwork generates a second bitstream, "bitstream 2," which contains that information (e.g., mean, variance, and correlation between samples in bitstream 1).

[0192] The second network is a quantized latent representation y ^ This is converted into side information z, and the side information z is converted into quantized side information z ^ Quantized, and the quantized side information z ^ The first bitstream is encoded (e.g., binarized) into bitstream 2. In this example, binarization is performed by arithmetic coding (AE). The decoding unit of the second network receives the decoded quantized side information z from the input bitstream 2. ^ Includes an arithmetic decoding unit (AD) 610 that converts to '. ^ ' is z ^ It may be identical, because arithmetic coding and decoding operations are lossless compression methods. Decoded quantized side information z ^ 'Then, the decrypted side information y ^ 607 is converted to '. y ^ ' is, y ^ Statistical properties of (e.g., y ^ This is the mean of the sample, or the variance of the sample values, or something similar. Decoded latent representation y ^ ' is, y ^ The arithmetic encoder 605 and arithmetic decoder 606 described above are provided to control the probability model.

[0193] FIG. 4 illustrates an example of a variational auto encoder (VAE), the details of which may differ in different implementations. For example, in a specific implementation, there may be additional components to more efficiently obtain the statistical characteristics of samples of the bitstream 1. In one such implementation, there may be a context modeler that aims to extract cross-correlation information of the bitstream 1. The statistical information provided by the second sub-network is used by an arithmetic encoder (AE) 605 and an arithmetic decoder (AD) 606.

[0194] FIG. 4 depicts the encoder and the decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and the decoder may be, and very often are, incorporated into different devices from each other.

[0195] FIG. 7 depicts an encoder and FIG. 8 separately depicts the decoder component of the VAE framework.

[0196] As input, the encoder receives a picture according to some embodiments. The input picture may comprise one or more channels, such as color channels or other types of channels, for example a depth channel or a motion information channel, or the like. The outputs of the encoder (as shown in FIG. 7) are bitstream 1 and bitstream 2. Bitstream 1 is the output of a first sub-network of the encoder, and bitstream 2 is the output of a second sub-network of the encoder.

[0197] Similarly, in FIG. 8, two bitstreams, bitstream 1 and bitstream 2, are received as input, and a reconstructed (decoded) image z ^ is generated.

[0198] As described above, a VAE can be divided into different logical units that perform different operations. This is illustrated in Figures 7 and 8, where Figure 7 shows the components involved in encoding signals such as video and provided encoded information. This encoded information is then received, for example, by the decoder components shown in Figure 8 for encoding. Note that the encoder and decoder components indicated by the digits 9xx and 10xx may, in their function, correspond to the components indicated by the digit 6xx referenced above in Figure 4.

[0199] Specifically, as shown in Figure 7, the encoder includes an encoder 901 that converts input x into a signal y, which is then provided to a quantizer 902. The quantizer 902 provides information to the arithmetic coding module 905 and the hyper-encoder 903. The hyper-encoder 903 provides the bitstream 2 already described above to the hyper-decoder 907, which signals the information to the arithmetic coding module 605.

[0200] Encoding can utilize convolution. Decoding can utilize deconvolution.

[0201] The output of the arithmetic coding module is bitstream 1. Bitstreams 1 and 2 are outputs of the signal coding unit, which are then provided (transmitted) to the decoding process.

[0202] Unit 901 is called the “encoder,” but the complete subnetwork shown in Figure 7 can also be called the “encoder.” The encoding process generally refers to a unit (module) that converts an input into an encoded (e.g., compressed) output. As can be seen from Figure 7, Unit 901 can actually be considered the core of the entire subnetwork because it performs the conversion of input x to a compressed version of x, which is y. Compression in encoder 901 can be achieved, for example, by applying a neural network, or in general, some processing network having one or more layers. In such a network, compression may be performed by cascading processes that include downsampling, which reduces the size and / or number of channels of the input. Thus, the encoder may be called, for example, a neural network (NN) based encoder, or something similar.

[0203] The remaining parts in the diagram (quantization unit, hyper-encoder, hyper-decoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or converting the compressed output y into a sequence of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 901 by lossy compression. The AE 905, in combination with the hyper-encoder 903 and hyper-decoder 907 used to construct the AE 905, can perform binarization, which allows for further compression of the quantized signal by lossless compression. Therefore, the entire subnetwork in Figure 7 can also be called the "encoder".

[0204] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary (bits). For example, in the VAE framework, the encoder, which is a nonlinear transformation, maps the input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height, and therefore a smaller size, the dimensionality (its size) of the signal is reduced, and thus it becomes easier to compress the signal y. It should be noted that, in general, an encoder does not necessarily have to reduce the size in both (or generally, all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally, a subset thereof).

[0205] The general principle of compression is illustrated in Figure 5. The latent space, which is the output of the encoder and the input of the decoder, represents the compressed data. Note that the size of the latent space can be much smaller than the size of the input signal. Here, the term size may refer to the resolution, for example, the number of samples in the feature map output by the encoder. The resolution may be given as the product of the number of samples per dimension (e.g., width × height × number of channels of the input image or feature map).

[0206] The reduction of input signal size is illustrated in Figure 5, which represents a deep learning-based encoder and decoder. In Figure 5, the input image x corresponds to the input data, which is the input to the encoder. The transformed signal y corresponds to the latent space, which has a smaller dimension or size than the input signal in at least one dimension. Each of the rows of circles represents a layer in the processing chain of the encoder or decoder. The number of circles in each layer indicates the size or dimension of the signal in that layer.

[0207] Figure 5 shows that the encoding operation corresponds to reducing the size of the input signal, and the decoding operation corresponds to reconstructing the image to its original size.

[0208] One method for reducing signal size is downsampling. Downsampling is the process of reducing the sampling rate of an input signal. For example, if the input image has sizes h and w, and the output of downsampling is h2 and w2, then at least one of the following is true:

[0209] h2 <h

[0210] w2 <w

[0211] Signal size reduction is typically performed stepwise, rather than all at once, along a chain of processing layers. For example, if an input image x has dimensions h and w (representing height and width) (or the size of the dimensions), and the latent space y has dimensions h / 16 and w / 16, then the size reduction might be performed in four layers during encoding, where each layer reduces the signal size by half in each dimension.

[0212] Some deep learning-based video / image compression methods use multiple downsampling layers. For example, the VAE framework in Figure 6 utilizes six downsampling layers, marked 801 through 806. Layers containing downsampling are indicated by downward arrows in the layer descriptions. The layer description "Conv Nx5x5 / 2↓" means that the layer is a convolutional layer with N channels and a convolutional kernel of size 5x5. As mentioned above, 2↓ means that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 results in one dimension of the input signal being halved in the output. In Figure 6, 2↓ indicates that both the width and height of the input image are reduced by half. Since there are six downsampling layers, if the width and height of the input image 814 (indicated by x) are given by w and h, then the output signal z ^ 813 has a width and height equal to w / 64 and h / 64, respectively. The modules indicated by AE and AD are the arithmetic encoder and arithmetic decoder, which have already been described above with respect to Figures 4, 7, and 8. The arithmetic encoder and decoder are concrete implementations of entropy coding. AE and AD (as part of components 813 and 815) may be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression method used to convert the value of a symbol into a binary representation, which is a recoverable process. Also, "Q" in the figures corresponds to the quantization operation mentioned above with respect to Figure 4, which will be further described above in the "Quantization" section. Furthermore, the quantization operation and the corresponding quantization unit as part of component 813 or 815 are not necessarily present and / or can be replaced by other units.

[0213] Figure 6 also shows a decoder including upsampling layers 807 to 812. A further layer 820 is provided between the upsampling layers 811 and 810 in the order of input processing; it is implemented as a convolutional layer, but does not bring upsampling to the received input. A corresponding convolutional layer 830 is also shown for the decoder. Such layers are provided within the NN to perform an operation on the input, which does not change the size of the input but modifies certain characteristics. However, providing such layers is not essential.

[0214] Viewed in the order of processing bitstream 2 as it passes through the decoder, the upsampling layers are executed in reverse order, i.e., from upsampling layer 812 to upsampling layer 807. Each upsampling layer is indicated to bring upsampling at an upsampling ratio of 2, as shown by ↑. Of course, it is not required that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8 or similar may be used. Layers 807 through 812 are implemented as convolutional layers (conv). Specifically, they may be intended to provide an operation on the input that is the reverse of the encoder operation, so the upsampling layers can apply a deconvolution operation to the received input, increasing in size by a factor corresponding to the upsampling ratio. However, this disclosure is not limited in general to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or nearest neighbor sample copying, or similar.

[0215] In the first subnetwork, after some convolutional layers (801 or 803), generalized division normalization is performed on the encoder side. Divisive normalization (GDN) follows, and on the decoder side, inverse GDN (IGDN) follows. In the second subnetwork, the activation function applied is ReLU. Note that this disclosure is not limited to such implementations, and in general, other activation functions may be used instead of GDN or ReLU.

[0216] Image and video compression systems generally cannot handle arbitrary input image sizes. This is because some processing units in the compression system (such as conversion units or motion compensation units) operate in relation to the smallest unit, and if the input image size is not an integer multiple of the smallest processing unit, the image cannot be processed.

[0217] For example, HEVC specifies four transposition unit (TU) sizes—4x4, 8x8, 16x16, and 32x32—for coding predictive residuals. Since the minimum transposition unit size is 4x4, it's not possible to process an input image with a size of 3x3 using an HEVC encoder and decoder. Similarly, if the image or picture size is not a multiple of 4 in one dimension, the image or picture cannot be processed either, because it cannot be divided into sizes that can be processed by valid transposition units (4x4, 8x8, 16x16, and 32x32). Therefore, a requirement of the HEVC standard is that the input image or picture must be a multiple of the minimum coding unit size, which is 8x8. Otherwise, the input image or picture cannot be compressed by HEVC. Similar requirements are imposed by other codecs. Maintaining such restrictions may be desirable in order to utilize existing hardware or software, or to maintain some interoperability, or even part thereof, of existing codecs. However, this disclosure is not limited to any specific transformation block size.

[0218] Some DNN (deep neural network) or NN (neural network) based image and video compression systems utilize multiple downsampling layers. In Figure 6, for example, four downsampling layers are included in the first subnetwork (layers 801 to 804), and two additional downsampling layers are included in the second subnetwork (layers 805 to 806). Therefore, given that the size of the input image is given by w and h (representing width and height), the output of the first subnetwork is w / 16 and h / 16, and the output of the second subnetwork is w / 64 and h / 64.

[0219] In deep neural networks, the term "deep" usually refers to the number of processing layers applied sequentially to the input. While a network with a large number of layers is called a deep neural network, there is no clear definition or guideline for which networks should be called deep networks. Therefore, for the purposes of this application, there is no significant difference between DNNs and NNs. A DNN may refer to an NN having more than one layer.

[0220] During downsampling, for example, when convolution is applied to the input, a fractional (or final) size may be obtained for the encoded picture. Such fractional sizes cannot be reasonably handled by subsequent layers of the neural network or by the decoder.

[0221] Entropy coding is used to encode and decode data based on probability distributions and requires memory to store probabilistic information such as the cumulative distribution function (CDF) / probability mass function (PMF) for range coders. Generally, a zero-mean Gaussian distribution is used for latent features r ^ Used to encode or decode r ^ The variance originates from the output (σ) of the hyperscale decoder. The variance is quantized into a finite number of quantized sigma values. Each quantized sigma value represents a specific distribution described by a predefined table. These tables are used occasionally during entropy coding, and are therefore typically stored in the read-only memory (ROM) of the ASIC codec or the L1 data cache of the software codec (typically 32 KB per CPU core). Smaller table sizes help reduce the cost of ASIC design and minimize cache misses in software codecs. However, a small number of sampled distributions can lead to a loss of compression efficiency due to the use of inaccurate probabilities in entropy coding.

[0222] Generally, two parameters, namely the quantization center (sampling) and the quantization interval (also called quantization boundary), control the manner of variance quantization. In one example, the quantization center σ of variance σ idx follows a power law form, as described by the following formula:

Formula

[0223] where L=64, σ min =0.11, σ max =256, L represents the number of quantized variances indicated as the number of quantization levels, that is, it requires 64 probability distributions to be stored,

Formula

[0224] The memory used by entropy encoders and decoders (e.g., CDF / PMF tables in range coders) is proportional to the number of quantized sigma values from the hyper-scale decoder. A smaller number of quantization levels results in less memory consumption, but may reduce coding efficiency. The proposal of the present invention optimizes the sigma (variance) quantization method to reduce the number of quantization levels, save the table size related to entropy coding, and achieve higher coding efficiency.

[0225] Skipping is widely used to improve the throughput of arithmetic coders. The skip logic results in not encoding some elements with variance / standard deviation below the skip threshold. In this invention, the variance / standard deviation quantization formula is modified to be consistent with the skip logic. Specifically, the minimum (min) variance / standard deviation value in the quantization formula is set to be equal to the skip threshold. Furthermore, the maximum (max) variance / standard deviation value in the quantization formula is reduced to reduce the dynamic range in the entropy decoder. Additionally, the table size (N) is also reduced. σ ) is set as a power of 2 to achieve a simpler search of the variance index. An example of a JPEG AI image encoder architecture is shown in Figure 9. The encoder uses a color transform (shown as "colorTr" in Figure 9) to separate the color into a primary component, shown as "Y", and a secondary component, shown as "UV" (these are essentially combinations of two non-primary color planes of the image). Each component is resampled using its own scaling factor (the downsampling module is shown as "s↓") and encoded separately using modules consisting of the same sequence of neural network layers, differing only in the size of the input tensor and the number of tensor channels.

[0226] The data (tensors and streams) are shown in "white" boxes, and the neural network modules required for encoding and decoding are shown in gray shaded boxes. The dashed borders and dashed arrows of the boxes indicate the encoder's unique operation.

[0227] The code stream consists of bitstreams stream z and stream y, which relate to the primary and secondary components, respectively.

[0228] Figure 10 shows an example of a JPEG AI decoder structure. The data (tensors and streams) are shown in the "white" boxes, and the neural network modules required for decoding are shown in the gray shaded boxes. For the primary and secondary color components, the code streams can be analyzed independently and reconstructed using modules consisting of the same sequence of neural network layers, differing only in the size of the input tensor and the number of tensor channels. A single-component decoder is shown in Figure 11.

[0229] The first stream z is to be analyzed by a lossless entropy decoder, and for example, the entropy decoder can be a me-tANS decoder. ^ The probability distribution for lossless coding is assumed to be a Gaussian distribution with pre-trained parameters (part of the trained model), and the cumulative distribution function (CDF(z)) is calculated based on these pre-trained parameters (in Figure 11, CDF(z) ^ (As shown) is used in lossless entropy decoders.

[0230] Decoded hyperpliar tensor z ^ It is used as input for two different processes, namely, hyper-decoders and hyper-scale decoders.

[0231] Next, stream y is to be analyzed by a lossless decoder (such as a me-tANS decoder). ^ The probability distribution for analysis is assumed to be a Gaussian with a standard deviation and a zero mean, given as the output of the following steps: The hyperscale decoder outputs the tensor σ[C,h4,w4] or the log-domain standard deviation tensor logσ. Next, the σ' or log-domain standard deviation tensor logσ' is generated by scaling according to the rate control parameter β within the "Sigma Scale". Next, within the "Adaptive Sigma Scale," the process involves masking or scaling according to the RVS parameter section to generate σ'' or the log-domain standard deviation tensor logσ''.

[0232] Finally, the tensor σ'' value or log-domain standard deviation tensor logσ'' is quantized in the sigma quantization process, and the tensor σ'' or log-domain standard deviation tensor logσ'' is converted into an index in the probability distribution table. According to the rules specified by the SKIP Mode, some elements of the residual tensor are skipped (not encoded / decoded) and replaced with zeros in the Decoder SKIP module, which receives the parsed set of syntax elements {s} from the tANS decoder and mask_sigma from the SKIP mask generation module, and the reconstructed residual tensor r is reshaped into a 3D shape. ^ The output is [C,h4,w4]. In one embodiment, the outputs and inputs of the hyperscale decoder, sigma scale, and adaptive sigma scale are all log-domain tensors.

[0233] On the decoder side, residual r ^ The inverse gain unit (Inverse) is calculated according to the parameter β. Scaled by Gain Unit, r ^ This generates '. Then, the residual tensor is scaled in the inv RVS (Inverse Residual and Variance Scale) module, and the residual tensor r ^ This forms the reconstructed latent tensor y ^ It is used for that purpose.

[0234] As shown in Figure 10, after reconstruction, the primary and secondary components are resampled along with their respective scaling factors (the upsampling module is indicated as "s↑" in Figure 10). After resampling to the original picture size, all three color components are resampled to obtain inter-channel correlation information (Inter Channel). The image passes through a Correlation Information (ICCI) filter. The ICCI filter uses information from the luminance (first-order component) to enhance the color information plane (second-order component) of the image.

[0235] The inputs to this ICCI filter process are as follows:

[0236] - x ^ ' Y [1,H,W] (In Figure 10, "s Y (After the resampling module, indicated as "↑")

[0237] - x ^ ' UV [2,H,W] (In Figure 10, "s UV (After the resampling module, indicated as "↑")

[0238] - Pre-trained ICCI model parameters for each color component (0, 1, 2) using icci_model_idx.

[0239] The output of this process is as follows:

[0240] - Full-size enhanced YUV ^ [3,H,W] (YUV in Figure 10)

[0241] The reconstruction process is completed by an inverse color transformation ("invColorTr" in Figure 10), and the input to the inverse color transformation process is the three color planes Y ^ [H,W],U ^ [H,W],V ^where [H,W] are the height and width. The output of this process is a reconstructed image in the form of a three-color plane tensor RGB^[3,H,W]. The inverse color conversion is performed for each pixel at coordinates [i,j], where 0≦i<H and 0≦j<W.

[0242] It should be understood that the hyper encoder, hyper decoder, and hyper scale decoder are all composed of two independent pipelines for the primary component and the secondary component, respectively.

[0243] The decoding operation of the me-tANS entropy decoder will be described below. The inputs of the me-tANS decoder are as follows:

[0244] - a bitstream (one of "stream-y Y ", "stream-y UV ", "stream-z Y ", "stream-z UV ")

[0245] The output of this process is as follows:

[0246] - a symbol sequence {s}, which subsequently undergoes a skip mode decoder process (for the residual tensor) or is simply reshaped into a 3D tensor to obtain the reshaped z ^ .

[0247] The hyper encoder will be described below. The inputs of the hyper encoder include:

[0248] - y[C,h4,w4] latent tensor,

[0249] - size H of the input / output tensor in , W in ,

[0250] - operating point indicator opIdx,

[0251] - Model parameters for the hyperencoder network defined by (modelIdx)

[0252] The output of the hyperencoder is a hypertensor of size [C,h6,w6].

[0253] An example of a hyperscale decoder is shown in Figure 12.

[0254] The inputs for the hyperscale decoder are as follows:

[0255] - z ^ [C,h6,w6] Reconstructed hypertensor

[0256] - Size H of input / output tensor in ,W in

[0257] - Operating point indicator opIdx

[0258] - Model parameters for a hyperscale decoder network defined by pairs (modelIdx, opIdx), where all multiplication parameters in these models are 8-bit integers.

[0259] The output of the hyperscale decoder is 0 ≤ I σ <((N σ The quasi-deviation logarithmic tensor I has integer values ​​within the range -1) << sigmaPrecision). σ [C,h4,w4], where N σ =32 and sigmaPrecision=7.

[0260] It can be understood that the output of a hyperscale decoder is a logarithmic domain standard deviation tensor, where logarithmic domain is an abbreviation for logarithmic domain.

[0261] In the scalable hyperdecoder, all operations are integers, the accumulator in all operations is in the range of 32-bit integers, and the model parameters are quantized to 8-bit integers. This guarantees the bit-exact behavior of this neural network module. The hyperscale decoder uses special types of operations: quantized convolution and quantized transposed convolution. For each quantized convolution in the process, a clipping value {d k} and de-scaling shift parameters parameters){p k A set of} is specified (1≦k≦(opIdx=0)?4:5). All clipping values ​​in the quantized convolution are d k =2 15 -1. De-scaling shift {p k} is part of the trained model.

[0262] NOTE - In the quantization model, the magnitude of the weights taken in is 2 15 The combination of shift and clipping values ​​does not exceed -1, and ensures that the register of the quantized convolution is 32 bits or less.

[0263] Depending on the operating point indicator (opIdx), the hyperscale decoder performs the following sequence of steps:

[0264] In a hyperscale decoder at the basic operating point (opIdx=0), the number of channels is C for all hidden layers. The hyperscale decoder starts with a quantized transposed convolution kernel with size 4x4 and stride 2, followed by a cropping layer (stride 2, depth 6) and a normalized linear unit. Next, a quantized convolution with stride 1 and kernel size 3x3 is followed by a normalized linear unit. Again, a quantized transposed convolution kernel with size 4x4 and stride 2 is followed by a cropping layer (stride 2, depth 5) and a normalized linear unit, as well as a quantized convolution with stride 1 and kernel size 3x3.

[0265] At the higher operating point (opIdx=1), two sets of quantized convolutions with a stride of 1 and kernel size of 3x3 increase the number of channels to 4C, followed by a pixel shuffle (stride 2), which returns the number of channels to C, then a cropping layer (stride 2, depths of 6 and 5 respectively), and finally a normalized linear unit. Then there are three quantized convolutions with a stride of 1 and kernel size of 3x3, two of which are followed by a normalized linear unit.

[0266] The sigma scale in Figure 11 is explained below. The sigma scale is applied to both the encoder and decoder sides, and the inputs to the sigma scale are as follows:

[0267] - Forward gain vector m

[0268] - A standard deviation tensor of size [C,h4,w4], which is the output of the hyperscale decoder (D.3).

[0269] The output of the sigma-scale process is as follows:

[0270] - A standard deviation tensor of size [C,h4,w4] scaled by a forward gain vector, which is directed toward the "Adaptive Sigma Scale".

[0271] The Sigma Scale takes the standard deviation tensor σ[C,h4,w4] as input, modifies it, and outputs the standard deviation tensor multiplied by the gain vector:

[0272]

number

[0273] All elements of a tensor within the same channel are scaled by the same multiplier.

[0274] Adaptive Sigma Scale (in Figure 11, "Adpt. Sigma") The "Scale" setting (as indicated) is applied on both the encoder and decoder sides, and the adaptive sigma scaling process is adaptively enabled based on the value of num_rvs_params (num_rvs_params==1).

[0275] The inputs to the adaptive sigma-scale process include:

[0276] - num_rvs_params;

[0277] - mask_rvs[num_rvs_params,C,h4,w4];

[0278] - presice_flag_rvs [num_rvs_params];

[0279] - scale_rvs [num_rvs_params];

[0280] - Standard deviation tensor σ'[C,h4,w4] or log-domain standard deviation tensor I' σ[C,h4,w4], which follows the "sigma scale".

[0281] The output of this process is as follows:

[0282] - Scaled residual tensor σ''[C,h4,w4] or log-domain residual tensor I'' σ [C,h4,w4].

[0283] The adaptive sigma-scaling process is as follows:

[0284] - First, σ'' is initialized to be equal to σ'.

[0285] - For idx=0 to num_rvs_params-1, the following sequential steps apply:

[0286] The descaling parameters are: shift=(precise_flag_rvs[idx]==1)?=13:7 and offset=(2^shift)-1 It is set as follows:

[0287] Scaled residual tensor σ ~ Each element is obtained as follows:

number

[0288] When num_rvs_params=0, the inverse sigma-scale process is essentially disabled, and the output tensor σ ~ This matches the input σ'.

[0289] In one embodiment, the derivation of the control parameters and gain vector is described as follows.

[0290] A total of five models are trained for different quality ranges. The models are selected based on the modelIdx coded in the picture header. The rate control parameter β controls the compression ratio and defines the operation in the sigma scale, inverse, and forward gain units.

[0291] The forward gain vector m is used on the encoder side in the gain unit. Logarithmic scale forward gain vector m log This is used in the sigma scale. Inverse gain vector m -1 These are used on the decoder side in the inverse gain unit. Forward gain vector m and inverse gain vector m -1 Both have a size [C] equal to the number of channels in the residual tensor.

[0292] The trainable models are defined for five models t=0,...,4, with a reference forward gain vector m on a logarithmic scale. t log Includes vector m t log Each element is a 12-bit value.

[0293] The inputs to the gain vector derivation process are as follows:

[0294] - A 12-bit variable betaDisplacementLog, equal to betaDisplacementLogY for the first-order component and betaDisplacementLogUV for the second-order component;

[0295] - modelIdx indicates the trainable model to be used;

[0296] - Reference forward gain vector m for t = modelIdx t log .

[0297] The output of the gain vector derivation process is as follows:

[0298] - Forward gain vector m on a logarithmic scale log ;

[0299] - Forward gain vector m;

[0300] - Inverse Gain Vector m -1 .

[0301] Forward gain vector m on a logarithmic scale log It is calculated as follows:

[0302] - m log =m t log +betaDisplacementLog

[0303] Forward and backward gain vectors m and m -1 It is calculated as follows:

[0304]

number

[0305]

number

[0306]

number

[0307] Here, σ min =0.2;σ max =30;N σ =32, sigmaPrecision=7.

[0308] In one embodiment, sigma quantization (Sigma quantization in Figure 11) The process is described as follows:

[0309] The inputs to this sigma quantization process include:

[0310] - σ_log"[C,h4,w4] or I" σ This is represented as [C,h4,w4] and is the output of the adaptive sigma scale.

[0311] The output of this process includes:

[0312] - sigma_Idx[C,h4,w4], this is further r ^ It is used in entropy decoders related to this.

[0313] The sigma quantization process is as follows:

[0314] For c=0,...C-1,i=0,...,h4-1,j=0,...,w4-1,

[0315]

number

[0316] And here, N σ =32,sigmaPrecision = 7

[0317] sigma_Idx=i,i∈[0,N σ When calculating the CDF table for ), the following assumptions are made:

[0318]

number

[0319] Here, σ min =0.2,σmax = 30.

[0320] Entropy coding is used to encode and decode data based on probability distributions and requires memory to store probability information such as cumulative distribution function (CDF) / probability mass function (PMF) tables for range coders. Generally, a zero-mean Gaussian distribution is used for latent features r ^ Used to encode or decode, the variance of latent features is derived from the output (σ) of the hyperscale decoder. The variance is quantized to a finite number of quantized sigma values. Each quantized sigma value represents a specific distribution described by a predefined table. These tables are sometimes used during entropy coding. σ This indicates the number (or table size) of probability distributions stored in memory for entropy coding and decoding. Each probability distribution has a variance (denoted as σ), and σ min This represents the minimum variance of the stored probability distribution, σ max This indicates the maximum variance of the stored probability distribution. sigma_Idx is used to indicate the index of the probability distribution used by the entropy decoder / encoder.

[0321] As an option, σ min This indicates the minimum value of variance / standard deviation in the sigma quantization process, σ max This represents the maximum value of variance / standard deviation in the sigma quantization process, and N σ This indicates the number of probability distribution tables stored in memory for entropy coding and decoding.

[0322] In one embodiment, r ^ The probability distribution for analyzing this is assumed to be the log-domain standard deviation and zero-mean Gaussian as the output of the following steps: The hyperscale decoder outputs a tensor idx[C,h4,w4], which is then scaled according to the control parameter β within sigma scale to generate idx', and then masked and scaled according to the RVS parameter section within adaptive sigma scale to generate idx''.

[0323] The skip process is widely used to improve the throughput of arithmetic coders. Skip logic results in that some elements having variance / standard deviation lower than the skip threshold are not encoded. If σ'' coming from the entropy decoder is incorrect (e.g., too small), the skip process is disabled based on cube_flag. When the skip process is enabled, due to the skip process, σ ” <(threshold_skip) all residual elements are not coded and set to 0, which means that Sigma_Idx for σ ” <(threshold_skip) is never used. Therefore, when σ min is set to be less than (threshold_skip), σ min and σ between (threshold_skip) ” will never have their sigma_index used, which causes waste of codec resources and reduction in coding efficiency, and in particular leads to inappropriate occupation of memory resources.

[0324] Therefore, the present application proposes that σ, which is the minimum value of variance / standard deviation in the sigma quantization process, min is modified to be equal to threshold_skip (something like the threshold used in the skip mask process) in skip logic. In one specific embodiment, σ min = (threshold_skip << shift). In one specific embodiment, threshold_skip is set to a default value of 0.2, and as a result, σ minThis can be corrected to 0.2. If threshold_skip is set to any other default value, σ min This should be modified to be equal to the default value of threshold_skip. σ in prior art min This is equal to 0.11, and the corrected σ min This is greater than the value of 0.11 in the prior art. Therefore, on the one hand, the sigma_index table size will be reduced, thereby reducing the memory consumed; on the other hand, the skip process will be disabled by cube_flag, which indicates that sigma (variance) has been incorrectly set to too small a value. min This is used.

[0325] Furthermore, the present invention relates to the table size / number of quantized variances / number of quantization levels / number of probability distributions (N σ We propose that ) is set as a power of 2. In one particular embodiment, N σ is, 2 5 = Equivalent to 32. N in prior art σ This is equal to 35, and modified N σ The result is smaller than 35, and this modification helps reduce the memory consumed and achieve a simpler search of the distributed index.

[0326] Furthermore, this invention relates to σ, the maximum variance / standard deviation value in the sigma quantization process. max This is modified to a value within the range of 30 to 64, and in certain embodiments, σ max It is set to 30. σ in prior art max is equal to 100, corrected σ max This value is smaller than the prior art value of 100. Therefore, the bitstream size can be reduced without changing the quality or affecting coding efficiency, and the dynamic range in entropy will be reduced.

[0327] In another embodiment, sigma quantization (Sigma quantization in Figure 11) The process is described as follows:

[0328] The inputs to this sigma quantization process include:

[0329] - σ”[C,h4,w4], the output of the adaptive sigma scale.

[0330] The output of this process includes:

[0331] - sigma_Idx[C,h4,w4], this is further r ^ It is used in entropy decoders related to this.

[0332] Set of values ​​{b i},i=0,...,N σ -1 is calculated as follows:

[0333]

number

[0334] Here, σ min =0.20,σ max =30..64,N σ = 32.

[0335] The process is as follows:

[0336] For c=0,...C-1,i=0,...,h4-1,j=0,...,w4-1,

[0337]

number

[0338] sigma_Idx=i,i∈[0,Nσ Regarding this, the following assumptions are made when calculating the CDF table:

[0339]

number

[0340] Here, b i indicates the quantization boundary, σ i The quantization center is indicated, and the continuous variance is quantized, thereby creating an interval (σ). i-1 ,σ i The value in ] is σ i It is quantized as follows. sigma_idx becomes equal to i-1.

[0341] The latent space prediction process is explained as follows:

[0342] The predicted μ is a tensor of the same size as the latent tensor y[C,h4,w4]. On the encoder side, the prediction is subtracted from the latent tensor to generate the residual tensor r[C,h4,w4], which is rounded to an integer (int 16) and encoded by the me-tANS entropy coder. On the decoder side, the prediction is added to the decoder residual to reconstruct the tensor y ^ Generates [C,h4,w4].

[0343] The hyperdecoder shown in Figure 11 is explained. The learning-based hyperdecoder consists of two independent pipelines that have the same neural network architecture except for the input size and number of channels.

[0344] The inputs to this hyperdecoder process are as follows:

[0345] - z ^ [C,h6,w6] Reconstructed hyperlatent tensor

[0346] - Model parameters for the hyperdecoder network defined by (modelIdx)

[0347] The output of this process is as follows:

[0348] - p[2C,h4,w4] is an explicit prediction (part of the prediction tensor derived from explicitly signaled information).

[0349] The hyperdecoder is an 8-stage neural network process and reconstructs residual r ^ Taking " as input, latent space tensor y ^ Generates an explicit_prediction that is input to the multi-stage context model - MCM that outputs '. Pre-composition latent scaling (Latent After Scaling Before Synthesis (LSBS), the reconstructed latent space tensor y ^ The signal is ready for reconstruction. The latent tensor reconstructions of the first and second components are independent of each other.

[0350] The Residual and Variance Scale (RVS) is described as follows:

[0351] The RVS module scales both the residual and standard deviation parameters used to create entropy-coding models. RVS works together and shares the same scaling factor.

[0352] The residual scaling is located after the gain unit on the encoder side. The inverse residual scaling is located immediately after the inverse gain unit. Adaptive sigma scale scaling is located after the sigma scale module. The RVS process achieves adaptive quantization of residual samples based on their corresponding standard deviations.

[0353] The RVS scaling tensor generation process is described as follows:

[0354] The inputs to the RVS scaling tensor generation process include:

[0355] - Logarithmic domain standard deviation tensor I' after sigma scale σ [C,h4,w4]

[0356] - rvs_num_idx

[0357] The output of the RVS mask generation process includes the following:

[0358] - scaling_rvs[C,h4,w4]

[0359] The variables used in this section are defined as follows:

[0360] - threshold_rvs;

[0361] - scale_rvs;

[0362] - num_rvs_param is equal to 3 if rvs_num_idx is 1; otherwise, num_rvs_param is equal to 1.

[0363] - The process of converting the standard deviation from a logarithmic scale to a linear scale is I' σ It is called with input σ' and output σ'.

[0364] The RVS mask in Figure 11 is explained as follows:

[0365] The residual and variance scale (RVS) scales both the residual and standard deviation parameters used to construct the entropy coding model. The residual scaling is located after the gain unit on the encoder side. The inverse residual scaling is located immediately after the inverse gain unit, as shown in Figure 11. The RVS process achieves adaptive quantization of residual samples based on their corresponding standard deviation values.

[0366] The input for the RVS mask generation process includes the standard deviation tensor σ', which is the output of the sigma-scale module. The output of the RVS mask generation process is then used as input for the Inv RVS process, which then processes the scaled residual tensor r ^ Outputs ".

[0367] The LSBS and LSBS mask in Figure 11 are described as follows:

[0368] - The pre-synthesis latent scale (LSBS) is applied on the decoder side to modify the reconstructed latent tensor based on the signaled scaling factor. The input to the LSBS mask generation process is the standard deviation tensor σ''[C,h4,w4] which is the output of the adaptive sigma scale process, and the output of the LSBS mask generation process is mask_lsbs, which is used as the input to the LSBS process, which then processes the modified latent tensor y ^ Outputs [C,h4,w4].

[0369] - In one embodiment, in the decoder, the input to the LSB process is the residual tensor r after entropy decoding (and skip mode if applicable). ^ [C,h4,w4], predicted tensor μ[C,h4,w4] after the prediction fusion process, latent tensor y ^ This is a binary mask generated using [C,h4,w4] and its standard deviation σ.

[0370] The skip mask and decoder skip in Figure 11 are explained as follows:

[0371] The skip mode allows skipping writing to / analysis from bitstream residual tensor elements that can be identified as zero by the encoder and decoder. The input to the skip mask generation process is: The standard deviation tensor σ''[C,h4,w4] is the output of the adaptive sigma-scale process. compIdx: 0 for primary (y), or 1 for secondary (uv), cube_luma_flag if(compIdx==0), or cube_chroma_flag[C,h_4,w_4] if(compIdx==1) The output of the skip mask generation process is mask_skip, which is used as the input to the decoder skip process, and the decoder skip process then generates the residual tensor r ^ Output [C,h4,w4]. Also, after decoding from "stream-y" using me-tANS, the ID array s[num_res_elements] is also the input for the decoder skip process.

[0372] In one embodiment, the skip-mode process is described as follows:

[0373] The skip-mode process is also called the SKIP-mode decoder process, the SKIP process, or the decoder-side SKIP operation / process. In a decoder, the inputs to the skip-mode process include:

[0374] - ID array s[num_res_elements] after decoding from "stream-y" using me-tANS

[0375] - mask_skip[num_skip_params[C,h4,w4]

[0376] The output of this process is as follows:

[0377] - Residual tensor r ^ [C,h4,w4]

[0378] The output of the lossless decoding process is a 1D array {s k} and its size is equal to the total number of "1"s in the maskAggregate tensor [C,h4,w4].

[0379] In other words, the maskAggregate tensor [C,h4,w4] is equal to the residual tensor r ^ This determines which of the samples will be included in the bitstream. All other samples in the quantized residual tensor are estimated to be equal to zero.

[0380] The residual skip mode process in the decoder is as follows:

[0381] - The dimension [C,h4,w4] is set to be equal to the number of channels, the height and width of the sigma tensor (Table 2).

[0382] - The tensor r^[C,h4,w4] and maskAggregate[C,h4,w4] are initialized to all zeros and all ones, respectively.

[0383] - Counter k=0

[0384] - The following sequential steps apply:

[0385] - For c=0..C-1,i=0..h4-1,j=0..w4-1,

number

[0386] The Synthesis Transform shown in Figure 11 is explained as follows:

[0387] An example of a composite transformation network is shown in Figure 13.

[0388] Learning-based reconstruction (also known as synthetic transformation) consists of two pipelines with identical neural network architectures, except for the input size and number of channels.

[0389] The input for the analysis transformation is as follows:

[0390] - Reconstructed latent space tensor y of shape [C,h4,w4] ^ Auxiliary information tensor y ~ [Cd,h4,w4] concatenated

[0391] - Operating point indicator opIdx

[0392] - Size H of input / output tensor in ,W in

[0393] - Model parameters for a composite transformation network defined by a pair (modelIdx, opIdx)

[0394] The output of the analysis transformation is size [C in ,H in ,W in This is the reconstructed color component x^ of the tensor ].

[0395] The composite transformation is the principal latent tensor (y ^ [C,h4,w4]) and auxiliary input (y ~ [C d It begins with a connection to [h4,w4]). Depending on the operating point indicator (opIdx), the decoder executes the following sequence of steps.

[0396] With respect to the basic operating point (opIdx=0) and the first-order component (compIdx=0), the first step of the composite transformation (at depth=4 of the deep neural network process) is C+C. d It consists of one lightweight residual block followed by a transposed convolution with a kernel size of 4x4, reducing the number of channels to C3. This transposed convolution precedes the cropping layer (stride 2, depth 4) and the residual activation unit. For the second-order component (compIdx=1), the first step of the synthesis transformation (depth=4) is just a latent synthesis block (LCB), reducing the number of channels to C+C d Change C3 to 2C. The next step for both components (depthe=3) is a transposed convolution with a kernel size of 4x4, changing the number of channels to C2. This transposed convolution precedes the cropping layer (stride 2, depth 3) and the residual activation unit. The next steps in the process (depth=2 and 1) are a combination of a regular convolution with a kernel size of 3x3, a stride of 1, and an invariant number of channels C2, and a residual activation unit. Next, there is a 3x3 convolution with a stride of 1, which increases the number of channels from C2 to 16C. in Increase it to this. This means that the output of the next layer (stride 4, depth 1) will have the number of channels C. in This is done to ensure that it has [the necessary properties]. The process ends with a cropping layer (stride 4, depth 1).

[0397] For the higher operating point (opIdx=1) and the first-order component (compIdx=0), the first step of the synthesis transformation (at depth=4 of the deep neural network process) is a residual block with C+Cd channels followed by a transposed convolution with a kernel size of 4x4, reducing the number of channels to C3. This transposed convolution is coupled with a cropping layer (stride 2, depth 4) and a residual activation unit. For the second-order component (compIdx=1), the first step of the synthesis transformation is just a latent synthesis block (LCB), reducing the number of channels to C+C d Change from to C3. The next step for both components (depthe=3) is a transposed convolution with a kernel size of 4x4, changing the number of channels to C2. A convolution-based attention block is then placed (shown as CAB). This is followed by a cropping layer (stride 2, depth 3) and a residual activation unit.

[0398] The next step in the process (depth=2) is a regular convolution with a kernel size of 1x1 and a stride of 1, resulting in 4C1 output channels. This is done to ensure that the output of the next layer (stride 2) has C1 channels. The final step (depth=1) begins with a transformer-based attention module (indicated as TAM(compIdx)) and executes a cropping layer (stride 2, depth 2) and a residual activation unit with a kernel size of 3x3. This process is carried out by a transposed convolution with a kernel size of 3x3 and a stride of 2, followed by a cropping layer (stride 2, depth 1).

[0399] Multistage Context Modeling (MCM) is described as follows:

[0400] The inputs for this MCM process are as follows:

[0401] - Operating point indicator opIdx,

[0402] - r ^ [C,h4,w4] is the reconstructed residual tensor, which is output from the skip model process.

[0403] - p[2C,h4,w4] is an explicit prediction and the output of the hyperdecoder.

[0404] - A model with parameters defined by eight MCMk,k=0,...,7 (modelIdx,k).

[0405] The output of the MCM process is as follows:

[0406] - y ^ '[C,h4,w4] Reconstructed latent tensor.

[0407] The MCM process consists of the following steps:

[0408] - After a padding layer (depth=5, stride=2), perform a down-shuffle of M=2 to obtain the explicit prediction tensor p[2C,h4,w4], p ·· This creates a reshaped prediction tensor in the form [8C,h5,w5].

[0409] - After a padding layer (depth=5, stride=2), a down-shuffle of M=1 is performed to reconstruct the residuals r ^ [C,h4,w4], r ·· This process results in a reshaped residual tensor in the form [4C, h5, w5].

[0410] - p ·· [8C,h4,w4] in four parts

number

[0411] - r ·· [4C,h4,w4] in 8 parts

number

[0412] - For k=0,...,3,

[0413] o MCM(k) process

[0414] ■ Please enter the following:

[0415] ● {y ·· m},m=0,...,k-1, and the previously reconstructed part of the reshaped latent space tensor

[0416] ● r ·· k - Collocated parts of the reconstructed residual tensor

[0417] ● p ·· k%4 - Part of the reshaped explicit predictor tensor

[0418] ■ The following will be generated as output.

[0419]

number

[0420] - y ·· [0:(2C-1),h5,w5] Channel Net Process for Tensors

[0421] - For k=3,...,7,

[0422] o MCM(k) process

[0423] ■ Please enter the following:

[0424] ● {y ·· m},m=0,...,k-1, and the previously reconstructed part of the reshaped latent space tensor

[0425] ● r ·· k - Parts in the same position as the reconstructed residual tensor

[0426] ● p ·· k%4 - Part of the reshaped explicit predictor tensor

[0427] ■ The following will be generated as output.

[0428]

number

[0429] - After an up-shuffle with M=2, a cropping layer (depth=5, stride=2) follows, y ·· [4C,h5,w5] to y ^ '[C,h4,w4]

[0430] - This is an up-shuffle with M=2, and μ ·· [4C,h5,w5] μ ^ '[C,h4,w4] (which is used further in the LSBS process)

[0431] This section describes an example of a multi-stage context modeling process. The MCM process is a recursive process: later stages use previously obtained elements of the output tensor as input.

[0432] Down-shuffle processing

[0433] The inputs for this process are as follows:

[0434] - M - Number of tensor slices

[0435] - a[MC,h4,w4] - 3D tensor

[0436] The output of this process is as follows:

[0437] - a[4MC,h5,w5] - A re-shuffled 3D tensor with the same elements.

[0438] In the down-shuffle process, the input slice is first divided into M segments by the channel dimension, and for each slice, the elements of the tensor are grouped into four groups: 0, 1, 2, 3 (note that the group numbers are in zigzag order, not raster order), and these groups are then reshuffled by the channel dimension.

[0439] Down-shuffling, like down-sampling convolution with a stride of 2, alters the spatial size of the tensor; therefore, this down-shuffling process is preceded by a padding layer (h4,w4,1,2). Zero-padding is performed. In other words, down-shuffling is used to change the size or shape of a tensor.

[0440] Up-shuffle processing

[0441] This process is the reverse of the down-shuffle process.

[0442] The inputs for this process are as follows:

[0443] - M - Number of tensor slices

[0444] - a ·· [4MC,h5,w5] - 3D tensor

[0445] The output of this process is as follows:

[0446] - A reshuffled 3D tensor a[MC,h4,w4] with the same elements.

[0447] In the up-shuffle process, the input slice is first divided into M segments by the channel dimension, then for each slice, the elements of the tensor are reshuffled in a zigzag order, and the number of channels is reduced fourfold while the spatial dimension is increased twofold.

[0448] Up-shuffling, like inverse convolution with a stride of 2, changes the spatial size of the tensor, so this up-shuffling process is followed by a cropping layer (h4,w4,1,2). In other words, up-shuffling is used to change the size or shape of a tensor.

[0449] Stage 0 of Multi-Stage Context Modeling

[0450] The inputs for this process are as follows:

[0451] - p ·· 0 = p ·· [0:2C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=0

[0452] - r ·· 0=r ·· [0:C / 2-1,h5,w5] - k=0 part of the reshuffled residual tensor

[0453] - Predictive fusion network parameters for k=0

[0454] The output of this process is as follows:

[0455] - y ·· k =y ·· [0:C / 2-1,h5,w5] - Part of the k=0 of the reshaped latent space tensor

[0456] The process is as follows:

[0457] - p ·· 0 proceeds through channel padding from 2C to 3C, which generates a0[3C,h5,w5].

[0458] - a0[3C,h4 / 2,w4 / 2] proceeds to the predicted fusion network k=0, which is μ ·· Generate 0=μ[0:C / 2-1,h5,w5].

[0459] - y ·· 0 = μ ·· 0+r ·· 0

[0460] Stage 1 of Multi-Stage Context Modeling

[0461] The inputs for this process are as follows:

[0462] - p ·· 1=p ·· [2C:4C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=1

[0463] - r ·· 1=r ·· [C / 2:C-1,h5,w5] - k=1 part of the reshuffled residual tensor

[0464] - y ·· 0 = y ·· [0:C / 2-1,h5,w5] - Part of the reshuffled and reconstructed latent tensor at k=0

[0465] - CONV k=1 (3x3, C / 2, C / 2) trained parameters

[0466] - Predictive fusion network parameters for k=1

[0467] The output of this process is as follows:

[0468] - y ·· k =y ·· [C / 2:C-1,h5,w5] - Part of the k=1 of the reshaped latent space tensor

[0469] The process is as follows:

[0470] - y ·· Step 0 proceeds through the convolutional layer CONV(3×3,C / 2,C / 2), which has a size of [C / 2,h5,w5] y ··~ Generate 0.

[0471] - p ·· 0 proceeds through channel padding 2C → 5C / 2 to generate P[5C / 2, h5, w5].

[0472] - y ··~ 0 and P1 are connected to form a1, and proceed to the predictive fusion network k=1, which is μ ·· This generates 1 = μ[C / 2:C-1,h5,w5].

[0473] - y ·· 1 = μ ·· 1+r ·· 1

[0474] Stage 2 of Multi-Stage Context Modeling

[0475] The inputs for this process are as follows:

[0476] - p ·· 2=p·· [4C:6C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=2

[0477] - r ·· 2=r ·· [C:3C / 2-1,h5,w5] - Part of the reshuffled residual tensor with k=2

[0478] - y ·· 0...1 [0:C-1,h5,w5] - k=0 and k=1 parts of the reshuffled and reconstructed latent tensor

[0479] - CONV k=2 (3×3,C,C / 2) trained parameters

[0480] - Predictive fusion network parameters for k=2

[0481] The output of this process is as follows:

[0482] - y ·· k =y ·· [C:3C / 2-1,h5,w5] - Part of the k=2 of the reshaped latent space tensor

[0483] The process is as follows:

[0484] - y ·· 0 and y ·· 1 is linked and proceeds through a convolutional layer CONV(3×3,C,C / 2), which has a size of [C / 2,h5,w5] y ··~ Generate 1.

[0485] - p ·· Step 2 proceeds through channel padding 2C → 5C / 2, generating P[5C / 2, h5, w5].

[0486] - y ··~ Connecting 1 and P2 forms a2, and proceeding to the predicted fusion network k=2, which is μ·· This generates 2 = μ[C:3C / 2-1,h5,w5].

[0487] - y ·· 2 = μ ·· 2+r ·· 2

[0488] Stage 3 of Multi-Stage Context Modeling

[0489] The inputs for this process are as follows:

[0490] - p ·· 3 = p ·· [6C:8C-1,h5,w5] - Part of the reshuffled explicit prediction tensor with k=3

[0491] - r ·· 3=r ·· [3C / 2:2C-1,h5,w5] - k=3 part of the reshuffled residual tensor

[0492] - y ·· 0...2 [0:3C / 2-1,h5,w5] - k=0, k=1 and k=2 parts of the reshuffled and reconstructed latent tensor

[0493] - CONV k=3 (3×3, 3C / 2, C / 2) trained parameters

[0494] - Predictive fusion network parameters for k=3

[0495] The output of this process is as follows:

[0496] - y ·· k =y ·· [3C / 2:2C-1,h5,w5] - Part of the k=3 of the reshaped latent space tensor

[0497] The process is as follows:

[0498] - y ·· 0, y ·· 1, and y ·· 2 is linked and proceeds through a convolutional layer CONV(3×3,3C / 2,C / 2), which has a y size [C / 2,h5,w5] ··~ Generate 2.

[0499] - p ·· Step 3 proceeds through channel padding of 2C → 5C / 2, generating P[5C / 2, h5, w5].

[0500] - y ··~ 2 and P3 are connected to form a3, which proceeds to the predicted fusion network k=3, which is μ ·· This generates 3 = μ[3C / 2:2C-1,h5,w5].

[0501] - y ·· 3 = μ ·· 3+r ·· 3

[0502] Stage 4 of Multi-Stage Context Modeling

[0503] The inputs for this process are as follows:

[0504] - p ·· 0 = p ·· [0:2C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=0

[0505] - r ·· 4=r ·· [2C:5C / 2-1,h5,w5] - Part of the reshuffled residual tensor with k=4

[0506] - y ·· 0 = y ~ [0:C / 2-1,h5,w5] - k=0 part of the output from the channel net (E.5.4)

[0507] - Predictive fusion network parameters for k=4

[0508] The output of this process is as follows:

[0509] - y ·· k =y ·· [2C:3C / 2-1,h5,w5] - Part of the k=4 of the reshaped latent space tensor

[0510] The process is as follows:

[0511] - p ·· 0 proceeds through channel padding 2C→5C / 2 to generate P4[5C / 2,h5,w5].

[0512] - y ··~ 0 and P4 are linked to form a4, and proceed to the predictive fusion network k=4, which is μ ·· This generates 4 = μ[2C:5C / 2-1, h5, w5].

[0513] - y ·· 4 = μ ·· 4+r ·· 4

[0514] Stage 5 of Multi-Stage Context Modeling

[0515] The inputs for this process are as follows:

[0516] - p ·· 1=p ·· [2C:4C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=1

[0517] - r ·· 5=r ·· [5C / 2:3C-1,h5,w5] - k=5 part of the reshuffled residual tensor

[0518] - y ~ 1=y ~[C / 2:C-1,h5,w5] - k=1 part of the output from the channel net (E.5.4)

[0519] - y ·· 4=y ·· [2C:3C / 2-1,h5,w5] - Part of the k=4 of the reshaped latent space tensor

[0520] - CONV k=5 (3x3, C / 2, C / 2) trained parameters

[0521] - Predictive fusion network parameters for k=5

[0522] The output of this process is as follows:

[0523] - y ·· k =y ·· [5C / 2:3C-1,h5,w5] - Part of the k=5 of the reshaped latent space tensor

[0524] The process is as follows:

[0525] - y ·· 4 = y ·· [2C:3C / 2-1,h5,w5] proceeds through the convolutional layer CONV(3×3,C / 2,C / 2), which has a size of [C / 2,h5,w5] y ··~ Generate 4.

[0526] - y ~ 1, y ··~ 4, and P ·· Connect 1 to form a5, proceed to the predictive fusion network k=5, and this μ ·· This generates 5 = μ[5C / 2:3C-1,h4 / 2,w4h5,w5 / 2]. - y ·· 5 = μ ·· 5+r ·· 5

[0527] Stage 6 of Multi-Stage Context Modeling

[0528] The inputs for this process are as follows:

[0529] - p ·· 2=p ·· [4C:6C-1,h5,w5] - Part of the reshuffled explicit prediction tensor at k=2

[0530] - r ·· 6=r ·· [3C:5C / 2-1,h5,w5] - Part of the reshuffled residual tensor at k=6

[0531] - y ~ 2=y ~ [C:3C / 2-1,h5,w5] - k=2 part of the output from the channel net (E.5.4)

[0532] - y ·· 4, y ·· 5 - Parts of the reshaped latent space tensor for k=4,5

[0533] - CONV k=6 (3×3,C,C / 2) trained parameters

[0534] - Predictive fusion network parameters for k=6

[0535] The output of this process is as follows:

[0536] - y ·· k =y ·· [3C:5C / 2-1,h5,w5] - Part of the k=6 of the reshaped latent space tensor

[0537] The process is as follows:

[0538] - y ·· 4 and y ··5 is linked and proceeds through a convolutional layer CONV(3×3,C,C / 2), which has a size of [C / 2,h5,w5] y ··~ Generate 5.

[0539] - y ~ 2 and y ··~ 5 and P ·· Connecting 2 forms a6, and proceeds to the predicted fusion network k=6, which is μ ·· This generates 6 = μ[3C:5C / 2-1, h5, w5].

[0540] - y ·· 6 = μ ·· 6+r ·· 6

[0541] Stage 7 of Multi-Stage Context Modeling

[0542] The inputs for this process are as follows:

[0543] - p ·· 3 = p ·· [6C:8C-1,h5,w5] - Part of the reshuffled explicit prediction tensor with k=3

[0544] - r ·· 7=r ·· [5C / 2:4C-1,h5,w5] - Part of the reshuffled residual tensor at k=7

[0545] - y ~ 3=y ~ [3C / 2:2C-1,h5,w5] - k=3 part of the output from the channel net (E.5.4)

[0546] - y ·· 4, y ·· 5, y ·· 6 - Parts of the reshaped latent space tensor for k=4, 5, and 6

[0547] - CONV k=7 (3×3, 3C / 2, C / 2) trained parameters

[0548] - Predictive fusion network parameters for k=7

[0549] The output of this process is as follows:

[0550] - y ·· k =y ·· [5C / 2:4C-1,h5,w5] - Part of the k=7 of the reshaped latent space tensor

[0551] The process is as follows:

[0552] - y ·· 4 and y ·· 5 and y ·· 6 is linked and proceeds through a convolutional layer CONV(3×3,3C / 2,C / 2), which has a y size [C / 2,h5,w5] ··~ Generate 6.

[0553] - y ~ 3 and y ··~ 6 and P ·· Connecting 3 forms a7, which proceeds to the predicted fusion network k=7, which is μ ·· This generates 7 = μ[5C / 2:4C-1,h5,w5]. - y ·· 7 = μ ·· 7+r ·· 7

[0554] MCM design elements

[0555] The three operations that modify the size or shape of a tensor are: 1) down-shuffle, 2) up-shuffle, and 3) channel-wise padding. Two subnetworks, a channel network and a predictive fusion network, are used.

[0556] Figure 14 shows an example of a decoder structure, which includes an entropy parameter decoder, a skip mask generation process, a sigma quantization process, an entropy decoder, and a decoder skip process.

[0557] The input to the entropy decoder includes a first bitstream and a sigma index, where the sigma index is used to indicate the index of the probability distribution table and is also required to analyze the symbol interpretation. The output of the entropy decoder includes a sequence of decoded symbols {s}, which cannot be considered equal to the residuals since not all residual elements are coded.

[0558] The input to the decoder-skip process includes the sequence of decoded symbols {s} output from the entropy decoder and a sigma mask (or skip mask), where the sigma mask is used to determine which residual symbols were coded, which were not coded, and which should be set to 0. The output of the decoder-skip process includes the sequence of decoded symbol residuals. On the decoder side, the decoded symbol residuals r ^ It is scaled by an inverse gain unit according to parameter β, and r ^ This generates '. Then the residual tensor is inv In the RVS (Inverse Residual and Variance Scale) module, the residual tensor r is scaled and ^ It forms the residual tensor r. ^ " is the reconstructed latent tensor y ^ Used for the reconstructed latent tensor y ^ This is the reconstructed color component x ^ Used to obtain x ^ The size is [C in ,H in ,W in It is a tensor of ].

[0559] Furthermore, the reconstructed color components may be resampled to the original picture size and combined to obtain an extended YUV signal, and the extended YUV signal is processed through an inverse color conversion process to obtain an RGB signal of the reconstructed image.

[0560] The input of the sigma quantization process includes sigma (variance value) from the entropy parameter decoder, and the output of the sigma quantization process includes a sigma index used as an input to the entropy decoder. In other words, sigma (variance value) from the entropy parameter decoder (the value of tensor σ") is quantized in the sigma quantization process, and in this sigma quantization process, the value of tensor σ" is converted into a sigma index of a probability distribution table. The entropy decoder decodes the input first bitstream based on the sigma index and a stored probability distribution table to obtain a sequence of decoded symbols. In an embodiment, the input of the sigma quantization process includes a sigma value in logarithmic space, the sigma index is obtained based on the logarithmic value of the sigma value, and in this embodiment, the output of the entropy parameter decoder is the logarithmic value of the sigma value.

[0561] In one embodiment, the table for converting standard deviation from logarithmic scale to linear scale is described as follows:

[0562] The table LogToLinear is used for converting from logarithmic scale to linear scale and vice versa. The values are stored with sigmaPrecision precision. The size of the table is (N σ -1)<<sigmaPrecision .

[0563] NOTE - LogToLinear is obtained using the following formula:

[0564]

Formula

[0565] The input to the skip mask generation process includes the sigma (variance value) (the value of the tensor σ'' or the logarithm of the σ'' value) from the entropy parameter decoder and the cube_flag, where the cube_flag is selected by the encoder and transmitted to the decoder. The output of the skip mask generation process includes the sigma mask, which is used as one input to the decoder skip process to know which residual symbols are coded, which are not coded, and which should be set to 0.

[0566] The input to the entropy parameter decoder contains a second bitstream, and the output to the entropy parameter decoder is sigma (the variance value, denoted as σ") or the logarithm of sigma (denoted as logσ"), where sigma is the variance value related to the probability distribution information of the first bitstream.

[0567] An entropy parameter decoder can be implemented by a neural network, which includes an entropy decoder, a hyperscale decoder, a sigma-scale process, and an adaptive sigma-scale process, where the entropy decoder analyzes a second bitstream to obtain the decoded hyperpre-tensor z, for example, z ^ The probability distribution for lossless coding is assumed to be Gaussian with pre-trained parameters. Then, the decoded hyperpre-tensor z ^The first sigma tensor σ is obtained by processing it with a hyperscale decoder, then the first sigma tensor is processed with a sigma-scale process to obtain a second sigma tensor σ', and further, the second sigma tensor is processed with an adaptive sigma-scale process to obtain a third sigma tensor σ''. The third sigma tensor σ'' is the above sigma (variance value), and is the output of the entropy parameter decoder, the input of the sigma quantization process, and the input of the skip mask generation process. In one example, the sigma-scale process includes a scaling process according to a rate control parameter β, and the adaptive sigma-scale process includes a masking and scaling process according to an RVS parameter.

[0568] Figure 15 shows an example of an encoder structure, which includes an entropy parameter decoder, a skip mask generation process, a sigma quantization process, an entropy encoder, and an encoder skip process.

[0569] The input to the entropy parameter decoder includes a second bitstream, which contains hyperprior information of the image encoded within the first bitstream, which is acquired by a hyperencoder that processes the image data, and the output to the entropy parameter decoder includes sigma (the variance value represented as σ'' in the linear domain) or the logarithm of sigma (represented as logσ''), where sigma is the variance value related to the probability distribution information of the first bitstream. In one embodiment, the input to the entropy parameter decoder is hyperprior information rather than the second bitstream, i.e., the encoder structure does not need to analyze the hyperprior information from the second bitstream.

[0570] An entropy parameter decoder may include an entropy decoder, a hyperscale decoder, a sigma-scale process, and an adaptive sigma-scale process, where the entropy decoder analyzes a second bitstream to obtain the decoded hyperpre-tensor z^, and in one example, z ^ The probability distribution for lossless coding is assumed to be Gaussian with pre-trained parameters. Then, the decoded hyperpre-tensor z ^ The first sigma tensor σ is obtained by processing it with a hyperscale decoder, then the first sigma tensor is processed with a sigma-scale process to obtain a second sigma tensor σ', and further, the second sigma tensor is processed with an adaptive sigma-scale process to obtain a third sigma tensor σ''. The third sigma tensor σ'' is the above sigma (variance value), and is the output of the entropy parameter decoder, the input of the sigma quantization process, and the input of the skip mask generation process. In one example, the sigma-scale process includes a scaling process according to a rate control parameter β, and the adaptive sigma-scale process includes a masking and scaling process according to an RVS parameter.

[0571] The sigma quantization process and skip mask generation process in Figure 15 are the same as those in Figure 14.

[0572] The input to the encoder skip process includes the sigma mask (or skip mask) from the skip mask generation process and a sequence of symbol residuals, while the output to the encoder skip process includes a sequence of encoded symbols {s}.

[0573] The input to the entropy encoder includes a sigma index from the sigma quantization process and a sequence of encoded symbols {s} from the encoder skip process, where the sigma index is used to indicate the index in the probability distribution table and is required to encode the symbols {s}. The output of the entropy decoder is a bitstream containing the encoded data of the input image.

[0574] One embodiment of the present invention discloses a method for decoding a bitstream as shown in Figure 16, the method 1600 including:

[0575] 1610. Obtain the first bitstream;

[0576] It can be understood that the first bitstream contains encoded image data or video data.

[0577] 1620. Obtaining a second bitstream;

[0578] Here, the second bitstream contains the hyperprior information of the first bitstream; in other words, the second bitstream contains the hyperprior information of the encoded image data or video data in the first bitstream. The hyperprior information can be called a hyperparameter tensor or hyperparameter.

[0579] 1630. Obtaining the first sigma tensor based on the second bitstream;

[0580] Here, the first sigma tensor contains several sigma values ​​or standard deviations, where the sigma values ​​are variances related to the probability distribution information of the first bitstream; and the standard deviations are related to the probability distribution information of the first bitstream. In other words, the first sigma tensor contains the variances and standard deviations of the samples of the first bitstream.

[0581] 1640. Obtaining the sigma index for the first sigma value of the first sigma tensor based on the quantization process;

[0582] Here, the sigma index represents one of the probability distributions in the probability distribution table, and the minimum sigma value of the quantization process is set to be equal to the skip threshold, which is used in the skip process to obtain the skip mask.

[0583] A skip mask can also be called a sigma mask. Skips are widely used to improve the throughput of arithmetic coders. The skip logic results in not encoding some elements that have a variance / standard deviation lower than the skip threshold. It can be understood that if the sigma value is incorrect (e.g., too small), the skip process will be disabled based on the cube_flag. If the skip process is enabled, the sigma will be affected by the skip process. ” All residual elements that are <(threshold_skip) are not coded and are set to 0, which is σ ” <(threshold_skip) means that Sigma_Idx is never used. Therefore, σ min If it is set to less than (threshold_skip), σ min The σ between (threshold_skip) and ” The sigma_index for this is never used, which leads to wasted codec resources and decreased coding efficiency, particularly resulting in improper allocation of memory resources. Therefore, this application is σ min However, it is suggested that this be modified to be equal to threshold_skip in the skip logic (something like a threshold used in the skip mask process). In one particular embodiment, σ minis (threshold_skip). In one specific embodiment, threshold_skip is set to a default value of 0.2, as a result, σ min can be corrected to 0.2. When threshold_skip is set to other default values, σ min shall be corrected to be equal to the default value of threshold_skip.

[0584] 1650. Entropy decoding the first bitstream based on the sigma index to obtain a sequence of decoded symbols;

[0585] 1660. Processing the sequence of decoded symbols based on a skip mask to obtain a residual tensor, wherein the residual tensor is used to obtain a reconstructed image.

[0586] Based on the skip mask, the decoder can know which symbols are coded and which are not coded and set to 0. Accordingly, the decoder can reconstruct the residual tensor.

[0587] In one embodiment, the skip mask is used to indicate which elements of a second tensor are present in a second bitstream, wherein the second tensor is based on sigma values derived from a first sigma tensor.

[0588] In one embodiment, the minimum sigma value of the quantization process is set equal to 0.2.

[0589] In one embodiment, the minimum sigma value of the quantization process is set equal to threshold_skip<<shift, where threshold_skip indicates a skip threshold and shift is used.

[0590] In one embodiment, the maximum sigma value of the quantization process is set equal to 30.

[0591] In one embodiment, the maximum sigma value of the quantization process is set to a value within the range of 30 to 64.

[0592] In one embodiment, the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within the range from the minimum sigma value to the maximum sigma value.

[0593] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

[0594] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within that range.

[0595] In one embodiment, the sigma number of the quantization process is set to be equal to 32.

[0596] In one embodiment, obtaining a first sigma tensor based on a second bitstream is: Entropy-decode the second bitstream and obtain the decoded hyperpre-tensor; To obtain the second sigma tensor, process the decoded hyperpre-tensor based on a hyperscale decoder; This involves scaling the second sigma tensor to obtain the first sigma tensor.

[0597] In one embodiment, the first sigma tensor includes sigma values ​​or standard deviation values ​​in the logarithmic domain.

[0598] In one embodiment, the output of the hyperscale decoder is a log-domain sigma tensor, and the first sigma tensor is a scaled log-domain sigma tensor.

[0599] In one embodiment, the first sigma value is the standard deviation of the linear domain or the variance of the linear domain.

[0600] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0601] This includes obtaining the sigma index based on the following formula:

[0602]

number

[0603] Here, sigma_idx represents the sigma index, and I σ ” represents the first sigma value in the logarithmic domain, or I σ ” represents the standard deviation in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ It is equal to 32.

[0604] In one embodiment, based on a quantization process, a sigma index for a first sigma value is obtained:

[0605] This includes the step of obtaining the sigma index based on the following formula:

[0606]

number

[0607]

number

[0608] Here, sigma_idx represents the sigma index, σ ” represents the first sigma value in the linear domain, σ min σ is the minimum sigma value,max This is the maximum sigma value, and N σ This is a sigma number.

[0609] In one embodiment, N σ It is equal to 32.

[0610] In one embodiment, σ min It is equal to 0.2.

[0611] In one embodiment, σ max This value is within the range of 30 to 64.

[0612] One embodiment of the present invention discloses a method for encoding a bitstream as shown in Figure 17, the method 1700 including:

[0613] 1710. Obtaining hyperprior information from image data;

[0614] On the encoder side, the image data and hyperprior information are encoded into two bitstreams.

[0615] 1720. Obtaining the first sigma tensor based on hyperprior information;

[0616] The first sigma tensor contains several sigma values ​​or standard deviations, where the sigma values ​​are variances related to the probability distribution information of the image data; and the standard deviations are related to the probability distribution information of the first bitstream. In other words, the first sigma tensor contains the variances and standard deviations of the samples of the first bitstream.

[0617] 1730. Obtaining the sigma index for the first sigma value of the first sigma tensor based on the quantization process;

[0618] The sigma index indicates one of the probability distributions in the probability distribution table, the minimum sigma value of the quantization process is set equal to the skip threshold, and the skip threshold is used in the skip process to obtain a skip mask. Skipping is widely used to improve the throughput of arithmetic coders. Skip logic results in not encoding some elements that have a variance / standard deviation lower than the skip threshold. It can be appreciated that if the sigma value is incorrect (e.g., too small), the skip process is disabled based on cube_flag. When the skip process is enabled, due to the skip process, σ ” <(threshold_skip<<shift), all residual elements are not coded and are set to 0, which means σ ” <(threshold_skip<<shift) means that Sigma_Idx is never used. Therefore, when σ min is set to be less than (threshold_skip<<shift), σ min and σ between (threshold_skip<<shift) ” the sigma index for is never used, which causes waste of codec resources and reduction in coding efficiency, and in particular leads to inappropriate occupation of memory resources. Accordingly, the present application proposes that σ min is modified to be equal to threshold_skip in skip logic (such as the threshold used in the skip mask process). In one specific embodiment, σ min =(threshold_skip<<shift). In one specific embodiment, threshold_skip is set as a default value of 0.2, as a result of which σ min can be modified to 0.2. When threshold_skip is set as another default value, σ minThis should be modified to be equal to the default value of threshold_skip.

[0619] 1740. Processing a sequence of symbols based on a skip mask to obtain an encoded sequence of symbols;

[0620] 1750. To obtain the first bitstream, the sequence of encoded symbols is entropy-encoded based on the sigma index.

[0621] In one embodiment, the maximum sigma value of the quantization process is set to a value within the range of 30 to 64.

[0622] In one embodiment, the maximum sigma value of the quantization process is set to be equal to 30.

[0623] In one embodiment, the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within a sigma range from the minimum sigma value to the maximum sigma value.

[0624] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

[0625] In one embodiment, the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within that range.

[0626] In one embodiment, the sigma number of the quantization process is set to be equal to 32.

[0627] In one embodiment, the first sigma tensor includes sigma values ​​or standard deviation values ​​in the logarithmic domain.

[0628] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0629] This includes the step of obtaining the sigma index based on the following formula:

[0630]

number

[0631] Here, sigma_idx represents the sigma index, and I σ ” represents the first sigma value in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ It is equal to 32.

[0632] In one embodiment, the input to the sigma quantization process is the standard deviation in the linear or logarithmic domain, or in other words, the input to the sigma quantization process is the logarithmic domain standard deviation tensor I ” σ Furthermore, the sigma quantization process converts the logarithmic domain standard deviation tensor into a sigma index.

[0633] In one embodiment, the first sigma value is the standard deviation or variance of the log domain.

[0634] In one embodiment, the first sigma value is the standard deviation of the linear domain or the variance of the linear domain.

[0635] In one embodiment, based on a quantization process, a sigma index for a first sigma value can be obtained:

[0636] This includes obtaining the sigma index based on the following formula:

[0637]

number

[0638]

number

[0639] sigma_idx represents the sigma index, σ ” represents the first sigma value in the linear domain, σ min σ is the minimum sigma value, max This is the maximum sigma value, and N σ This is a sigma number.

[0640] In one embodiment, N σ It is equal to 32.

[0641] In one embodiment, σ min It is equal to 0.2.

[0642] In one embodiment, σ max This value is within the range of 30 to 64.

[0643] Figure 18 shows an exemplary block diagram of a decoder structure, the decoder 1800 comprising a storage medium 1801 and one or more processors 1802, the storage medium 1801 configured to store computer executable instructions, and the one or more processors 1802 configured to perform any one of the aforementioned methods 1600 and related embodiments.

[0644] Figure 19 shows an exemplary block diagram of an encoder structure, the encoder 1900 comprising a storage medium 1901 and one or more processors 1902, the storage medium 1901 being configured to store computer executable instructions, and the one or more processors 1902 being configured to perform any one of the aforementioned methods 1700 and related embodiments.

[0645] Figure 20 shows an exemplary block diagram of a decoder structure, the decoder 2000 comprising a receiver 2001, a transmitter 2002, and one or more processors 2003, wherein the receiver 2001 is configured to receive a first bitstream and a second bitstream, the transmitter 2002 is configured to output a decoded picture, and the one or more processors 2003 are configured to perform any one of the aforementioned methods 1600 and related embodiments.

[0646] Figure 21 shows an exemplary block diagram of an encoder structure, the encoder 2100 comprising a receiver 2101, a transmitter 2102, and one or more processors 2103, the receiver 2101 being configured to receive a picture, the transmitter 2102 being configured to output a bitstream, and the one or more processors 2103 being configured to perform any one of the aforementioned methods 1700 and related embodiments.

[0647] Embodiments of the present invention disclose a computer program product that, when executed on a computer system, includes computer-executable instructions that cause the computer system to perform a method according to any one of the embodiments described above.

[0648] Embodiments of the present invention disclose a computer program stored in a storage medium, which, when executed on a computer or processor, includes a computer-executable instruction causing the computer or processor to execute any one of methods 1600 or 1700, or any one of the related embodiments.

[0649] Embodiments of the present invention disclose a computer-readable storage medium comprising computer-executable instructions, which, when executed in a computer or processor, cause the computer or processor to perform a method according to Method 1600 or any one of the related embodiments.

[0650] Embodiments of the present invention disclose a computer-readable storage medium comprising computer-executable instructions, which, when executed in a computer or processor, cause the computer or processor to perform a method according to Method 1700 or any one of the related embodiments.

[0651] Embodiments of the present invention disclose that the decoder includes functional units corresponding to the method steps 1610 to 1660 described above, for example, the decoder includes a first acquisition unit for carrying out step 1610, a second acquisition unit for carrying out step 1620, a sigma tensor acquisition unit for carrying out step 1630, a sigma quantization unit for carrying out step 1640, an entropy decoding unit for carrying out step 1650, and a decoder skip unit for carrying out step 1660.

[0652] Embodiments of the present invention disclose that the encoder includes functional units corresponding to the method steps 1710 to 1750 described above, for example, the encoder includes a first acquisition unit for performing step 1710, a sigma tensor acquisition unit for performing step 1720, a sigma quantization unit for performing step 1730, an encoder skip unit for performing step 1740, and an entropy coding unit for performing step 1750.

[0653] For the purposes of this specification, the following terms and definitions apply.

[0654] Padding layer

[0655] The padding layer is Padd(H in ,W in ,d,s d ) is shown as, where H in ,W in is the input for the tensor height and width-analysis transformation, and s d d is the stride of the convolution in progress, and d is the depth of the convolutional layer in the deep learnable encoder. The padding layer is sized [C,h]. d-1 ,w d-1 It receives a tensor of size [C,s d h d ,s d w d Output the tensor of ], where h d =ceil(h d-1 / s d );w d =ceil(w d-1 / s d ), h0=H in ,w0=W in By default, padding is performed by duplication. It is possible to specify a different padding model (for example, zero-padding).

[0656] Cropping layer

[0657] The cropping layer is Crop(H in ,W in ,d,s d ) is shown as, where H in ,W in is the input for the tensor height and width-composition transformation, and s d d is the stride of the ongoing transposed convolution, and d is the depth of the convolutional layer in the deep-learnable reconstruction process. The cropping layer is of size [C,s d h d ,s d w d It receives a tensor of size [C,h d-1 ,w d-1 Output the tensor of ], where h d =ceil(hd-1 / s d );w d =ceil(w d-1 / s d ), h0=H in ,w0=W in Padding is performed by discarding redundant elements.

[0658] Convolutional layer

[0659] Two-dimensional convolution is CONV(K ver ×K hor ,C in ,C out It is shown as ,s↓). Convolution is size [C in ,h in ,w in It receives a tensor of size [C out ,h out ,w out Output the tensor of ], where h in =s·h out ;w in =s·w out The factor s is called the stride. If the stride argument is missing, no change is made to the spatial resolution.

[0660] Inversion and convolution

[0661] Transposition convolution is CONV -1 (K ver ×K hor ,C in ,C out It is shown as ,s↑). The transpose convolution is of size [C in ,h in ,w in It receives a tensor of size [C out ,h out ,w out Output the tensor of ], where h out =s·h in ;w out =s·w in Factor s is called stride.

[0662] Quantized convolution

[0663] Two-dimensional quantized convolution is qCONV(K ver ×K hor ,C in ,C out The convolution process is represented as ,s↓,d,p). The size [C in ,h in ,w in It receives an integer tensor of size [C]. out ,h out ,w out Output the integer tensor of ], where h in =s·h out ;w in =s·w out The factor s is called the stride. If the stride argument is missing, it is assumed to be equal to 1 and no change in spatial resolution is made. The parameter d is a non-negative integer that determines the maximum size of the input tensor elements after clipping. Tensor p[C out ] includes the descale shift for each channel of the output tensor.

[0664] Quantized transpose convolution

[0665] Two-dimensional quantized transpose convolution is qCONV -1 (K ver ×K hor ,C in ,C out It is represented as ,s↑,d,p). The convolution process is of size [C in ,h in ,w in It receives a 16-bit integer tensor of size [C]. out ,h out ,w out Output the integer tensor of ], where h out =s·h in ;w out =s·w in The factor s is called the stride. The parameter d is a non-negative integer that determines the maximum size of the input tensor elements after clipping. Tensor p[C out ] includes the descale shift for each channel of the output tensor.

[0666] Pixel Shuffle Layer

[0667] The pixel shuffle layer, also known as sub-pixel convolution, is denoted as PixelShuffle(s), where s>1 is the upscaling factor. This layer is [C in ,h in ,w in The elements in the tensor input of the shape ] are [C out ,h out ,w out This sorts the output into a tensor of the shape ], h out =s·h in ;w out =s·w in ;C out =C in / s 2 That is the case.

[0668] Residual activation unit

[0669] The residual activation unit is ResAU(K ver ×K hor This is shown as [C,h]. This layer has a size [C,h]. k ,w k After executing the sequence of steps shown in Figure 22A, the system receives a tensor of ] and outputs a tensor of the same size. Here,

number

number

[0670] residual activation

[0671] Residual activation is ResA(K ver ×K hor This is shown as [C,h]. This layer has a size [C,h].k ,w k After executing the sequence of steps shown in Figure 22B, the system receives a tensor of ] and outputs a tensor of the same size. Here,

number

[0672] Residuals, nonlocal attention block

[0673] Residual, nonlocal attention, and block are shown as RNAB(∝). This layer is size [C,h k ,w k After executing the sequence of steps shown in Figure 22C, the system receives a tensor of ] and outputs a tensor of the same size. Here,

number

number

[0674] Residual block

[0675] The residual block is shown as RB. This layer has a size [C,h k ,w k After executing the sequence of steps shown in Figure 22D, receive a tensor of size ] and output a tensor of the same size.

[0676] Lightweight residual block

[0677] Lightweight residual blocks are shown as LRBs. This layer has a size [C,hk ,w k After executing the sequence of steps shown in Figure 22E, receive a tensor of size ] and output a tensor of the same size.

[0678] Normalized Linear Unit

[0679] The normalized linear unit is denoted as ReLU(), which is the following element-wise function:

number

[0680] Leaky rectified linear unit

[0681] The leaky normalized linear unit is denoted as LeakyReLU(), which is the following element-wise function:

number

[0682] negative_slope=0.01

[0683] opIdx is an indicator of the operating point, where 0 represents the "basic" operating point and 1 represents the "higher" operating point.

[0684] ABS calculation

[0685] The abs operation is denoted as ABS(). This is the following element-wise function:

[0686] ABS(x) = abs(x) = |x|

[0687] mathematical operators The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are more precisely defined, and additional operations such as exponentiation and real-valued division are defined. The numbering and counting conventions generally start from 0, for example, "1st" corresponds to the 0th, "2nd" to the 1st, and so on.

[0688] Arithmetic operators The following arithmetic operators are defined as follows: [Table 1]

[0689] Logical operators The following logical operators are defined as follows: [Table 2]

[0690] Relational operators The following relational operators are defined as follows: [Table 3] When a relational operator is applied to a syntax element or variable assigned the value "na" (not applicable), the value "na" is treated as a separate value for the syntax element or variable. The value "na" is considered not to be equal to any other value.

[0691] Bitwise operators The following bitwise operators are defined as follows: [Table 4]

[0692] Assignment operator The following arithmetic operators are defined as follows: [Table 5]

[0693] Range markings Use the following notation to specify a range of values: [Table 6]

[0694] mathematical function The following mathematical function is defined: [Table 7]

[0695] Operator precedence If the precedence in an expression is not explicitly indicated by parentheses, the following rules apply: - Operations with higher priority are evaluated before any operations with lower priority. - Operations with the same priority are evaluated from left to right.

[0696] The following table defines the order of operations from highest to lowest: higher positions in the table indicate higher priority. With respect to operators also used in the C programming language, the precedence used herein is the same as that used in the C programming language. Table: Order of operations from highest (top of the table) to lowest (bottom of the table) [Table 8]

[0697] Text description of logical operations In the text, logical operation statements are mathematically described in the following format: if( Condition 0) Statement 0 else if (Condition 1) Statement 1 ... else / * Reference notes for the remaining conditions * / statement n It may also be written in the following way: ... The following is the case / ... The following applies: - If condition 0, then statement 0 - Otherwise, if condition 1 is true, then statement 1 - ... - Otherwise (reference notes for the remaining conditions), statement n

[0698] Each "if...otherwise, if...otherwise..." statement in the text is introduced with "...as follows" or "...the following applies" immediately preceding the "if...". The final condition of "if...otherwise, if...otherwise..." is always "otherwise,...". Interleaved "if...otherwise, if...otherwise..." statements can be identified by matching the "...as follows" or "...the following applies" with the final "otherwise,...".

[0699] In the text, logical operation statements are mathematically described in the following format: if( Condition 0a && Condition 0b) Statement 0 else if( condition 1a | | condition 1b ) Statement 1 ... else statement n It may also be written in the following way: ... The following is the case / ... The following applies: - If all of the following conditions are true, then statement 0: - Condition 0a - Condition 0b - Otherwise, if one or more of the following conditions are true, then statement 1: - Condition 1a - Condition 1b ... - Otherwise, statement n In the text, logical operation statements are mathematically described in the following format: if( Condition 0) Statement 0 if( Condition 1) Statement 1 It may also be written in the following way: If condition 0, statement 0 If condition 1 is met, statement 1

[0700] While embodiments of the present invention have been described primarily in terms of video coding, it should be noted that embodiments of the coding system 10, encoder 20, and decoder 30 (and corresponding systems 10), as well as other embodiments described herein, may be configured for still image processing or coding, i.e., for processing or coding individual pictures that are independent of any preceding or consecutive pictures, as in video coding. Generally, if picture processing coding is limited to a single picture 17, only the interpretation units 244 (encoder) and 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and video decoder 30 may be equally used for still image processing, e.g., residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, splitting 262 / 362, intra prediction 254 / 354, and / or loop filtering 220, 320, as well as entropy coding 270 and entropy decoding 304. In general, embodiments of the present disclosure may be applied to audio signals or other source signals such as the like.

[0701] For example, embodiments of the encoder 20 and decoder 30, and the functions described herein with reference to, for example, the encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted over a communication medium and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that facilitates the transfer of computer programs from one location to another, for example, according to a communication protocol. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for implementing the techniques described herein. Computer program products may include computer-readable media.

[0702] Such computer-readable storage media may include, but are not limited to, examples, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary, tangible storage media. The terms "disk" and "disc" used in this context include compact discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks, and Blu-ray discs, where a "disk" typically reproduces data magnetically, while a "disc" reproduces data optically using a laser. Any combination of these should also be included within the scope of computer-readable media.

[0703] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or individual logic circuits. Therefore, the term “processor” as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. The techniques can also be fully implemented in one or more circuits or logic elements.

[0704] The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight the functional aspects of a device configured to perform the techniques disclosed, but do not necessarily require implementation by various hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, along with suitable software and / or firmware.

Claims

1. A method for decoding a bitstream: Steps to obtain the first bitstream; A step of obtaining a second bitstream, wherein the second bitstream includes hyperpre-information of the first bitstream; A step of obtaining a first sigma tensor based on the second bitstream, wherein the first sigma tensor includes several sigma values, the sigma values ​​being variances or standard deviations, and the variances or standard deviations relating to the probability distribution information of the first bitstream; A step of obtaining a sigma index for a first sigma value of the first sigma tensor based on a quantization process, wherein the sigma index represents one of the probability distributions in a probability distribution table, the minimum sigma value of the quantization process is set to be equal to a skip threshold, and the skip threshold is used in the skip process to obtain a skip mask; The steps include entropy decoding the first bitstream based on the sigma index and obtaining a sequence of decoded symbols; A step of processing the sequence of decoded symbols based on the skip mask in order to obtain a residual tensor, wherein the residual tensor is used to obtain a reconstructed image; A method that includes this.

2. A method according to claim 1, wherein the minimum sigma value of the quantization process is set to be equal to 0.

2.

3. A method according to claim 1 or 2, wherein the minimum sigma value of the quantization process is set to be equal to threshold_skip, where threshold_skip indicates the skip threshold.

4. A method according to any one of claims 1 to 3, wherein the maximum sigma value of the quantization process is set to be equal to a value in the range of 30 to 64.

5. A method according to any one of claims 1 to 4, wherein the maximum sigma value of the quantization process is set to be equal to 30.

6. A method according to any one of claims 1 to 5, wherein the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within the range from the minimum sigma value to the maximum sigma value.

7. A method according to any one of claims 1 to 6, wherein the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

8. A method according to claim 6, wherein the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within the range.

9. A method according to claim 7 or 8, wherein the sigma number of the quantization process is set to be equal to 32.

10. The method according to any one of claims 1 to 9, wherein the step of obtaining a first sigma tensor based on the second bitstream is: The steps include entropy decoding the second bitstream and obtaining the decoded hyperpre-tensor; To obtain a second sigma tensor, the steps include processing the decoded hyperpre-tensor based on a hyperscale decoder; The step of scaling the second sigma tensor in order to obtain the first sigma tensor; A method that includes this.

11. A method according to any one of claims 1 to 10, wherein the first sigma value is the standard deviation of the log domain or the variance of the log domain.

12. A method according to any one of claims 1 to 10, wherein the first sigma value is the standard deviation of a linear domain or the variance of a linear domain.

13. The method according to any one of claims 1 to 12, wherein the step of obtaining a sigma index for the first sigma value based on the quantization process is: The step includes obtaining the sigma index based on the following formula: [Math 1] The aforementioned sigma_idx represents the sigma index, and the aforementioned I σ ” represents the first sigma value in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ The method is equal to 32.

14. The method according to any one of claims 1 to 12, wherein the step of obtaining a sigma index for the first sigma value based on the quantization process is: The step includes obtaining the sigma index based on the following formula: [Math 2] The sigma_idx represents the sigma index, and the σ'' represents the first sigma value in the linear domain. min This is the minimum sigma value, and the σ max This is the maximum sigma value, and the above N σ The method is that it is a sigma number.

15. In the method according to claim 14, N σ The method is equal to 32.

16. In the method according to claim 14 or 15, σ min The method is equal to 0.

2.

17. The method according to any one of claims 14 to 16, wherein σ max is a value within the range of 30 to 64.

18. A method for encoding a bitstream: Steps to obtain hyperprior information for image data; A step of obtaining a first sigma tensor based on the hyperprior information, wherein the first sigma tensor includes several sigma values, the sigma values ​​being variances or standard deviations, and the variances or standard deviations relating to the probability distribution information of the image data; A step of obtaining a sigma index for a first sigma value of the first sigma tensor based on a quantization process, wherein the sigma index represents one of the probability distributions in a probability distribution table, the minimum sigma value of the quantization process is set to be equal to a skip threshold, and the skip threshold is used in the skip process to obtain a skip mask; A step of processing the sequence of symbols based on the skip mask in order to obtain the sequence of encoded symbols; To obtain a first bitstream, the steps include entropy encoding the sequence of encoded symbols based on the sigma index; A method that includes this.

19. A method according to claim 18, wherein the minimum sigma value of the quantization process is set to be equal to 0.

2.

20. A method according to claim 18 or 19, wherein the minimum sigma value of the quantization process is set to be equal to threshold_skip, where threshold_skip indicates the skip threshold.

21. A method according to any one of claims 18 to 20, wherein the maximum sigma value of the quantization process is set to be equal to a value in the range of 30 to 64.

22. A method according to any one of claims 18 to 21, wherein the maximum sigma value of the quantization process is set to be equal to 30.

23. A method according to any one of claims 18 to 22, wherein the quantization process quantizes the sigma values ​​in the first sigma tensor to sigma values ​​among several discrete values ​​within a range of sigma from the minimum sigma value to the maximum sigma value.

24. A method according to any one of claims 18 to 23, wherein the sigma number of the quantization process is set to a power of 2, and the sigma number represents the number of probability distributions in the probability distribution table.

25. A method according to claim 23, wherein the sigma number of the quantization process is set as a power of 2, and the sigma number represents the number of discrete values ​​within the range.

26. A method according to claim 24 or 25, wherein the sigma number of the quantization process is set to be equal to 32.

27. A method according to any one of claims 18 to 26, wherein the first sigma value is the standard deviation of the log domain or the variance of the log domain.

28. A method according to any one of claims 18 to 26, wherein the first sigma value is the standard deviation of a linear domain or the variance of a linear domain.

29. The method according to any one of claims 18 to 28, wherein the step of obtaining a sigma index for the first sigma value based on the quantization process is: The step includes obtaining the sigma index based on the following formula: [Math 3] The aforementioned sigma_idx represents the sigma index, and the aforementioned I σ ” represents the first sigma value in the logarithmic domain, where sigmaPrecision is equal to 7, and N σ The method is equal to 32.

30. The method according to any one of claims 18 to 28, wherein the step of obtaining a sigma index for the first sigma value based on the quantization process is: The step includes obtaining the sigma index based on the following formula: [Math 4] The sigma_idx represents the sigma index, and the σ'' represents the first sigma value in the linear domain. min This is the minimum sigma value, and the σ max This is the maximum sigma value, and N σ The method is that it is a sigma number.

31. In the method according to claim 30, N σ The method is equal to 32.

32. In the method according to claim 30 or 31, σ min The method is equal to 0.

2.

33. In the method according to any one of claims 30 to 32, σ max The method is a value within the range of 30 to 64.

34. A decoder for processing a bitstream, comprising a storage medium and one or more processors, wherein the storage medium is configured to store computer executable instructions, and the one or more processors are configured to perform the method according to any one of claims 1 to 17.

35. An encoder for processing a bitstream, comprising a storage medium and one or more processors, wherein the storage medium is configured to store computer executable instructions, and the one or more processors are configured to perform the method according to any one of claims 18 to 33.

36. A computer-readable storage medium comprising computer-executable instructions, wherein, when executed by a computer or processor, the computer-executable instructions cause the computer or processor to perform the method according to any one of claims 1 to 17.

37. A computer-readable storage medium comprising computer-executable instructions, wherein, when executed in a computer or processor, the computer-executable instructions cause the computer or processor to perform the method described in any one of claims 18 to 33.

38. An encoder for encoding a picture, comprising a receiver for receiving a picture, a transmitter for outputting a bitstream, and one or more processors configured to perform the method described in any one of claims 18 to 33.

39. A decoder for decoding a bitstream representing a picture, comprising: a receiver that receives a first bitstream and a second bitstream; a transmitter that outputs a decoded picture; and one or more processors configured to perform the method according to any one of claims 1 to 17.

40. A computer program stored on a storage medium, which, when executed on a computer or processor, includes a computer-executable instruction that causes the computer or processor to perform the method described in any one of claims 1 to 17 or any one of claims 18 to 33.

Images