Method, apparatus and medium for visual data processing

Abstract

The embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is proposed. The method includes obtaining a first representation of visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data, adjusting a plurality of sets of first samples of the first representation with different parameters, and performing a conversion based on the plurality of sets of adjusted first samples, for conversion between the visual data and a bitstream of visual data.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate generally to visual data processing techniques, and more particularly to neural network-based visual data encoding. [Background technology]

[0002] In the past decade, deep learning has been rapidly developed in various fields, especially in computer vision and image processing. Neural networks were originally invented by interdisciplinary research of neuroscience and mathematics. It has shown powerful abilities in terms of nonlinear transformation and classification. Neural network-based image / video compression technology has made great progress in the past five years. It has been reported that the latest neural network-based image compression algorithms have achieved rate-distortion (RD) performance comparable to that of Versatile Video Coding (VVC). As the performance of neural image compression is continuously improved, neural network-based video compression has become an actively developing research field. However, the coding efficiency and / or coding quality of neural network-based visual data coding are generally expected to be further improved. Summary of the Invention

[0003] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is proposed, for conversion between the visual data and a bitstream of visual data, the method comprising the steps of obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data, tuning a plurality of sets of first samples of the first representation with different parameters, and performing a conversion based on the plurality of sets of tuned first samples.

[0005] According to a method according to a first aspect of the present disclosure, samples of a first representation of visual data (such as a quantized latent representation or a quantized residual latent representation) are split into multiple sets, and the multiple sets of first samples are modulated with different parameters. This allows the proposed method to quantize samples with different parameters, which can maximize the quality of the reconstructed visual data while reducing the number of transmitted bits. In this way, the proposed method can advantageously improve coding quality and coding efficiency.

[0006] In a second aspect, an apparatus for visual data processing is proposed, the apparatus comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is proposed, the non-transitory computer-readable storage medium storing instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is proposed, which stores a bitstream of visual data generated by a method executed by an apparatus for visual data processing, the method comprising: obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; tuning a plurality of sets of first samples of the first representation with different parameters; and generating a bitstream based on the plurality of sets of tuned first samples.

[0009] In a fifth aspect, a method for storing a bitstream of visual data is proposed, the method comprising the steps of obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data, adjusting a plurality of sets of first samples of the first representation with different parameters, generating a bitstream based on the plurality of sets of adjusted first samples, and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] This Summary is provided to introduce in a simplified form some of the concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0011] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals generally refer to like components. [Figure 1] FIG. 1 shows a block diagram illustrating an example visual data encoding system according to some embodiments of the present disclosure. [Diagram 2] A typical transform coding method is shown below. [Diagram 3] 1 shows an image from the Kodak dataset and different representations of that image. [Figure 4] We present an autoencoder network architecture that implements a hyperprior model. [Diagram 5] A block diagram of a composite model is shown. [Figure 6] 1 shows the encoding process of a composite model. [Figure 7] 4 shows a decoding process for a composite model. [Figure 8] Two possible implementations of the entropy coding and quantization process are shown. [Figure 9] Here is a decoder implementation that corresponds to the encoder implementation. [Figure 10] 1 illustrates two possible encoder implementations according to embodiments of the present disclosure. [Figure 11] 1 illustrates two possible decoder implementations according to embodiments of the present disclosure. [Figure 12] 1 illustrates an encoder implementation according to an embodiment of the present disclosure. [Figure 13] 1 illustrates a decoder implementation according to an embodiment of the present disclosure. [Figure 14] 1 illustrates two possible probability distributions according to an embodiment of the present disclosure. [Figure 15] 15 illustrates an example of an exemplary visual data encoding process 1500 according to some embodiments of the present disclosure. [Figure 16] 16 illustrates an example visual data decoding process 1600 according to some embodiments of the present disclosure. [Figure 17] 17 shows a flowchart of a method 1700 for visual data processing according to some embodiments of the present disclosure. [Figure 18] FIG. 1 illustrates a block diagram of a computing device in which various embodiments of the present disclosure may be implemented.

[0012] Throughout the drawings, the same or similar reference numbers typically refer to the same or similar elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Next, the principles of the present disclosure will be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only, to help those skilled in the art understand and practice the present disclosure, and do not imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various forms other than those described below.

[0014] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0015] References in this disclosure to "one embodiment," "an embodiment," "an exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments need to include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an exemplary embodiment, it is submitted that it is within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0016] Although terms such as "first" and "second" may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It is further understood that as used herein, the terms "comprises," "comprising," "has," "having," "includes," and / or "including" specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0018] Example Environment 1 is a block diagram illustrating an example visual data encoding system 100 that may utilize techniques of this disclosure. As shown, the visual data encoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a visual data encoding device, and the destination device 120 may also be referred to as a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0019] Visual data sources 112 may include sources such as visual data capture devices, examples of which include, but are not limited to, an interface for receiving visual data from a visual data provider, a computer graphics system for generating visual data, and / or combinations thereof.

[0020] The visual data may include one or more pictures or one or more images of a video. The visual data encoder 114 encodes the visual data from the visual data source 112 to generate a bitstream. The bitstream may include a bit sequence that forms an encoded representation of the visual data. The bitstream may include an encoded picture and associated visual data. An encoded picture is an encoded representation of a picture. The associated visual data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded visual data may be transmitted directly to the destination device 120 through the I / O interface 116 over the network 130A. The encoded visual data may also be stored in a storage medium / server 130B for access by the destination device 120.

[0021] The destination device 120 may include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded visual data from the source device 110 or the storage medium / server 130B. The visual data decoder 124 may decode the encoded visual data. The display device 122 may display the decoded visual data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 configured to interface with an external display device.

[0022] The visual data encoder 114 and the visual data decoder 124 may operate in accordance with a visual data encoding standard, such as a video encoding standard or a still image encoding standard, and other current and / or additional standards.

[0023] Some exemplary embodiments of the present disclosure are described in detail below. It should be understood that section headings are used herein for ease of understanding and are not intended to limit the embodiments disclosed in a section to that section alone. Furthermore, although certain embodiments are described with reference to Versatile Video Coding or other specific visual data codecs, the disclosed techniques are applicable to other encoding techniques. Furthermore, although some embodiments describe an encoding step in detail, it should be understood that the corresponding step of decoding that undoes the encoding is performed by a decoder. Furthermore, the term visual data processing encompasses visual data encoding or compression, visual data decoding or decompression, and visual data transcoding, where visual data is represented from one compressed format to another compressed format or at a different compressed bit rate. [Table 1] TIFF2025508529000003.tif245169TIFF2025508529000004.tif238169TIFF2025508529000005.tif238169TIFF20255085290 00006.tif243169TIFF2025508529000007.tif238169TIFF2025508529000008.tif244169TIFF2025508529000009.tif240169 TIFF2025508529000010.tif242167TIFF2025508529000011.tif233165TIFF2025508529000012.tif239169TIFF20255085290 00013.tif249169TIFF2025508529000014.tif250169TIFF2025508529000015.tif230169TIFF2025508529000016.tif223169 TIFF2025508529000017.tif243169TIFF2025508529000018.tif242170TIFF2025508529000019.tif236169TIFF20255085290 00020.tif246169TIFF2025508529000021.tif237169TIFF2025508529000022.tif234161TIFF2025508529000023.tif225169 TIFF2025508529000024.tif239169TIFF2025508529000025.tif243169TIFF2025508529000026.tif238169TIFF20255085290 00027.tif243169TIFF2025508529000028.tif244169TIFF2025508529000029.tif243166TIFF2025508529000030.tif178169

[0024] In the following, details of embodiments of the present disclosure relating to neural network-based visual data encoding are described. As used herein, the term "visual data" may refer to images, pictures in a video, or any other visual data suitable for being encoded.

[0025] As described above, in existing designs, the quantization process is applied to samples of the latent representation y of the visual data or the residual latent representation w by using a predefined quantization step size that is constant for all samples. However, not all samples are equally important, and some samples may have a greater impact on the quality of the reconstructed visual data than other samples. Existing designs do not allow for different quantization precisions to be applied to different samples of the latent representation y of the visual data or the residual latent representation w.

[0026] To solve the above problems and some other problems not mentioned, the visual data processing solution described below is disclosed.

[0027] 15 illustrates an example visual data encoding process 1500 according to some embodiments of the present disclosure. For example, the visual data encoding process 1500 may be performed by the visual data encoder 114, as shown in FIG. It should be understood that the visual data encoding process 1500 may also include additional blocks not shown and / or blocks that are shown may be omitted. The scope of the present disclosure is not limited in this respect.

[0028] As shown in FIG. 15, in a first neural network 1512, an analytical transformation may be performed on the visual data 1510 to obtain a latent representation (denoted as y in FIG. 15) of the visual data 1510. The visual data 1510 may include one or more pictures in an image or video. In block 1514, a first statistic is subtracted from the latent representation to obtain a residual latent representation (denoted as w in FIG. 15). The first statistic may be generated by a second neural network 1522 and may indicate a prediction of the latent representation. As a non-limiting example, the first statistic may be a mean (denoted as μ in FIG. 15) of a probability distribution, such as a Gaussian probability distribution. For example, the probability distribution may describe a probability distribution of values ​​of one or more samples of the latent representation. It should be appreciated that the statistic may also be referred to as a statistical parameter, a probability parameter, a probability distribution parameter, or the like. The scope of the present disclosure is not limited in this respect.

[0029] The first statistical value may be generated by the second neural network 1522. For example, the second neural network 1522 may be referred to as an estimation model. Note that in this application, a model may also be referred to as a module. That is, the second neural network may also be referred to as a first module, and the estimation model may also be referred to as an estimation module. The input of the second neural network 1522 may include a latent representation, and the output of the second neural network 1522 may include at least one statistical value. In one example, the second neural network 1522 may include a neural network-based sub-network. In another example, the second neural network 1522 may include a first sub-network for generating the first statistical value and a second sub-network for generating the second statistical value. As a non-limiting example, the second statistical value may be the variance (denoted as σ in FIG. 15 ) of a probability distribution such as a Gaussian probability distribution. For example, the probability distribution may describe the probability distribution of values ​​of one or more samples of the latent representation. The first sub-network may be a hyperdecoder sub-network and the second sub-network may be a hyperscale decoder sub-network. In one example, the second neural network 1522 may further include a hyperencoder sub-network. It should be appreciated that the above-mentioned mean and variance are only two illustrative examples of statistics that may be used. Any other suitable statistics, such as standard deviation, may also be used. The scope of the present disclosure is not limited in this respect.

[0030] In the scaling block 1524, the samples of the second statistics (hereinafter also referred to as "third samples") may be divided into a plurality of sets to obtain a plurality of sets of third samples. In one example, this division process may be performed at a sample level. That is, a division decision is made for each of the third samples. Alternatively, the division process may be performed at a block level. More specifically, the third samples of the second statistics may be grouped into a plurality of blocks. Each of the plurality of blocks may have a predetermined size of N×M, such as 8×8. The predetermined size may be indicated in the bitstream. The third samples may be divided based on a plurality of blocks. That is, a division decision is made for each of the plurality of blocks, and samples in the same block are divided into the same set.

[0031] In some embodiments, a single third sample of the second statistical value may be divided into one of a plurality of sets based on the sample of the first statistical value corresponding to the third sample and the single third sample itself. As a non-limiting example, the value of the sample of the first statistical value and the value of the single third sample itself may be compared to a first threshold and a second threshold, respectively. If the value of the sample of the first statistical value is less than the first threshold and the value of the single third sample is less than the second threshold, the third sample may be divided into the first set. Otherwise, the third sample may be divided into a second set different from the first set.

[0032] In another embodiment, an index of a single third sample may be compared to a third threshold during the splitting process. If the index is less than the third threshold, the third sample may be split into the first set. Otherwise, the third sample may be split into a second set different from the first set. The index of the sample may indicate a channel number of the sample, a feature map identifier of the sample, a spatial coordinate of the sample, etc. The above comparison may be performed for each of the third samples to split the third samples.

[0033] It should be appreciated that the single third sample may be split based on any other suitable parameter or value, such as at least one sample of at least one statistical value corresponding to the single third sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, or a function of the at least one sample. Section 4.3 above discusses this in more detail. In some embodiments, the threshold used may be indicated in the bitstream or may be predetermined.

[0034] In some alternative embodiments, when the splitting process is performed at the block level, the third samples in the block may be split into one of a plurality of sets based on a comparison of each of the third samples or each of the probability values ​​determined based on the respective third samples with a threshold. Alternatively, the third samples in the block may be split into one of a plurality of sets based on a comparison of the average of the third samples with a threshold. In a further example, the third samples in the block may be split into one of a plurality of sets based on a comparison of the maximum or minimum value of the third samples with a threshold. The above comparison may be performed for each of the plurality of blocks to split the third samples.

[0035] It should be understood that the third sample in the block may be divided based on any other suitable parameter or value, such as a set of samples of at least one reference statistical value corresponding to the third sample in the block, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the third sample in the third block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index. The metric may be an average, a maximum, or a minimum value.

[0036] In a scaling block 1524, the resulting third set of samples is further scaled with different parameters to produce scaled variances (σ s The second statistical value may be adjusted to generate an adjusted second statistical value, such as a fifth parameter (denoted as sigma ...

[0037] As shown in FIG. 15, the output of the scaling block 1524, i.e., the adjusted second statistics, may be used by the entropy coder. Note that in the following, the statistics used by the entropy coder (entropy encoder or entropy decoder, etc.) may also be called target statistics, and the statistics generated by the second neural network 1522 may also be called reference statistics. Also, when the statistics generated by the second neural network 1522 are directly used in the entropy coder without being adjusted, the statistics may also be called reference statistics and may also be called target statistics. In the example of FIG. 15, the mean μ and variance σ are reference statistics, and the adjusted variance σ are reference statistics. s is the target statistic.

[0038] In scaling block 1516, the samples of the residual latent representation (hereinafter also referred to as "second samples") may be split into multiple sets to obtain multiple sets of second samples. The second samples of the residual latent representation may be split at a sample level or block level in a manner similar to the splitting process described with respect to scaling block 1524. In addition to the reference statistics (such as mean μ and / or variance σ), the second samples of the residual latent representation may also be scaled to target statistics (such as adjusted variance σ) in a manner similar to the reference statistics. s It should be noted that the second sample may be divided based on a first set of second samples, etc. In one non-limiting example, the second sample may be divided in the same manner as the third sample. In other words, if the third sample is divided into a first set of third samples, then the second sample corresponding to the third sample may also be divided into the first set of second samples.

[0039] Further, the obtained multiple sets of second samples are further adjusted with different parameters, and multiple sets of adjusted second samples (w in FIG. 15) are obtained. s As a non-limiting example, a first set of second samples of the plurality of sets of second samples may be adjusted by scaling the first set of second samples with a third parameter, and a second set of second samples of the plurality of sets of second samples may be adjusted by scaling the second set of second samples with a fourth parameter that is different from the third parameter. In one example, one of the third parameter and the fourth parameter is 1. At least one of the third parameter or the fourth parameter may be signaled in the bitstream. It should be appreciated that in addition to scaling, the plurality of sets of second samples may be adjusted in any suitable manner, such as by adding a different parameter. The scope of the present disclosure is not limited in this respect.

[0040] In a quantization block 1518, the set of adjusted second samples is quantized to produce samples of the quantized residual latent representation (in FIG. 15

number

[0041] In the entropy coder 1520, the output of the quantization block 1518 (e.g., the quantized samples of the scaled residual latent representation) is subjected to a target statistic (e.g., the scaled variance σ s ), an entropy coding process is performed to generate at least a portion of a bitstream. The entropy coding process performed by the entropy encoder 1520 may be an arithmetic coding process, a Huffman coding process, etc. Additionally or alternatively, either the reference statistics or the target statistics may be encoded into the bitstream.

[0042] From the above, samples of the residual latent representation of visual data are split into multiple sets, and the multiple sets of samples are adjusted with different parameters. This allows the samples to be quantized with different parameters, maximizing the quality of the reconstructed visual data while reducing the number of transmitted bits. In this way, the proposed method can advantageously improve the coding quality and coding efficiency.

[0043] Although an example of a visual data encoding process has been described above with respect to Figure 15, it should be understood that other suitable variations of the visual data encoding process are also contemplated in view of the present disclosure. In another example, the latent representation may be trained rather than the residual latent representation, which is exemplarily shown in the left sub-diagram of Figure 12. In a further example, at least one statistic generated by the second neural network may be used directly by the entropy encoder without being trained, which is exemplarily shown in Figure 10.

[0044] 16 illustrates an example visual data decoding process 1600 according to some embodiments of the present disclosure. For example, the visual data decoding process 1600 may be performed by the visual data decoder 124 shown in FIG. It should be understood that the visual data decoding process 1600 may also include additional blocks not shown and / or blocks that are shown may be omitted. The scope of the present disclosure is not limited in this respect.

[0045] As shown in FIG. 16, the statistics (such as the mean μ and variance σ) may be generated by the second neural network 1622. For example, the second neural network 1622 may be referred to as an estimation model. The input of the second neural network 1622 may include a bit stream, and the output of the second neural network 1622 includes at least one statistical value. In one example, the second neural network 1622 may include a neural network-based sub-network. In another example, the second neural network 1622 may include a first sub-network for generating the first statistical value and a second sub-network for generating the second statistical value. The first sub-network may be a hyperdecoder sub-network, and the second sub-network may be a hyperscale decoder sub-network. It should be understood that the mean and variance mentioned above are only two illustrative examples of statistics that may be used. Other suitable statistics, such as standard deviation, may also be used. The scope of the present disclosure is not limited in this respect.

[0046] In the scaling block 1620, the samples of statistics may be divided and adjusted in a manner similar to the division and adjustment process described with respect to the scaling block 1524, and therefore will not be described in detail for the sake of brevity.

[0047] In the entropy decoder 1618, the samples of the quantized residual latent representation (ŵ in FIG. 16) are s An entropy decoding process may be performed on the bitstream based on at least one target statistic to obtain a target statistic (denoted as

[0048] In the scaling block 1616, the samples of the quantized residual latent representation (hereinafter also referred to as “first samples”) may be divided into multiple sets to obtain multiple sets of first samples. Also, the first samples of the quantized residual latent representation may be divided into multiple sets of first samples, as described with respect to the scaling block 1524. s In a manner similar to the splitting process, the split may be performed at the sample level or at the block level. In addition to the reference statistics (such as the mean μ and / or variance σ), the first sample of the residual latent representation may also be split at the target statistics (such as the adjusted variance σ s It should be noted that the first sample may be divided based on a first set of first samples, etc. In one non-limiting example, the first sample may be divided in the same manner as the third sample. In other words, if the third sample is divided into a first set of third samples, then the first sample corresponding to the third sample may also be divided into a first set of first samples.

[0049] Additionally, the resulting multiple sets of first samples may be further adjusted with a different parameter to generate multiple sets of adjusted first samples (denoted as ŵ in FIG. 16). As a non-limiting example, a first set of first samples of the multiple sets of first samples may be adjusted by scaling the first set of first samples with a first parameter, and a second set of first samples of the multiple sets of first samples may be adjusted by scaling the second set of first samples with a second parameter different from the first parameter. In one example, one of the first parameter and the second parameter may be 1, and at least one of the first parameter or the second parameter may be signaled in the bitstream. In some embodiments, the first parameter may be determined based on a third parameter, and the second parameter may be determined based on a fourth parameter. As a non-limiting example, the first parameter may be determined as the inverse of the third parameter, and the second parameter may be determined as the inverse of the fourth parameter. In some further embodiments, the fifth parameter may be the same as one of the first parameter or the second parameter, and the sixth parameter may be the same as the other one of the first parameter or the second parameter.

[0050] It should be appreciated that in addition to scaling, the multiple sets of first samples may be adjusted in any suitable manner, such as by adding different parameters, etc. The scope of the present disclosure is not limited in this respect.

[0051] In block 1614, the quantized latent representation (in FIG.

number

[0052] From the above, samples of the quantized residual latent representation of visual data are split into multiple sets, and the multiple sets of samples are adjusted with different parameters. This allows quantizing the samples with different parameters, which can maximize the quality of the reconstructed visual data while reducing the number of transmitted bits. In this way, the proposed method can advantageously improve the coding quality and coding efficiency.

[0053] Although an example of a visual data decoding process is described above with respect to Figure 16, it should be understood that other suitable variations of the visual data decoding process are also contemplated in view of the present disclosure. In another example, a latent representation may be used rather than a residual latent representation, as exemplarily shown in the lower sub-diagram of Figure 13. In a further example, at least one statistic generated by the second neural network may be used directly by the entropy decoder without adjustment, as exemplarily shown in Figure 11.

[0054] It should be understood that the above illustrations and / or examples are set forth for illustrative purposes only. The scope of the present disclosure is not limited in this respect. The embodiments of the present disclosure should be considered as examples for illustrating the general concept and should not be interpreted in a narrow sense. Moreover, these embodiments can be applied individually or combined in any manner.

[0055] 17 illustrates a flowchart of a method 1700 for visual data processing according to some embodiments of the present disclosure. The method 1700 may be implemented during conversion between visual data and a bitstream of visual data. The visual data may include video, pictures of a video, images, or any other suitable visual data.

[0056] As shown in FIG. 17, method 1700 begins at 1702 where a first representation of visual data is obtained for conversion between visual data and a bitstream of visual data. The first representation is obtained by quantizing a second representation of the visual data. The second representation is generated based on applying a first neural network to the visual data. In one example, the first neural network may be used to perform an analytical transformation on the visual data. As a non-limiting example, the second representation may be a latent representation of the visual data or a residual latent representation of the visual data.

[0057] At block 1704, the multiple sets of first samples of the first representation having different parameters are adjusted. As a non-limiting example, a first set of first samples of the multiple sets of first samples may be adjusted by scaling the first set of first samples with a first parameter. A second set of first samples of the multiple sets of first samples may be adjusted by scaling the second set of first samples with a second parameter that is different from the first parameter. It should be appreciated that in addition to scaling, the multiple sets of first samples may be adjusted in any suitable manner, such as by adding a different parameter. The scope of the present disclosure is not limited in this respect.

[0058] At block 1706, a transformation is performed based on the sets of adjusted first samples. In one example, the transformation may include encoding the visual data into a bitstream. Alternatively or additionally, the transformation may include decoding the visual data from the bitstream. It should be understood that the above illustrations are provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect.

[0059] Accordingly, samples of a first representation of visual data (e.g., a quantized latent representation or a quantized residual latent representation) are split into multiple sets, and the multiple sets of first samples are modulated with different parameters. The proposed method thereby enables quantizing samples with different parameters, which can maximize the quality of the reconstructed visual data while simultaneously reducing the number of transmitted bits. In this way, the proposed method can advantageously improve coding quality and coding efficiency.

[0060] In some embodiments, at 1704, the plurality of sets of first samples may be determined from the first representation based on at least one of an index of the sample in the first representation or at least one reference statistic associated with the second representation. As a non-limiting example, the at least one reference statistic may be generated using a second neural network. It should be appreciated that the at least one reference statistic may be generated in any other suitable manner, such as using a machine learning based model. The scope of the present disclosure is not limited in this respect.

[0061] In some embodiments, to determine the multiple sets of first samples from the first representation, the single first sample may be divided into one of the multiple sets of first samples based on at least one of the at least one reference statistical value, at least one sample corresponding to the single first sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single first sample.

[0062] In some alternative embodiments, to determine the multiple sets of first samples from the first representation, the first sample in the first block of the first representation may be divided into one of the multiple sets of first samples based on at least one of the following: a set of samples corresponding to the first sample in the first block of at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the first sample in the first block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

[0063] In some alternative embodiments, the at least one target statistical value may be generated based on the at least one reference statistical value. A plurality of sets of first samples may be determined from the first representation based on the at least one target statistical value. In one example, to determine a plurality of sets of first samples from the first representation based on the at least one target statistical value, a single first sample may be divided into one of a plurality of sets of first samples based on at least one of the following: at least one sample corresponding to the single first sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, or a function of the at least one sample of the at least one target statistical value.

[0064] Alternatively, to determine the multiple sets of first samples from the first representation based on the at least one target statistical value, the first sample in the first block of the first representation may be divided into one of the multiple sets of first samples based on at least one of the following: a set of samples corresponding to the first sample in the first block, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a function of the set of samples, or a metric determined based on one of the set of samples, the probability distribution, or the probability value of the at least one target statistical value.

[0065] In some embodiments, the first representation may be obtained by performing an entropy decoding process on the bitstream based on at least one reference statistic. Alternatively, the first representation may be obtained by performing an entropy decoding process on the bitstream based on at least one target statistic.

[0066] In some embodiments, the second representation may be a latent representation of the visual data. At 1706, the visual data may be reconstructed by performing a synthesis transform on the sets of conditioned first samples. For example, the synthesis transform may be performed using a neural network.

[0067] In some embodiments, the second representation may be a residual latent representation of the visual data. At 1706, a quantized latent representation of the visual data may be generated based on the plurality of sets of adjusted first samples and a first reference statistic of the at least one reference statistic. The visual data may be reconstructed by performing a synthesis transform on the quantized latent representation.

[0068] In some embodiments, the bitstream may be generated by performing an entropy coding process on the multiple sets of the first samples based on at least one reference statistic at 1706. Alternatively, the bitstream may be generated by performing an entropy coding process on the multiple sets of the first samples based on at least one target statistic.

[0069] In some embodiments, at 1702, a plurality of sets of second samples may be determined from the second representation based on at least one of an index of the sample in the second representation or at least one reference statistic associated with the second representation. The plurality of sets of second samples may be scaled with different parameters. Further, the scaled plurality of sets of second samples may be quantized to obtain a plurality of sets of first samples.

[0070] In some embodiments, to determine a plurality of sets of second samples from the second representation, the single second sample may be divided into one of the plurality of sets of second samples based on at least one of the at least one reference statistical value, at least one sample corresponding to the single second sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single second sample.

[0071] In some embodiments, to determine the multiple sets of second samples from the second representation, the second samples in the second block of the second representation may be divided into one of the multiple sets of second samples based on at least one of the following: a set of samples corresponding to the second samples in the second block of at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the second sample in the second block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

[0072] In some embodiments, a first set of second samples of the multiple sets of second samples may be adjusted by scaling the first set of second samples with a third parameter, and further, a second set of second samples of the multiple sets of second samples may be adjusted by scaling the second set of second samples with a fourth parameter, different from the third parameter.

[0073] In some embodiments, the sets of adjusted second samples may be quantized using a rounding function. Alternatively, the sets of adjusted second samples may be quantized by using a floor function. In some further embodiments, the sets of adjusted second samples may be quantized by using a ceiling function. It should be appreciated that the sets of adjusted second samples may also be quantized in any other suitable manner. The scope of the present disclosure is not limited in this respect.

[0074] In some embodiments, the at least one target statistical value may include a first target statistical value, and the at least one reference statistical value may include a second reference statistical value corresponding to the first target statistical value. To generate the at least one target statistical value, a plurality of sets of third samples may be determined from the second reference statistical value based on at least one of the at least one reference statistical value or an index of a sample of the second reference statistical value. Further, the plurality of sets of third samples may be adjusted with different parameters to obtain the first target statistical parameter.

[0075] In some embodiments, to determine the multiple sets of third samples from the second reference statistical value, the single third sample may be divided into one of the multiple sets of third samples based on at least one of the at least one sample of the at least one reference statistical value corresponding to the single third sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single third sample.

[0076] In some alternative embodiments, to determine the multiple sets of third samples from the second reference statistics, the third samples in the third block of the second reference statistics may be divided into one of the multiple sets of third samples based on at least one of the following: a set of samples corresponding to the third samples in the third block of the at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the third sample in the third block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index. As a non-limiting example, the threshold may be indicated in the bitstream. The metric may be an average value, a minimum value, or a maximum value. The index of the sample indicates one of a channel number of the sample, a feature map identifier of the sample, and a spatial coordinate of the sample.

[0077] In some embodiments, the multiple sets of first samples may include the first set of first samples, and a third sample corresponding to the first set of first samples may be split into a single set.

[0078] In some embodiments, a first set of third samples of the multiple sets of third samples may be adjusted by scaling the first set of third samples with a fifth parameter, and a second set of third samples of the multiple sets of third samples may be adjusted by scaling the second set of third samples with a sixth parameter, different from the fifth parameter.

[0079] In some embodiments, the second neural network may be an estimation model. In one example, the second neural network may include a neural network-based sub-network. For example, the input of the second neural network may include a bit stream. Additionally or alternatively, the second neural network may include a first sub-network for generating a first reference statistic and a second sub-network for generating a second reference statistic. As a non-limiting example, the first reference statistic may be a mean and the second reference statistic may be a variance. The first sub-network may be a hyper-decoder sub-network and the second sub-network may be a hyper-scale decoder sub-network.

[0080] In some embodiments, at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter may be a scalar number. Alternatively, at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter may be a vector.

[0081] In some embodiments, at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter may be indicated in the bitstream. Alternatively, an indication of at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter may be indicated in the bitstream. In some further embodiments, at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter may be obtained based on the list and the index value.

[0082] In some embodiments, the first parameter may be determined based on the third parameter. As a non-limiting example, the first parameter may be determined as the reciprocal of the third parameter. For example, one of the first parameter and the second parameter may be 1. Additionally or alternatively, one of the third parameter and the fourth parameter may be 1. Additionally or alternatively, one of the fifth parameter and the sixth parameter may be 1.

[0083] In some embodiments, the fifth parameter may be the same as one of the first parameter or the second parameter, and the sixth parameter may be the same as the other one of the first parameter or the second parameter. In such a case, the first set of samples and the third set of samples may be adjusted in the same manner.

[0084] In some embodiments, the second representation may be a latent representation of the visual data. The second representation may be obtained by performing an analytical transformation on the visual data using the first neural network.

[0085] In some embodiments, the second representation may be a residual latent representation of the visual data. The latent representation of the visual data may be generated by performing an analytical transformation on the visual data using a first neural network. The second representation may be generated based on the latent representation and a first reference statistic of the at least one reference statistic. In some embodiments, the first reference statistic may be an average.

[0086] In some embodiments, the at least one reference statistic may include at least one of a mean or a variance of a probability distribution. By way of non-limiting example, the probability distribution may be a Gaussian probability distribution or a Laplace probability distribution.

[0087] In some embodiments, at least one of the size of the first block, the size of the second block, or the size of the third block may be predetermined or indicated in the bitstream.

[0088] In some embodiments, at least one of the information regarding whether to apply the method or how to apply the method may be indicated in the bitstream. Alternatively, at least one of the information regarding whether to apply the method or how to apply the method may depend on the color format and / or color components of the visual data.

[0089] In some embodiments, the values ​​included in the bitstream may be coded at one of a sequence level, a picture level, a slice level, or a block level. Additionally or alternatively, the values ​​included in the bitstream may be binarized before being coded. Additionally or alternatively, the values ​​included in the bitstream are coded with at least one arithmetic coding context.

[0090] According to a further embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided that stores a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method including obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data, adjusting a plurality of sets of first samples of the first representation with different parameters, and generating a bitstream based on the plurality of sets of adjusted first samples.

[0091] According to a further embodiment of the present disclosure, there is provided a method for storing a bitstream of visual data, the method including obtaining a first representation of the visual data, the first representation obtained by quantizing a second representation of the visual data, the second representation generated based on applying a first neural network to the visual data, adjusting a plurality of sets of first samples of the first representation with different parameters, generating a bitstream based on the plurality of sets of adjusted first samples, and storing the bitstream on a non-transitory computer-readable recording medium.

[0092] Implementations of the present disclosure can be described in terms of the following clauses, the features of which can be combined in any reasonable manner.

[0093] Item 1. A method for visual data processing, for conversion between visual data and a bitstream of visual data, comprising: obtaining a first representation of visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; adjusting a plurality of sets of first samples of the first representation with different parameters; and performing conversion based on the plurality of sets of adjusted first samples.

[0094] Clause 2. The method of clause 1, wherein adjusting the plurality of sets of first samples includes determining the plurality of sets of first samples from the first representation based on at least one of an index of a sample of the first representation or at least one reference statistical value associated with the second representation.

[0095] Clause 3. The method of clause 2, wherein the at least one reference statistical value is generated using a second neural network.

[0096] Clause 4. The method of clause 2 or 3, wherein determining a plurality of sets of first samples from the first representation includes dividing the single first sample into one of the plurality of sets of first samples based on at least one of at least one reference statistical value, at least one sample corresponding to the single first sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single first sample.

[0097] Clause 5. The method of clause 2 or clause 3, wherein determining the multiple sets of first samples from the first representation includes dividing the first sample in the first block of the first representation into one of the multiple sets of first samples based on at least one of the following: a set of samples corresponding to the first sample in the first block of at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the first sample in the first block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

[0098] Clause 6. The method of clause 2 or clause 3, wherein determining a plurality of sets of first samples from the first representation includes generating at least one target statistical value based on at least one reference statistical value, and determining a plurality of sets of first samples from the first representation based on the at least one target statistical value.

[0099] Clause 7. The method of clause 6, wherein determining a plurality of sets of first samples from the first representation based on the at least one target statistical value includes dividing the single first sample into one of the plurality of sets of first samples based on at least one of the following functions of the at least one target statistical value: at least one sample corresponding to the single first sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, or the at least one sample.

[0100] Clause 8. The method of clause 6, wherein determining a plurality of sets of first samples from the first representation based on at least one target statistical value includes dividing a first sample in a first block of the first representation into one of a plurality of sets of first samples based on at least one of the following: a set of samples corresponding to the first sample in the first block, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a function of the set of samples, or a metric determined based on one of the set of samples, the probability distribution, or the probability value of the at least one target statistical value.

[0101] Clause 9. The method of any one of clauses 1 to 8, wherein adjusting the multiple sets of first samples includes adjusting a first set of first samples of the multiple sets of first samples by scaling the first set of first samples with a first parameter, and adjusting a second set of first samples of the multiple sets of first samples by scaling the second set of first samples with a second parameter different from the first parameter.

[0102] Clause 10. A method according to any one of clauses 6 to 9, wherein the first representation is obtained by performing an entropy decoding process on the bitstream based on at least one reference statistic or at least one target statistic.

[0103] Clause 11. The method of any one of clauses 1 to 10, wherein the second representation comprises a latent representation of the visual data or a residual latent representation of the visual data.

[0104] Clause 12. The method of any one of clauses 1 to 11, wherein the second representation is a latent representation of the visual data, and performing the transformation includes reconstructing the visual data by performing a synthesis transformation on a plurality of sets of adjusted first samples.

[0105] Clause 13. The method of any one of clauses 2 to 11, wherein the second representation is a residual latent representation of the visual data, and performing the transformation includes generating a quantized latent representation of the visual data based on a plurality of sets of adjusted first samples and a first reference statistic of the at least one reference statistic, and reconstructing the visual data by performing a synthesis transformation on the quantized latent representation.

[0106] Clause 14. A method according to any one of clauses 6 to 11, wherein performing the transformation includes generating a bitstream by performing an entropy coding process on a plurality of sets of first samples based on at least one reference statistical value or at least one target statistical value.

[0107] Clause 15. The method of any one of clauses 1 to 14, wherein obtaining the multiple sets of first samples includes determining the multiple sets of second samples from the second representation based on at least one of indexes of samples of the second representation or at least one reference statistical value associated with the second representation, adjusting the multiple sets of second samples with different parameters, and quantizing the adjusted multiple sets of second samples to obtain the multiple sets of first samples.

[0108] Clause 16. The method of clause 15, wherein determining a plurality of sets of second samples from the second representation includes dividing the single second sample into one of the plurality of sets of second samples based on at least one of the following: at least one sample corresponding to the single second sample of at least one reference statistical value, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single second sample.

[0109] Clause 17. The method of clause 15, wherein determining a plurality of sets of second samples from the second representation includes dividing the second samples in the second block of the second representation into one of a plurality of sets of second samples based on at least one of the following: a set of samples corresponding to the second samples in the second block of at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold, a function of the set of samples, an index of the second sample in the second block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

[0110] Clause 18. The method of any one of clauses 15 to 17, wherein adjusting the plurality of sets of second samples includes adjusting a first set of second samples of the plurality of sets of second samples by scaling the first set of second samples with a third parameter, and adjusting a second set of second samples of the plurality of sets of second samples by scaling the second set of second samples with a fourth parameter different from the third parameter.

[0111] Clause 19. The method of any one of clauses 15 to 18, wherein the multiple sets of adjusted second samples are quantized using one of a rounding function, a floor function, or a ceiling function.

[0112] Clause 20. The method of any one of clauses 6 to 19, wherein the at least one target statistical value includes a first target statistical value, and the at least one reference statistical value includes a second reference statistical value corresponding to the first target statistical value, and generating the at least one target statistical value includes determining a plurality of sets of third samples from the second reference statistical value based on at least one of an index of a sample of the second reference statistical value or the at least one reference statistical value, and adjusting the plurality of sets of third samples with different parameters to obtain the first target statistical parameter.

[0113] Clause 21. The method of clause 20, wherein determining a plurality of sets of third samples from the second reference statistical value includes dividing the single third sample into one of the plurality of sets of third samples based on at least one of the following: at least one sample of the at least one reference statistical value corresponding to the single third sample, a probability distribution determined based on the at least one sample, a probability value determined based on the at least one sample, a threshold, a function of the at least one sample, or an index of the single third sample.

[0114] Clause 22. The method of clause 20, wherein determining the multiple sets of third samples from the second reference statistic includes dividing the third samples in the third block of the second reference statistic into one of the multiple sets of third samples based on at least one of the following: a set of samples corresponding to the third samples in the third block of the at least one reference statistic, a probability distribution determined based on the set of samples, a probability value determined based on the set of samples, a threshold value, a function of the set of samples, an index of the third sample in the third block, or a metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

[0115] Clause 23. The method of any one of clauses 20 to 22, wherein the multiple sets of first samples include a first set of first samples, and a third sample corresponding to the first set of first samples is divided into a single set.

[0116] Clause 24. The method of any one of clauses 20 to 23, wherein adjusting the plurality of sets of third samples includes adjusting a first set of third samples of the plurality of sets of third samples by scaling the first set of third samples with a fifth parameter, and adjusting a second set of third samples of the plurality of sets of third samples by scaling the second set of third samples with a sixth parameter different from the fifth parameter.

[0117] Clause 25. The method of any one of clauses 20 to 24, wherein the threshold is indicated in the bitstream.

[0118] Clause 26. The method of any one of clauses 5 to 25, wherein the metric is an average value, a minimum value, or a maximum value.

[0119] Clause 27. A method according to any one of clauses 5 to 26, wherein the index of the sample indicates one of a channel number of the sample, a feature map identifier of the sample, or a spatial coordinate of the sample.

[0120] Clause 28. The method of any one of clauses 3 to 27, wherein the second neural network is an estimation model.

[0121] Clause 29. The method of any one of clauses 3 to 28, wherein the second neural network includes a neural network-based sub-network.

[0122] Clause 30. The method of any one of clauses 3 to 29, wherein the input of the second neural network comprises a bit stream.

[0123] Clause 31. The method of any one of clauses 3 to 28, wherein the second neural network includes a first sub-network for generating a first reference statistical value and a second sub-network for generating a second reference statistical value.

[0124] Clause 32. The method of clause 31, wherein the first reference statistic is a mean, the second reference statistic is a variance, the first sub-network is a hyper-decoder sub-network, and the second sub-network is a hyper-scale decoder sub-network.

[0125] Clause 33. The method of any one of clauses 9 to 32, wherein at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter is a scalar number.

[0126] Clause 34. The method of any one of clauses 9 to 32, wherein at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter is a vector.

[0127] Clause 35. The method of any one of clauses 9 to 34, wherein at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter is indicated in a bitstream.

[0128] Clause 36. The method of any one of clauses 9 to 34, wherein an indication of at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter is indicated in a bitstream.

[0129] Clause 37. The method of any one of clauses 9 to 34, wherein at least one of the first parameter, the second parameter, the third parameter, the fourth parameter, the fifth parameter, or the sixth parameter is obtained based on the list and an index value.

[0130] Clause 38. The method of any one of clauses 9 to 37, wherein the first parameter is determined based on a third parameter.

[0131] Clause 39. The method of clause 38, wherein the first parameter is determined as the reciprocal of the third parameter.

[0132] Item 40. The method according to any one of items 9 to 39, wherein one of the first parameter and the second parameter is 1.

[0133] Item 41. The method according to any one of items 18 to 40, wherein one of the third parameter and the fourth parameter is 1.

[0134] Item 42. The method according to any one of items 24 to 41, wherein one of the fifth parameter and the sixth parameter is 1.

[0135] Item 43. The method according to any one of items 24 to 42, wherein the fifth parameter is the same as one of the first parameter or the second parameter, and the sixth parameter is the same as the other of the first parameter or the second parameter.

[0136] Clause 44. The method of any one of clauses 1 to 43, wherein the second representation is a latent representation of the visual data, and the second representation is obtained by performing an analytical transformation on the visual data using the first neural network.

[0137] Clause 45. The method of any one of clauses 2 to 44, wherein the second representation is a residual latent representation of the visual data, and the second representation is obtained by generating a latent representation of the visual data by performing an analytical transformation on the visual data using a first neural network, and generating the second representation based on the latent representation and a first reference statistical value of at least one reference statistical value.

[0138] Item 46. The method of any one of items 13 to 45, wherein the first reference statistical value is an average.

[0139] Clause 47. The method of any one of clauses 2 to 13, wherein the at least one reference statistical value comprises at least one of the mean or variance of the probability distribution.

[0140] Clause 48. The method of clause 47, wherein the probability distribution is a Gaussian probability distribution.

[0141] Clause 49. A method according to any one of clauses 5 to 48, wherein at least one of the size of the first block, the size of the second block, or the size of the third block is predetermined or indicated in the bitstream.

[0142] Clause 50. A method according to any one of clauses 1 to 49, in which at least one of information regarding whether to apply the method or information regarding how to apply the method is indicated in the bitstream.

[0143] Clause 51. A method according to any one of clauses 1 to 49, wherein at least one of the information regarding whether to apply the method or the information regarding how to apply the method depends on the color format and / or color components of the visual data.

[0144] Clause 52. A method according to any one of clauses 1 to 51, wherein the values ​​included in the bitstream are coded at one of the sequence level, the picture level, the slice level or the block level.

[0145] Clause 53. A method according to any one of clauses 1 to 52, wherein values ​​contained in the bitstream are binarised before being encoded.

[0146] Clause 54. A method according to any one of clauses 1 to 53, wherein values ​​included in the bitstream are coded with at least one arithmetic coding context.

[0147] Clause 55. The method of any one of clauses 1 to 54, wherein the visual data includes a picture or image of a video.

[0148] Clause 56. The method of any one of clauses 1 to 55, wherein the transforming includes encoding the visual data into a bitstream.

[0149] Clause 57. The method of any one of clauses 1 to 55, wherein the converting includes decoding the visual data from a bitstream.

[0150] Clause 58. An apparatus for visual data processing comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to any one of clauses 1 to 57.

[0151] Clause 59. A non-transitory computer-readable storage medium having stored thereon instructions for causing a processor to perform the method of any one of clauses 1 to 57.

[0152] Clause 60. A non-transitory computer-readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method including: obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; adjusting a plurality of sets of first samples of the first representation with different parameters; and generating a bitstream based on the plurality of sets of adjusted first samples.

[0153] Clause 61. A method for storing a bitstream of visual data, the method comprising: obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; adjusting a plurality of sets of first samples of the first representation with different parameters; generating a bitstream based on the plurality of sets of adjusted first samples; and storing the bitstream on a non-transitory computer-readable recording medium.

[0154] Exemplary Devices 18 illustrates a block diagram of a computing device 1800 in which various embodiments of the disclosure may be implemented. The computing device 1800 may be implemented as or included in the source device 110 (or the visual data encoder 114) or the destination device 120 (or the visual data decoder 124).

[0155] It should be understood that the computing device 1800 shown in FIG. 18 is for illustrative purposes only and is not intended to suggest any limitation on the functionality and scope of the embodiments of the present disclosure.

[0156] 18, the computing device 1800 includes a general-purpose computing device 1800. The computing device 1800 may include at least one or more processors or processing units 1810, a memory 1820, a storage unit 1830, one or more communication units 1840, one or more input devices 1850, and one or more output devices 1860.

[0157] In some embodiments, the computing device 1800 may be implemented as any user terminal or server terminal having computing capabilities. A server terminal may be a server provided by a service provider, a large-scale computing device, etc. A user terminal may be, for example, any type of mobile, fixed, or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, further including accessories and peripherals of these devices, or any combination thereof. It is contemplated that the computing device 1800 may support any type of interface to a user, such as "wearable" circuitry.

[0158] The processing unit 1810 may be a physical or virtual processor and may perform various processes based on programs stored in the memory 1820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the computing device 1800. The processing unit 1810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0159] Computing device 1800 typically includes a variety of computer storage media. Such media may be any media accessible by computing device 1800, including but not limited to volatile and nonvolatile media, or removable and non-removable media. Memory 1820 may be volatile memory (e.g., registers, cache, Random Access Memory (RAM)), non-volatile memory (Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory, etc.), or any combination thereof. Storage unit 1830 may be any removable or non-removable media that may be used to store information and / or data and may include a machine-readable medium, such as a memory, a flash memory drive, a magnetic disk, or other media that is accessible within computing device 1800.

[0160] Computing device 1800 may further include other removable / non-removable, volatile / non-volatile memory media. Although not shown in FIG. 18, it is possible to provide a magnetic disk drive for reading from and / or writing to a removable, non-volatile magnetic disk, and an optical disk drive for reading from and / or writing to a removable, non-volatile optical disk. In such a case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0161] The communication unit 1840 communicates with further computing devices via a communication medium. Furthermore, the functionality of the components in the computing device 1800 may be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1800 may operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or more general network nodes.

[0162] The input device(s) 1850 may be one or more of a variety of input devices such as a mouse, a keyboard, a tracking ball, a voice input device, etc. The output device(s) 1860 may be one or more of a variety of output devices such as a display, a speaker, a printer, etc. The communication unit 1840 allows the computing device 1800 to further communicate with one or more external devices (not shown), such as a storage device and a display device, with one or more devices that allow a user to interact with the computing device 1800, or with any device (such as a network card, a modem, etc.) that allows the computing device 1800 to communicate with one or more other computing devices, as needed. Such communication may occur via an input / output (I / O) interface (not shown).

[0163] In some embodiments, instead of being integrated into a single device, some or all of the components of computing device 1800 may be located in a cloud computing architecture. In a cloud computing architecture, the components may be provided remotely and may cooperate to implement the functionality described in this disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services without the end user needing to be aware of the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications accessible through a web browser or other computing components over the wide area network. Software or components of the cloud computing architecture and corresponding data may be stored on servers in remote locations. Computing resources of a cloud computing environment may be consolidated or distributed across locations in remote data centers. The cloud computing infrastructure may act as a single access point for users, but provide services through a shared data center. Thus, a cloud computing architecture may be used to provide the components and functionality described herein from a remote service provider. Alternatively, it may be provided from a conventional server or installed directly or in other manner on the client device.

[0164] The computing device 1800 may be used to perform visual data encoding / decoding in embodiments of the present disclosure. The memory 1820 may include one or more visual data encoding modules 1825 having one or more program instructions. These modules are accessible and executable by the processing unit 1810 to perform the functions of the various embodiments described herein.

[0165] In an example embodiment performing visual data encoding, the input device 1850 may receive visual data as input to be encoded 1870. The visual data may be processed, for example, by the visual data encoding module 1825 to generate an encoded bitstream. The encoded bitstream may be provided as output 1880 via the output device 1860.

[0166] In an example embodiment performing visual data decoding, input device 1850 may receive an encoded bitstream as input 1870. The encoded bitstream may be processed, for example, by visual data encoding module 1825 to generate decoded visual data. The decoded visual data may be provided as output 1880 via output device 1860.

[0167] Although the present disclosure has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. Such variations are intended to be covered by the scope of the present application. Thus, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. obtaining a first representation of the visual data for conversion between visual data and a bitstream of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; Adjusting a plurality of sets of first samples of the first representation with different parameters; performing the transformation based on a plurality of sets of the adjusted first samples; 16. A method for visual data processing, comprising:

2. Adjusting the first set of samples includes: an index of the sample of the first representation; or At least one reference statistic associated with said second representation.

2. The method of claim 1, further comprising: determining a plurality of sets of the first samples from the first representation based on at least one of:

3. The method of claim 2 , wherein the at least one reference statistic is generated using a second neural network.

4. Determining a plurality of sets of first samples from the first representation includes: at least one sample of the at least one reference statistic corresponding to the single first sample; A probability distribution determined based on the at least one sample. a probability value determined based on said at least one sample; Threshold, a function of said at least one sample; or The index of the single first sample 4. The method of claim 2, further comprising splitting the single first sample into one of a plurality of sets of the first sample based on at least one of:

5. Determining a plurality of sets of first samples from the first representation includes: a set of samples of the at least one reference statistic corresponding to a first sample in a first block; a probability distribution determined based on the set of samples; a probability value determined based on the set of samples; Threshold, a function of said set of samples, the index of the first sample within the first block; or A metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

4. The method of claim 2, further comprising dividing the first samples in the first block of the first representation into one of a plurality of sets of the first samples based on at least one of:

6. Determining a plurality of sets of first samples from the first representation includes: generating at least one target statistical value based on the at least one reference statistical value; determining a plurality of sets of the first samples from the first representation based on the at least one target statistic; The method of claim 2 or 3, comprising:

7. Determining a plurality of sets of first samples from the first representation based on the at least one target statistic includes: at least one sample of the at least one target statistic corresponding to the single first sample; A probability distribution determined based on the at least one sample. a probability value determined based on the at least one sample; or A function of said at least one sample 7. The method of claim 6, further comprising: dividing the single first sample into one of a plurality of sets of the first sample based on at least one of:

8. Determining a plurality of sets of first samples from the first representation based on the at least one target statistic includes: a set of samples of the at least one target statistic corresponding to a first sample in a first block; a probability distribution determined based on the set of samples; a probability value determined based on the set of samples; a function of the set of samples; or a metric determined based on one of the set of samples, the probability distribution, or the probability value; 7. The method of claim 6, further comprising dividing the first samples in the first block of the first representation into one of a plurality of sets of the first samples based on at least one of:

9. Adjusting the first set of samples includes: adjusting a first set of first samples by scaling the first set of first samples of the plurality of sets of first samples with a first parameter; adjusting the second set of first samples by scaling the second set of first samples of the plurality of sets of first samples with a second parameter different from the first parameter; The method according to any one of claims 1 to 8, comprising:

10. 10. The method according to claim 6, wherein the first representation is obtained by performing an entropy decoding operation on the bitstream based on the at least one reference statistic or the at least one target statistic.

11. The method of any one of claims 1 to 10, wherein the second representation comprises a latent representation of the visual data or a residual latent representation of the visual data.

12. The second representation is a latent representation of the visual data, and performing the transformation includes: The method of any one of claims 1 to 11, comprising reconstructing the visual data by performing a synthesis transform on a plurality of sets of the adjusted first samples.

13. The second representation is a residual latent representation of the visual data, and performing the transformation comprises: generating a quantized latent representation of the visual data based on a plurality of sets of the adjusted first samples and a first reference statistic of the at least one reference statistic; reconstructing the visual data by performing a synthesis transform on the quantized latent representation; The method according to any one of claims 2 to 11, comprising:

14. Performing the conversion includes:

12. The method according to claim 6, comprising generating the bitstream by performing an entropy coding operation on a plurality of sets of the first samples based on the at least one reference statistic or the at least one target statistic.

15. Obtaining the plurality of sets of first samples includes: determining a plurality of sets of second samples from the second representation based on at least one of an index of the samples in the second representation or at least one reference statistic associated with the second representation; conditioning the second set of samples with different parameters; and quantizing the adjusted sets of second samples to obtain the sets of first samples; The method according to any one of claims 1 to 14, comprising:

16. Determining a plurality of sets of second samples from the second representation includes: at least one sample of the at least one reference statistic corresponding to a single second sample; A probability distribution determined based on the at least one sample. a probability value determined based on said at least one sample; Threshold, a function of said at least one sample; or The index of the single second sample 16. The method of claim 15, further comprising splitting the single second sample into one of a plurality of sets of second samples based on at least one of:

17. Determining a plurality of sets of second samples from the second representation includes: a set of samples of the at least one reference statistic corresponding to a second sample in a second block; a probability distribution determined based on the set of samples; a probability value determined based on the set of samples; Threshold, a function of said set of samples, the index of the second sample within the second block; or A metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

16. The method of claim 15, comprising dividing the second samples in the second block of the second representation into one of a plurality of sets of the second samples based on at least one of:

18. Adjusting the second set of samples includes: adjusting the first set of second samples of the plurality of sets of second samples by scaling the first set of second samples with a third parameter; adjusting the second set of second samples by scaling the second set of second samples of the plurality of sets of second samples with a fourth parameter different from the third parameter; The method according to any one of claims 15 to 17, comprising:

19. The method of any one of claims 15 to 18, wherein the sets of adjusted second samples are quantized using one of a rounding function, a floor function, or a ceiling function.

20. The at least one target statistical value includes a first target statistical value, and the at least one reference statistical value includes a second reference statistical value corresponding to the first target statistical value, and generating the at least one target statistical value includes: determining a plurality of sets of third samples from the second reference statistical value based on at least one of an index of a sample of the second reference statistical value or the at least one reference statistical value; adjusting a plurality of sets of the third samples with different parameters to obtain the first target statistical parameter; The method according to any one of claims 6 to 19, comprising:

21. Determining the third plurality of sets of samples from the second reference statistics includes: at least one sample of the at least one reference statistic corresponding to a single third sample; A probability distribution determined based on the at least one sample. a probability value determined based on said at least one sample; Threshold, a function of said at least one sample; or The index of the single third sample 21. The method of claim 20, comprising splitting the single third sample into one of a plurality of sets of third samples based on at least one of:

22. Determining the third plurality of sets of samples from the second reference statistics includes: a set of samples of the at least one reference statistic corresponding to a third sample in a third block; a probability distribution determined based on the set of samples; a probability value determined based on the set of samples; Threshold, a function of said set of samples, the index of the third sample within the third block; or A metric determined based on one of the set of samples, the probability distribution, the probability value, or the index.

21. The method of claim 20, further comprising dividing the third samples in the third block of the second reference statistic into one of a plurality of sets of the third samples based on at least one of:

23. 23. The method of any one of claims 20 to 22, wherein the multiple sets of first samples include a first set of first samples, and a third sample corresponding to the first set of first samples is split into a single set.

24. Adjusting the third set of samples includes: adjusting the first set of third samples of the plurality of sets of third samples by scaling the first set of third samples with a fifth parameter; adjusting the second set of third samples by scaling the second set of third samples of the plurality of sets of third samples with a sixth parameter, the sixth parameter being different from the fifth parameter; The method according to any one of claims 20 to 23, comprising:

25. The method of any one of claims 20 to 24, wherein the threshold value is indicated in the bitstream.

26. The method of any one of claims 5 to 25, wherein the metric is an average value, a minimum value or a maximum value.

27. The sample index is the channel number of said sample, a feature map identifier for the sample; or The spatial coordinates of the sample The method according to any one of claims 5 to 26, wherein the method is one of the following:

28. The method according to any one of claims 3 to 27, wherein the second neural network is an estimation model.

29. The method of any one of claims 3 to 28, wherein the second neural network comprises a neural network-based sub-network.

30. A method according to any one of claims 3 to 29, wherein an input of the second neural network comprises the bit stream.

31. 29. The method according to any one of claims 3 to 28, wherein the second neural network comprises a first sub-network for generating a first reference statistical value and a second sub-network for generating a second reference statistical value.

32. 32. The method of claim 31 , wherein the first reference statistic is a mean, the second reference statistic is a variance, the first sub-network is a hyper-decoder sub-network, and the second sub-network is a hyper-scale decoder sub-network.

33. The first parameter, The second parameter, The third parameter, The fourth parameter, The fifth parameter, or The sixth parameter The method of any one of claims 9 to 32, wherein at least one of is a scalar number.

34. The first parameter, The second parameter, The third parameter, The fourth parameter, The fifth parameter, or The sixth parameter The method of any one of claims 9 to 32, wherein at least one of is a vector.

35. The first parameter, The second parameter, The third parameter, The fourth parameter, The fifth parameter, or The sixth parameter The method according to any one of claims 9 to 34, wherein at least one of: is indicated in the bitstream.

36. The first parameter, The second parameter, The third parameter, The fourth parameter, The fifth parameter, or The sixth parameter A method according to any one of claims 9 to 34, wherein an indication of at least one of is indicated in the bitstream.

37. The first parameter, The second parameter, The third parameter, The fourth parameter, The fifth parameter, or The sixth parameter The method according to any one of claims 9 to 34, wherein at least one of is obtained based on the list and an index value.

38. The method according to any one of claims 9 to 37, wherein the first parameter is determined based on the third parameter.

39. 39. The method of claim 38, wherein the first parameter is determined as the reciprocal of the third parameter.

40. The method of any one of claims 9 to 39, wherein one of the first parameter and the second parameter is one.

41. The method of any one of claims 18 to 40, wherein one of the third parameter and the fourth parameter is one.

42. The method of any one of claims 24 to 41, wherein one of the fifth parameter and the sixth parameter is 1.

43. 43. The method according to any one of claims 24 to 42, wherein the fifth parameter is the same as one of the first parameter or the second parameter, and the sixth parameter is the same as the other of the first parameter or the second parameter.

44. the second representation is a latent representation of the visual data; The method of any one of claims 1 to 43, wherein the second representation is obtained by performing an analytical transformation on the visual data using the first neural network.

45. the second representation is a residual latent representation of the visual data; generating a latent representation of the visual data by performing an analytical transformation on the visual data using the first neural network; generating the second representation based on the latent representation and a first reference statistic of the at least one reference statistic; The method of any one of claims 2 to 44, wherein the second representation is obtained by:

46. The method of any one of claims 13 to 45, wherein the first reference statistic is an average.

47. The method of any one of claims 2 to 13, wherein the at least one reference statistical value comprises at least one of a mean or a variance of a probability distribution.

48. 48. The method of claim 47, wherein the probability distribution is a Gaussian probability distribution.

49. 49. The method of any one of claims 5 to 48, wherein at least one of the size of the first block, the size of the second block, or the size of the third block is predetermined or indicated in the bitstream.

50. A method according to any preceding claim, wherein at least one of information regarding whether to apply the method or information regarding how to apply the method is indicated in the bitstream.

51. 50. The method of any one of claims 1 to 49, wherein at least one of the information regarding whether to apply the method or how to apply the method depends on the colour format and / or colour components of the visual data.

52. A method according to any one of the preceding claims, wherein values ​​included in the bitstream are coded at one of a sequence level, a picture level, a slice level or a block level.

53. A method according to any preceding claim, wherein values ​​contained in the bitstream are binarised before being encoded.

54. A method according to any one of claims 1 to 53, wherein values ​​included in the bitstream are coded with at least one arithmetic coding context.

55. A method according to any preceding claim, wherein the visual data comprises an image or a video picture.

56. A method according to any preceding claim, wherein the transforming comprises encoding the visual data into the bitstream.

57. A method according to any preceding claim, wherein said converting comprises decoding said visual data from said bitstream.

58. 1. An apparatus for visual data processing comprising a processor and a non-transitory memory having instructions thereon, Apparatus for visual data processing, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any preceding claim.

59. A non-transitory computer readable storage medium storing instructions that cause a processor to perform the method of any one of claims 1 to 57.

60. 1. A non-transitory computer readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method comprising: obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; Adjusting a plurality of sets of first samples of the first representation with different parameters; generating the bitstream based on a plurality of sets of the adjusted first samples; A non-transitory computer-readable recording medium comprising:

61. 1. A method for storing a bitstream of visual data, comprising the steps of: obtaining a first representation of the visual data, the first representation being obtained by quantizing a second representation of the visual data, the second representation being generated based on applying a first neural network to the visual data; Adjusting a plurality of sets of first samples of the first representation with different parameters; generating the bitstream based on a plurality of sets of the adjusted first samples; storing the bitstream on a non-transitory computer readable recording medium; 16. A method for storing a bitstream of visual data, comprising:

Images