Feature data encoding and decoding method and apparatus

Abstract

To provide picture or audio encoding and decoding methods and apparatuses.SOLUTION: The encoding method includes: obtaining a to-be-encoded target, where the to-be-encoded target includes a plurality of feature elements, and the plurality of feature elements include a first feature element. The method further includes: obtaining a probability estimation result of the first feature element; determining, based on the probability estimation result of the first feature element, whether to perform entropy encoding on the first feature element; and performing entropy encoding on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element. In this application, whether to encode a feature element is determined based on a probability estimation result. In this way, encoding and decoding complexity can be reduced without affecting encoding and decoding performance.SELECTED DRAWING: Figure 11A

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to the field of artificial intelligence (AI)-based picture or audio compression technology, and in particular to feature data encoding and decoding methods and apparatus. [Background technology]

[0002] Picture or audio encoding and decoding (encoding and decoding for short) is widely used in digital picture or audio applications, such as digital television broadcasting, picture or audio transmission over the Internet and mobile networks, real-time conversation applications such as video or audio chat, video or audio conferencing, DVD and Blu-ray discs, picture or audio content capture and editing systems, and secure applications of camcorders. A video contains multiple frames of pictures. Therefore, a picture in this application may be a single picture or a picture in a video.

[0003] The large amount of video data required to render even a short video can be substantial, which can pose difficulties when the data is to be streamed or communicated over networks with limited bandwidth capacity. Therefore, picture (or audio) data is generally compressed before being communicated over modern telecommunications networks. Because memory resources may be limited, the size of the picture (or audio) data can also be an issue when the picture (or audio) data is stored on a storage device. Picture (or audio) compression devices often use source-side software and / or hardware to encode the picture (or audio) data before transmission or storage. This reduces the amount of data needed to represent a digital picture (or audio). The compressed data is then received at the destination side by a picture (or audio) decompression device. With limited network resources and an ever-increasing demand for higher picture (or audio) quality, improved compression and decompression techniques are desirable that improve compression ratios with little or no sacrifice in picture (or audio) quality.

[0004] In recent years, deep learning has gained popularity in the field of picture (or audio) encoding and decoding. For example, Google has held the Challenge on Learned Image Compression (CLIC) competition at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) for several consecutive years. CLIC focuses on using deep neural networks to improve picture compression efficiency. A Picture Challenge category was also added to CLIC 2020. Based on performance evaluation of the solutions in the competition, the comprehensive compression efficiency of current picture encoding and decoding solutions based on deep learning technology is comparable to that of the latest generation video picture encoding and decoding standard, Versatile Video Coding (VVC), and has unique advantages in improving user-perceived quality.

[0005] The VVC video standard was finalized in June 2020. This standard includes almost all technical algorithms that can significantly improve compression efficiency. Therefore, it is difficult to achieve technological breakthroughs in a short period of time by continuing to research new compression coding algorithms along the traditional signal processing path. Unlike traditional picture algorithms that optimize picture compression modules through manual design, end-to-end AI picture compression is optimized as a whole. Therefore, AI picture compression has a higher compression efficiency. The variational autoencoder (VAE) method is the current mainstream technical solution for AI picture lossy compression technology. In the current mainstream technical solution, a picture feature map is obtained for the picture to be encoded using an encoder network, and then entropy coding is performed on the picture feature map. However, the entropy coding process is overly complicated. Summary of the Invention [Means for solving the problem]

[0006] This application provides a method and apparatus for encoding and decoding feature data to reduce the complexity of encoding and decoding without affecting the performance of encoding and decoding.

[0007] According to a first aspect, obtaining feature data to be encoded, the feature data to be encoded comprising a plurality of feature elements, the plurality of feature elements comprising a first feature element; obtaining a probability estimation result of a first feature element; determining whether to perform entropy coding on the first feature based on the probability estimation result of the first feature; performing entropy coding on the first feature only when it is determined that entropy coding needs to be performed on the first feature; A feature data encoding method is provided, including:

[0008] The feature data may be one-dimensional, two-dimensional, or multi-dimensional data output by the encoder network, including a picture feature map, an audio feature variable, or a picture feature map and an audio feature variable, and each of the data is a feature element. It should be noted that the meanings of feature points and feature elements in this application are the same.

[0009] Specifically, the first feature element is any feature element to be coded of the feature data to be coded.

[0010] In one possibility, the probability estimation process for obtaining the probability estimation result of the first feature element may be performed by using a probability estimation network. In another possibility, the probability estimation process may use a conventional non-network probability estimation method to perform probability estimation on the feature data.

[0011] It should be noted that when only side information is used as the input of probability estimation, the probability estimation results of feature elements can be output in parallel. When the input of probability estimation includes context information, the probability estimation results of feature elements need to be output in serial. The side information is feature information that is further extracted by inputting feature data into a neural network, and the number of feature elements included in the side information is less than the number of feature elements in the feature data. Optionally, the side information of the feature data can be coded into a bitstream.

[0012] Possibly, when the first feature element of the feature data does not satisfy the preset condition, entropy coding does not need to be performed on the first feature element of the feature data.

[0013] Specifically, if the current first feature element is the Pth feature element of the feature data, the determination of the Pth feature element is completed, and entropy coding is or is not performed based on the determination result. After that, the determination of the (P+1)th feature element of the feature data is started, and the entropy coding process is or is not performed based on the determination result. P is a positive integer and is smaller than M, and M is the number of feature elements in the entire feature data. For example, for the second feature element, when it is determined that entropy coding does not need to be performed on the second feature element, performing entropy coding on the second feature element is skipped.

[0014] In the above technical solution, whether entropy coding needs to be performed is determined for each feature to be coded, so that the entropy coding process of some features can be skipped, and the number of elements that need to be entropy coded can be significantly reduced. In this way, the complexity of entropy coding can be reduced.

[0015] In a possible implementation, determining whether to perform entropy coding on the first feature element includes determining that entropy coding needs to be performed on the first feature element when the probability estimation result of the first feature element satisfies a predetermined condition, or determining that entropy coding does not need to be performed on the first feature element when the probability estimation result of the first feature element does not satisfy the predetermined condition.

[0016] In a possible implementation, when the probability estimation result of the first feature element is a probability value that the value of the first feature element is k, the preset condition is that the probability value that the value of the first feature element is k is less than or equal to a first threshold, where k is an integer.

[0017] k is a value within the range of possible values ​​of the first feature. For example, the range of values ​​of the first feature may be [-255, 255]. k may be set to 0, and entropy coding is performed for first features with a probability value less than or equal to 0.5. Entropy coding is not performed for first features with a probability value greater than 0.5.

[0018] In a possible implementation, the probability value of the first feature element having the value k is the maximum probability value among the probability values ​​of all possible values ​​of the first feature element.

[0019] The first threshold selected for the low-bitrate encoded bitstream is less than the first threshold selected for the high-bitrate encoded bitstream. The specific bitrate is related to the picture resolution and picture content. For example, the public Kodak dataset is used. A bitrate below 0.5 bpp is a low bitrate; otherwise, the bitrate is a high bitrate.

[0020] In the case of a specific bit rate, the first threshold value can be configured based on the actual requirements, which is not limited here.

[0021] In the foregoing technical solution, the complexity of entropy coding can be flexibly reduced based on requirements and by flexibly setting the flexible first threshold.

[0022] In a possible implementation, the probability estimation result for the first feature includes a first parameter and a second parameter of a probability distribution for the first feature.

[0023] When the probability distribution is a Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean value of the Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian distribution of the first feature element. Alternatively, when the probability distribution is a Laplace distribution, the first parameter of the probability distribution of the first feature element is the location parameter of the Laplace distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the scale parameter of the Laplace distribution of the first feature element. The preset conditions are as follows, namely: the absolute value of the difference between a first parameter of the probability distribution of the first feature element and the value k of the first feature element is equal to or greater than a second threshold; a second parameter of the probability distribution of the first feature element is greater than or equal to a third threshold; or The sum of the second parameter of the probability distribution of the first feature element and the absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element is equal to or greater than a fourth threshold. It can be one of the following:

[0024] When the probability distribution is a Gaussian mixture distribution, the first parameter of the probability distribution of the first feature element is the mean value of the Gaussian mixture distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian mixture distribution of the first feature element. The preset conditions are as follows: the sum of any variance of the Gaussian mixture distribution of the first feature element and the sum of the absolute values ​​of the differences between all mean values ​​of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is equal to or greater than a fifth threshold; The difference between any mean value of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is equal to or greater than a sixth threshold value; or Any variance of the Gaussian mixture distribution of the first feature element is equal to or greater than the seventh threshold. It can be one of the following:

[0025] When the probability distribution is an asymmetric Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean value of the asymmetric Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the first variance and the second variance of the asymmetric Gaussian distribution of the first feature element. The preset conditions are as follows: the absolute value of the difference between the mean value of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is equal to or greater than an eighth threshold; a first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to a ninth threshold; or The second variance of the asymmetric Gaussian distribution of the first feature element is equal to or greater than the tenth threshold. It can be one of the following:

[0026] When the probability distribution of the first feature is a Gaussian mixture distribution, a range of the decision value of the first feature is determined, and a plurality of mean values ​​of the probability distribution of the first feature are not within the range of the decision value of the first feature.

[0027] When the probability distribution of the first feature is a Gaussian distribution, a range of the determined values ​​of the first feature is determined, and the mean value of the probability distribution of the first feature is not within the range of the determined values ​​of the first feature.

[0028] When the probability distribution of the first feature is a Gaussian distribution, a range of decision values ​​of the first feature is determined, the range of decision values ​​including a plurality of possible values ​​of the first feature, and the absolute value of the difference between the mean parameter of the Gaussian distribution of the first feature and each value in the range of decision values ​​of the first feature is equal to or greater than an eleventh threshold, or the variance of the probability distribution of the first feature is equal to or greater than a twelfth threshold.

[0029] The value of the first feature is not within the range of determined values ​​for the first feature.

[0030] The probability value corresponding to the value of the first feature element is less than or equal to the thirteenth threshold.

[0031] In a possible implementation, the method further includes the steps of constructing a threshold candidate list for a first threshold, entering the first threshold into the threshold candidate list for the first threshold, where the first threshold has a corresponding index number, and writing the index number of the first threshold into the encoded bitstream, where the length of the threshold candidate list for the first threshold may be set to T, where T is an integer greater than or equal to 1. It may be understood that another threshold may be constructed in a manner similar to constructing the threshold candidate list for the first threshold. The other threshold has a corresponding index number that is written into the encoded bitstream.

[0032] Specifically, the index number may be written into the bitstream, stored in a sequence header, a picture header, a slice header, or supplemental enhancement information (SEI), and transmitted to the decoder side. Alternatively, another method may be used, which is not limited here. The manner of constructing the candidate list is not limited.

[0033] In another possibility, the decision information is obtained by inputting the probability estimation results into a generative network, which may be a convolutional network and may include multiple network layers, any of which may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like.

[0034] In a possible implementation, the probability estimation result of the feature data is input to a generative network to obtain decision information for the first feature element, the decision information indicating whether to perform entropy coding on the first feature element.

[0035] In a possible implementation, the decision information of the feature data is a decision map, which may also be called a decision map. The decision map is preferably a binary map, which may also be called a binary map. The value of the decision information of a feature element in the binary map is usually 0 or 1. Therefore, when the value corresponding to the position where the first feature element is located in the decision map is a preset value, entropy coding needs to be performed on the first feature element. When the value corresponding to the position where the first feature element is located in the decision map is not a preset value, entropy coding does not need to be performed on the first feature element.

[0036] In a possible implementation, the decision information of the feature element of the feature data is a preset value. The preset value of the decision information is typically 1. Therefore, when the decision information is the preset value, entropy coding needs to be performed on the first feature element. When the decision information is not the preset value, entropy coding does not need to be performed on the first feature element. The decision information may be an identifier or an identifier value. Determining whether to perform entropy coding on the first feature element depends on whether the identifier or the identifier value is a preset value. When the identifier or the identifier value is a preset value, entropy coding needs to be performed on the first feature element. When the identifier or the identifier value is not the preset value, entropy coding does not need to be performed on the first feature element. Alternatively, the decision information set of the feature element of the feature data may be a floating-point number. In other words, the value may be another value other than 0 and 1. In this case, a preset value may be set. When the value of the decision information of the first feature element is equal to or greater than the preset value, it is determined that entropy coding needs to be performed on the first feature element. When the value of the decision information of the first feature element is smaller than the preset value, it is determined that entropy coding does not need to be performed on the first feature element.

[0037] In a possible implementation, the method further includes a step of obtaining the feature data by passing the picture to be encoded through an encoder network, a step of obtaining the feature data by rounding the picture to be encoded after it has passed through the encoder network, or a step of obtaining the feature data by quantizing and rounding the picture to be encoded after it has passed through the encoder network.

[0038] The encoder network may use an autoencoder structure. The encoder network may be a convolutional neural network. The encoder network may include multiple subnets, each of which includes one or more convolutional layers. The network structures between the subnets may be the same or different.

[0039] The picture to be coded can be the original picture or a residual picture.

[0040] It should be understood that the picture to be encoded may be in RGB format, or a representation format such as YUV or RAW. Before being input to the encoder network, pre-processing operations may be performed on the picture to be encoded. The pre-processing operations may include operations such as transforming, block division, filtering, and pruning.

[0041] It should be understood that multiple pictures to be coded or multiple picture blocks to be coded are allowed to be input to the encoder and decoder network for processing within the same timestamp or at the same moment in time to obtain feature data.

[0042] According to a second aspect, obtaining a bitstream of feature data to be decoded, the feature data to be decoded includes a plurality of feature elements, the plurality of feature elements including a first feature element; obtaining a probability estimation result of a first feature element; determining whether to perform entropy decoding on the first feature based on the probability estimation result of the first feature; performing entropy decoding on the first feature only when it is determined that entropy decoding needs to be performed on the first feature; A feature data decoding method is provided, including:

[0043] It can be understood that the first feature element is any feature element of the feature data to be decoded. After all feature elements of the feature data to be decoded have been determined, and entropy decoding is or is not performed based on the determination result, the decoded feature data is obtained.

[0044] The decoded feature data may be one-dimensional, two-dimensional, or multi-dimensional data, and each piece of data is a feature element. It should be noted that the meaning of feature point and feature element in this application is the same.

[0045] Specifically, the first feature element is any feature element to be decoded of the feature data to be decoded.

[0046] In one possibility, the probability estimation process for obtaining the probability estimation result of the first feature element may be implemented by using a probability estimation network. In another possibility, the probability estimation process may use a conventional non-network probability estimation method to perform probability estimation on the feature data.

[0047] It should be noted that when only side information is used as the input of probability estimation, the probability estimation results of the feature elements can be output in parallel. When the input of probability estimation includes context information, the probability estimation results of the feature elements need to be output in serial. The number of feature elements included in the side information is less than the number of feature elements in the feature data.

[0048] Possibly, the bitstream contains side information, which needs to be decoded in the process of decoding the bitstream.

[0049] Specifically, the determination process for each feature element of the feature data includes determining a condition and determining whether to perform entropy decoding based on the result of the condition determination.

[0050] Possibly, the entropy decoding can be performed by using a neural network.

[0051] In another possibility, the entropy decoding may be performed through conventional entropy decoding.

[0052] Specifically, if the current first feature element is the Pth feature element of the feature data, after the determination of the Pth feature element is completed and entropy decoding is or is not performed based on the determination result, the determination of the (P+1)th feature element of the feature data is started, and the entropy decoding process is or is not performed based on the determination result. P is a positive integer, P is smaller than M, and M is the number of feature elements in the entire feature data. For example, for the second feature element, when it is determined that entropy decoding does not need to be performed for the second feature element, performing entropy decoding for the second feature element is skipped.

[0053] In the above technical solution, whether entropy decoding needs to be performed is determined for each feature to be decoded, so that the entropy decoding process of some features can be skipped, and the number of elements that need to be entropy decoded can be significantly reduced.In this way, the complexity of entropy decoding can be reduced.

[0054] In a possible implementation, determining whether to perform entropy decoding on a first feature element of the feature data includes determining that entropy decoding needs to be performed on the first feature element when a probability estimation result of the first feature element of the feature data satisfies a predetermined condition, or determining that entropy decoding does not need to be performed on the first feature element when the probability estimation result of the first feature element does not satisfy the predetermined condition, and setting a feature value of the first feature element to k, where k is an integer.

[0055] In a possible implementation, when the probability estimation result of the first feature element is a probability value that the value of the first feature element is k, the preset condition is that the probability value that the value of the first feature element is k is less than or equal to a first threshold, where k is an integer.

[0056] In the case where the predetermined condition is not met, the first feature is set to k. For example, the value range of the first feature may be [-255, 255]. k may be set to 0, and entropy coding is performed on first features whose probability value is less than or equal to 0.5. Entropy coding is not performed on first features whose probability value is greater than 0.5.

[0057] In another possibility, when a preset condition is not met, the value of the first feature element is determined by using a list.

[0058] In another possibility, when the preset condition is not met, the first feature is set to a fixed integer value.

[0059] k is a value within the range of possible values ​​of the first feature element.

[0060] In terms of likelihood, k is the value that corresponds to the maximum probability within the range of all possible values ​​of the first feature element.

[0061] The first threshold selected for the decoded bitstream at a low bitrate is smaller than the first threshold selected for the decoded bitstream at a high bitrate. The specific bitrate is related to the picture resolution and picture content. For example, the public Kodak dataset is used. A bitrate lower than 0.5 bpp is a low bitrate; otherwise, the bitrate is a high bitrate.

[0062] In the case of a specific bit rate, the first threshold value can be configured based on the actual requirements, which is not limited here.

[0063] In the foregoing technical solution, the complexity of entropy decoding can be flexibly reduced based on requirements and by flexibly setting the flexible first threshold.

[0064] In a possible implementation, the probability estimation result for the first feature includes a first parameter and a second parameter of a probability distribution for the first feature.

[0065] When the probability distribution is a Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean value of the Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian distribution of the first feature element. Alternatively, when the probability distribution is a Laplace distribution, the first parameter of the probability distribution of the first feature element is the location parameter of the Laplace distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the scale parameter of the Laplace distribution of the first feature element. The preset conditions are as follows, namely: the absolute value of the difference between a first parameter of the probability distribution of the first feature element and the value k of the first feature element is equal to or greater than a second threshold; The second parameter of the first feature element is equal to or greater than a third threshold; or The sum of the second parameter of the probability distribution of the first feature element and the absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element is equal to or greater than a fourth threshold. It can be one of the following:

[0066] When the probability distribution is a Gaussian mixture distribution, the first parameter of the probability distribution of the first feature element is the mean value of the Gaussian mixture distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian mixture distribution of the first feature element. The preset conditions are as follows: the sum of any variance of the Gaussian mixture distribution of the first feature element and the sum of the absolute values ​​of the differences between all mean values ​​of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is equal to or greater than a fifth threshold; The difference between any mean value of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is greater than a sixth threshold value; or Any variance of the Gaussian mixture distribution of the first feature element is equal to or greater than the seventh threshold. It can be one of the following:

[0067] When the probability distribution is an asymmetric Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean value of the asymmetric Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the first variance and the second variance of the asymmetric Gaussian distribution of the first feature element. The preset conditions are as follows: the absolute value of the difference between the mean parameter of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is greater than an eighth threshold; a first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to a ninth threshold; or The second variance of the asymmetric Gaussian distribution of the first feature element is equal to or greater than the tenth threshold. It can be one of the following:

[0068] When the probability distribution of the first feature is a Gaussian mixture distribution, a range of the decision value of the first feature is determined, and a plurality of mean values ​​of the probability distribution of the first feature are not within the range of the decision value of the first feature.

[0069] When the probability distribution of the first feature is a Gaussian distribution, a range of the determined values ​​of the first feature is determined, and the mean value of the probability distribution of the first feature is not within the range of the determined values ​​of the first feature.

[0070] When the probability distribution of the first feature is a Gaussian distribution, a range of decision values ​​of the first feature is determined, the range of decision values ​​including a plurality of possible values ​​of the first feature, and the absolute value of the difference between the mean parameter of the Gaussian distribution of the first feature and each value in the range of decision values ​​of the first feature is equal to or greater than an eleventh threshold, or the variance of the probability distribution of the first feature is equal to or greater than a twelfth threshold.

[0071] The value k of the first feature is not within the range of determined values ​​of the first feature.

[0072] The probability value corresponding to the value k of the first feature element is less than or equal to the thirteenth threshold.

[0073] In a possible implementation, a threshold candidate list for a first threshold is constructed, an index number of the threshold candidate list for the first threshold is obtained by decoding the bitstream, and a value at a position in the threshold candidate list for the first threshold corresponding to the index number of the first threshold is used as the value of the first threshold. The length of the threshold candidate list for the first threshold may be set to T, where T is an integer equal to or greater than 1. It may be understood that any other threshold may be constructed in the same manner as constructing a threshold candidate list for the first threshold. An index number corresponding to the threshold may be obtained through decoding, and a value in the constructed list is selected as the threshold based on the index number.

[0074] In another possibility, the decision information is obtained by inputting the probability estimation results into a generative network, which may be a convolutional network and may include multiple network layers, any of which may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like.

[0075] In a possible implementation, the probability estimation result of the feature data is input to a generative network to obtain decision information for the first feature element, the decision information indicating whether to perform entropy decoding on the first feature element.

[0076] In a possible implementation, the decision information of the feature elements of the feature data is a decision map, which may also be called a decision map. The decision map is preferably a binary map, which may also be called a binary map. The value of the decision information of the feature elements in the binary map is usually 0 or 1. Therefore, when the value corresponding to the position where the first feature element is located in the decision map is a preset value, entropy decoding needs to be performed on the first feature element. When the value corresponding to the position where the first feature element is located in the decision map is not the preset value, entropy decoding does not need to be performed on the first feature element.

[0077] Alternatively, the set of determination information for the feature elements of the feature data may be floating-point numbers. In other words, the values ​​may be other values ​​than 0 and 1. In this case, a preset value may be set. When the value of the determination information for the first feature element is equal to or greater than the preset value, it is determined that entropy decoding needs to be performed on the first feature element. When the value of the determination information for the first feature element is smaller than the preset value, it is determined that entropy decoding does not need to be performed on the first feature element.

[0078] In a possible implementation, the feature data is passed through a decoder network to obtain a reconstructed picture.

[0079] In another possible implementation, the feature data is passed through a decoder network to obtain machine-directed task data. Specifically, the feature data is passed through a machine-directed task module to obtain machine-directed task data, and the machine-directed module includes a target recognition network, a classification network, or a semantic segmentation network.

[0080] According to a third aspect, an acquisition module configured to acquire feature data to be encoded, the feature data to be encoded comprising a plurality of feature elements, the plurality of feature elements comprising a first feature element, the acquisition module configured to acquire a probability estimation result of the first feature element; an encoding module configured to determine whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element, and to perform entropy encoding on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element; There is provided a feature data encoding device, comprising:

[0081] For further implementation functions of the acquisition module and the encoding module, please refer to the first aspect or any one of the implementations of the first aspect, and the details will not be described again here.

[0082] According to a fourth aspect, an acquisition module configured to acquire a bitstream of feature data to be decoded, the feature data to be decoded comprising a plurality of feature elements, the plurality of feature elements including a first feature element, the acquisition module configured to acquire a probability estimation result of the first feature element; a decoding module configured to determine whether to perform entropy decoding on the first feature element based on the probability estimation result of the first feature element, and to perform entropy decoding on the first feature element only when it is determined that entropy decoding needs to be performed on the first feature element; A feature data decoding device is provided, comprising:

[0083] For further implementation functions of the acquisition module and the decoding module, please refer to the second aspect or any one of the implementations of the second aspect, and the details will not be described again here.

[0084] According to a fifth aspect, the application provides an encoder including a processing circuit configured to determine the method according to any one of the first aspect and the first aspect.

[0085] According to a sixth aspect, the application provides a decoder including a processing circuit configured to determine the method according to any one of the second aspect and the second aspect.

[0086] According to a seventh aspect, the present application provides a computer program product including program code, which, when executed in a computer or processor, is used to determine a method according to any one of the first aspect and the first embodiment, and a method according to any one of the second aspect and the second embodiment.

[0087] According to an eighth aspect, the present application provides an encoder including one or more processors, and a non-transitory computer-readable storage medium coupled to the processors and storing a program determined by the processors, wherein when the program is determined by the processors, a decoder is enabled to determine the method according to any one of the first aspect and the first aspect.

[0088] According to a ninth aspect, the present application provides a decoder including one or more processors, and a non-transitory computer-readable storage medium coupled to the processors and storing a program determined by the processors, wherein when the program is determined by the processors, an encoder is enabled to determine the method according to any one of the second aspect and the second aspect.

[0089] According to a tenth aspect, the present application provides a non-transitory computer-readable storage medium containing program code, which, when determined by a computer device, is used to determine the method according to any one of the first aspect and the first aspect, and the method according to any one of the second aspect and the second aspect.

[0090] According to an eleventh aspect, the present invention relates to an encoding device having a function for performing the behavior according to the first aspect or any one of the method embodiments of the first aspect. The function may be implemented by hardware or by hardware that determines corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. In a possible design, the encoding device includes: an acquisition module configured to transform an original picture or a residual picture into a feature space by using an encoder network and extract feature data for compression, where probability estimation is performed on the feature data to obtain probability estimation results for feature elements of the feature data; and an encoding module configured to determine whether entropy coding is performed on the feature elements of the feature data based on a specific condition by using the probability estimation results for the feature elements of the feature data, and complete the encoding process for all feature elements of the feature data to obtain an encoded bitstream of the feature data. These modules may determine corresponding functions in the example of the first aspect and any one of the method embodiments according to the first aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0091] According to a twelfth aspect, the present invention relates to a decoding device having a function for performing the behavior according to the second aspect or any one of the method embodiments of the second aspect. The function may be implemented by hardware or by hardware that determines corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. In a possible design, the decoding device includes: an acquisition module configured to acquire a bitstream of feature data to be decoded and perform probability estimation based on the bitstream of feature data to be decoded to acquire probability estimation results for feature elements of the feature data; and a decoding module configured to use the probability estimation results for the feature elements of the feature data to determine whether entropy decoding is performed on the feature elements of the feature data based on a specific condition, complete the decoding process for all feature elements of the feature data to acquire feature data, and decode the feature data to acquire a reconstructed picture or machine-oriented task data. These modules may determine corresponding functions in the example of the second aspect and any one of the method embodiments according to the second aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0092] According to a thirteenth aspect, obtaining feature data to be encoded, the feature data comprising a plurality of feature elements, the plurality of feature elements comprising a first feature element; obtaining side information of the feature data, and inputting the side information of the feature data into a joint network to obtain judgment information of a first feature element; determining whether to perform entropy coding on the first feature element based on the determination information of the first feature element; performing entropy coding on the first feature only when it is determined that entropy coding needs to be performed on the first feature; A feature data encoding method is provided, including:

[0093] The feature data is one-dimensional, two-dimensional, or multi-dimensional data output by the encoder network, each of which is a feature element.

[0094] Possibly, side information of the feature data can be coded into the bitstream, the side information being feature information that is further extracted by inputting the feature data into a neural network, and the number of feature elements included in the side information is less than the number of feature elements of the feature data.

[0095] The first feature element is an arbitrary feature element of the feature data.

[0096] Possibly, the set of decision information of the feature elements of the feature data can be represented in a manner such as a decision map, which can be one-dimensional, two-dimensional, or multi-dimensional picture data, and the size of the decision map corresponds to the size of the feature data.

[0097] In a possible case, the joint network further outputs a probability estimation result of the first feature, which includes a probability value of the first feature and / or a first parameter and a second parameter of the probability distribution.

[0098] In the above technical solution, whether entropy coding needs to be performed is determined for each feature to be coded, so that the entropy coding process of some features can be skipped, and the number of elements that need to be entropy coded can be significantly reduced. In this way, the complexity of entropy coding can be reduced.

[0099] Possibly, when the value corresponding to the position where the first feature is located in the decision map is a preset value, entropy coding needs to be performed on the first feature. When the value corresponding to the position where the first feature is located in the decision map is not a preset value, entropy coding does not need to be performed on the first feature.

[0100] According to a fourteenth aspect, obtaining a bitstream of feature data to be decoded and side information of the feature data to be decoded, the feature data to be decoded includes a plurality of feature elements, the plurality of feature elements including a first feature element; inputting side information of feature data to be decoded into a joint network to obtain decision information of a first feature element; determining whether to perform entropy decoding on the first feature element based on the determination information of the first feature element; performing entropy decoding on the first feature only when it is determined that entropy decoding needs to be performed on the first feature; A feature data decoding method is provided, including:

[0101] Possibly, the bitstream of feature data to be decoded is decoded to obtain side information, the number of feature elements contained in the side information being less than the number of feature elements in the feature data.

[0102] The first feature element is an arbitrary feature element of the feature data.

[0103] Possibly, the decision information of the feature elements of the feature data can be expressed in a manner such as a decision map, which can be one-dimensional, two-dimensional, or multi-dimensional picture data, and the size of the decision map corresponds to the size of the feature data.

[0104] In a possible case, the joint network further outputs a probability estimation result of the first feature, which includes a probability value of the first feature and / or a first parameter and a second parameter of the probability distribution.

[0105] Possibly, when the value corresponding to the position where the first feature element is located in the decision map is a preset value, entropy decoding needs to be performed on the first feature element. When the value corresponding to the position where the first feature element is located in the decision map is not a preset value, entropy decoding does not need to be performed on the first feature element, and the feature value of the first feature element is set to k, where k is an integer.

[0106] In the above technical solution, whether entropy decoding needs to be performed is determined for each feature to be coded, so that the entropy decoding process of some features can be skipped, and the number of elements that need to be entropy decoded can be significantly reduced. In this way, the complexity of entropy decoding can be reduced.

[0107] In existing mainstream end-to-end feature data encoding and decoding solutions, the entropy encoding and decoding or arithmetic encoding and decoding processes are overly complex. In this application, information about the probability distribution of feature points in the feature data to be encoded is used to determine whether entropy encoding needs to be performed for each feature element of the feature data to be encoded and whether entropy decoding needs to be performed for each feature element of the feature data to be decoded, thereby skipping the entropy encoding and decoding processes of some feature elements and significantly reducing the number of elements that need to be encoded and decoded. This reduces the complexity of encoding and decoding. In another aspect, the threshold can be flexibly set based on the requirement of the actual value of the bitstream bitrate to control the value of the bitrate of the generated bitstream.

[0108] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0109] The following describes the accompanying drawings used in the embodiments of this application. [Brief explanation of the drawings]

[0110] [Figure 1A] FIG. 1 is an example block diagram of a picture decoding system. [Figure 1B] 1 is an implementation of a processing circuit for a picture decoding system. [Figure 1C] FIG. 1 is a schematic block diagram of a picture decoding device. [Figure 1D] 1 is a diagram illustrating an implementation of an apparatus according to an embodiment of the present application. [Figure 2A] FIG. 1 is a system architecture diagram for a possible scenario according to this application. [Figure 2B] FIG. 1 is a system architecture diagram for a possible scenario according to this application. [Figure 3A] FIG. 2 is a schematic block diagram of an encoder. [Figure 3B] FIG. 2 is a schematic block diagram of an encoder. [Figure 3C] FIG. 2 is a schematic block diagram of an encoder. [Figure 3D] FIG. 2 is a schematic block diagram of an encoder. [Figure 4A] FIG. 2 is a schematic diagram of an encoder network unit. [Figure 4B] FIG. 2 is a schematic diagram of the network structure of an encoder network. [Figure 5] FIG. 10 is a schematic diagram of the structure of an encoding decision implementation unit. [Figure 6] FIG. 10 is an exemplary output diagram of a joint network. [Figure 7] FIG. 10 is an exemplary output diagram of a generative network. [Figure 8] FIG. 10 is a schematic implementation diagram of a decoding decision implementation. [Figure 9] FIG. 1 is an exemplary diagram of a network structure of a decoder network. [Figure 10A] FIG. 2 is an exemplary diagram of a coding method according to an embodiment of the present application. [Figure 10B] FIG. 2 is a schematic block diagram of a picture feature map decoder according to an embodiment of the present application; [Figure 11A] FIG. 2 is an exemplary diagram of a coding method according to an embodiment of the present application. [Figure 12] FIG. 1 is an exemplary diagram of a network structure of a side information extraction module. [Figure 13A] FIG. 2 is an exemplary diagram of a coding method according to an embodiment of the present application. [Figure 13B] FIG. 2 is a schematic block diagram of a picture feature map decoder according to an embodiment of the present application; [Figure 14] FIG. 2 is an exemplary diagram of a coding method according to an embodiment of the present application. [Figure 15] FIG. 1 is an exemplary diagram of a network structure of a joint network. [Figure 16]FIG. 2 is a schematic block diagram of a picture feature map decoder according to an embodiment of the present application; [Figure 17] FIG. 2 is an exemplary diagram of a coding method according to an embodiment of the present application. [Figure 18] 1 is a schematic diagram of an exemplary structure of an encoding device according to the present application; [Figure 19] 1 is a schematic diagram of an exemplary structure of a decoding device according to the present application; DETAILED DESCRIPTION OF THE INVENTION

[0111] Terms such as "first" and "second" in the embodiments of this application are used only for distinction and description, but cannot be understood as indicating or implying relative importance or order. In addition, the terms "include," "comprise," and any variations thereof are intended to cover non-exclusive inclusions, for example, the inclusion of a series of steps or units. A method, system, product, or device is not necessarily limited to the explicitly recited steps or units, but may include other steps or units that are not explicitly recited and that are inherent to the process, method, product, or device.

[0112] In this application, it should be understood that "at least one (item)" refers to one or more, and "multiple" refers to two or more. The term "and / or" describes an association relationship between associated objects and indicates that three relationships may exist. For example, "A and / or B" may indicate three cases: only A is present, only B is present, and both A and B are present, where A and B may be singular or plural. The character " / " generally indicates an "or" relationship between associated objects. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of a single item or multiple items. For example, at least one of a, b, or c may represent a, b, c, "a and b," "a and c," "b and c," or "a, b, and c," where a, b, and c may be singular or plural.

[0113] An embodiment of the present application provides an AI-based feature data encoding and decoding technology, in particular, a neural network-based picture feature map and / or audio feature variable encoding and decoding technology, and more specifically, an end-to-end based picture feature map and / or audio feature variable encoding and decoding system.

[0114] In the field of picture coding, the terms "picture" and "image" may be used synonymously. Picture coding (or commonly referred to as coding) includes two parts: picture encoding and picture decoding. Video includes multiple pictures and is a representation method for consecutive pictures. Picture encoding is determined at the source side and typically involves processing (e.g., compressing) the original video picture to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Picture decoding is determined at the destination side and typically involves a reverse process compared to the encoder's process to reconstruct the picture. Embodiments referring to "coding" of a picture or audio shall be understood as "encoding" or "decoding" of a picture or audio. The combination of the encoding and decoding parts is also referred to as encoding and decoding (CODEC).

[0115] In the case of lossless picture coding, the original picture can be reconstructed. In other words, the reconstructed picture has the same quality as the original picture (assuming that no transmission loss or other data loss occurs during storage or transmission). In the case of conventional lossy picture coding, further compression is determined, for example through quantization, to reduce the amount of data required to represent the video picture, and the video picture cannot be completely reconstructed at the decoder side. In other words, the quality of the reconstructed video picture is lower or worse than that of the original video picture.

[0116] Since the embodiments of this application relate to large-scale applications of neural networks, for ease of understanding, the following describes terms and concepts related to neural networks that may be used in the embodiments of this application.

[0117] (1) Neural Networks

[0118] A neural network may include neurons, which take as input x s and an operation unit using an intercept of 1. The output of the operation unit may be:

number

[0119] s=1, 2, ..., or n, where n is a natural number greater than 1, and W s x s where is the weight of the neuron, and b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce nonlinear features into the neural network to convert the input signal in the neuron into an output signal. The output signal of the activation function may be used as the input of the next convolutional layer, and the activation function may be a sigmoid function. A neural network is a network constructed by connecting multiple single neurons together. Specifically, the output of a neuron can be the input of another neuron. The input of each neuron can be connected to a local receptive field in the previous layer to extract features of the local receptive field. A local receptive field can be a region containing several neurons.

[0120] (2) Deep Neural Networks

[0121] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. A DNN is divided based on the location of different layers. The neural network inside a DNN can be classified into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layers are hidden layers. The layers are fully connected. Specifically, every neuron in the i-th layer is necessarily connected to every neuron in the (i+1)-th layer.

[0122] Although DNNs may seem complicated, the work done at each layer is not. Simply put, a DNN is a linear relationship:

number

number

number

number

number

number

number

number

[0123] In conclusion, the coefficient from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer is

number

[0124] It should be noted that the input layer does not have the parameter W. In a deep neural network, the more hidden layers, the more capable the network is of describing complex cases in the real world. Theoretically, a model with more parameters has higher complexity and greater "capability," which indicates that the model can complete more complex learning tasks. The process of training a deep neural network is a process of learning weight matrices, and the ultimate goal of training is to obtain the weight matrices of all layers in the deep neural network being trained (weight matrices that include the vector W in multiple layers).

[0125] (3) Convolutional Neural Networks

[0126] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network includes a feature extractor including a convolutional layer and a subsampling layer, and the feature extractor can be considered a filter. A convolutional layer is a neuron layer within a convolutional neural network where convolution processing is performed on an input signal. In a convolutional layer of a convolutional neural network, one neuron may be connected to only a portion of neurons in an adjacent layer. A convolutional layer usually includes several feature planes, and each feature plane may include several neurons arranged in a rectangular shape. Neurons in the same feature plane share weights, and the shared weights here are convolution kernels. Weight sharing can be understood as a position-independent picture information extraction method. The convolution kernels can be initialized in the form of a random-size matrix. In the process of training a convolutional neural network, the convolution kernels can acquire appropriate weights through learning. Additionally, a direct benefit of weight sharing is that it reduces the connections between layers in a convolutional neural network, lowering the risk of overfitting.

[0127] (4) Entropy coding

[0128] Entropy coding is used, for example, to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique) to the quantized coefficients and other syntax elements so that a decoder, or the like, may receive and use parameters for decoding to obtain coded data that may be output through an output in the form of a coded bitstream. The coded bitstream may be transmitted to a decoder or stored in a memory for subsequent transmission or retrieval by the decoder.

[0129] In the following embodiments of the coding system 10, the encoder 20A and the decoder 30A are described based on FIGS. 1A to 15. FIG.

[0130] 1A is a schematic block diagram illustrating an example coding system 10, e.g., a picture (or audio) coding system 10 (or coding system 10 for short), that may utilize the techniques of this application. Encoder 20A and decoder 30A in picture coding system 10 represent devices and the like that may be configured to determine various techniques based on various examples described in this application.

[0131] As shown in FIG. 1A, coding system 10 includes a source device 12 configured to provide an encoded bitstream 21, e.g., encoded pictures (or audio), for a destination device 14 configured to decode the encoded bitstream 21.

[0132] The source device 12 includes an encoder 20A, and optionally includes a picture source 16, a preprocessor (or preprocessing unit) 18, a communication interface (or communication unit) 26, and a probability estimation (or probability estimation unit) 40.

[0133] Picture (or audio) source 16 may include or be any type of picture capture device configured to capture real-world pictures (or audio) and / or any type of picture generation device, e.g., a computer graphics processing unit configured to generate computer-animated pictures, or any type of device configured to acquire and / or provide real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Audio or picture source may be any type of memory or storage device that stores any of the aforementioned audio or pictures.

[0134] In distinction from the preprocessor (or preprocessing unit) 18 and the processing determined by the preprocessor (or preprocessing unit) 18, the picture or audio (picture or audio data) 17 may also be referred to as the original picture or audio (original picture or audio data) 17.

[0135] The pre-processor 18 is configured to receive (original) picture (or audio) data 17 and perform pre-processing on the picture (or audio) data 17 to obtain pre-processed picture or audio (or pre-processed picture or audio data) 19. For example, the pre-processing determined by the pre-processor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component.

[0136] The encoder 20A includes an encoder network 20, an entropy coding 24, and optionally a pre-processor 22.

[0137] A picture (or audio) encoder network (or encoder network) 20 is configured to receive the pre-processed picture (or audio) data 19 and to provide encoded picture (or audio) data 21 .

[0138] The pre-processor 22 is configured to receive the feature data 21 to be encoded and pre-process the feature data 21 to obtain pre-processed feature data to be encoded 23. For example, the pre-processing determined by the pre-processor 22 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 22 may be an optional component.

[0139] The entropy coding 24 is used to receive feature data to be coded (or pre-process feature data to be coded) 23 and generate a coded bitstream 25 based on the probability estimation results 41 provided by the probability estimation 40.

[0140] The communication interface 26 of the source device 12 may be configured to receive the encoded bitstream 25 and transmit the encoded bitstream 25 (or any further processed version thereof) over a communication channel 27 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0141] The destination device 14 includes a decoder 30A, and may optionally include a communications interface (or communications unit) 28, a post-processor (or post-processing unit) 36, and a display device 38.

[0142] The communications interface 28 of the destination device 14 is configured to receive the encoded bitstream 25 (or any further processed version thereof) directly from the source device 12 or from any other source device, such as a storage device, e.g., a storage device for encoded bitstream data, and to provide the encoded bitstream 25 for the decoder 30A.

[0143] The communication interface 26 and the communication interface 28 may be configured to transmit or receive the encoded bitstream (or encoded bitstream data) 25 over a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or over any type of network, e.g., a wired or wireless network or any combination thereof, or any type of private and public network, or any combination thereof.

[0144] The communications interface 26 may be configured to process the encoded bitstream 25, for example, by packaging the encoded bitstream 25 into an appropriate format, e.g., packets, and / or by using any type of transmission coding or processing for transmission over a communications link or network.

[0145] The communication interface 28 corresponds to the communication interface 26 and may be configured, for example, to receive transmitted data and process the transmitted data to obtain the encoded bitstream 25 by using any type of corresponding transmitted decoding or processing and / or decapsulation.

[0146] Both communication interface 26 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow for communication channel 27 in FIG. 1A going from source device 12 to destination device 14, or as bidirectional communication interfaces, and may be configured to send and receive messages, set up connections, acknowledge and exchange any other information related to the communication link and / or data transmission, e.g., encoded picture data transmission.

[0147] The decoder 30A includes a decoder network 34, an entropy decoding 30, and optionally a post-processor 32.

[0148] The entropy decoder 30 is used to receive the encoded bitstream 25 and provide decoded feature data 31 based on the probability estimation results 42 provided by the probability estimation 40 .

[0149] The post-processor 32 is configured to perform post-processing on the decoded feature data 31 to obtain post-processed decoded feature data 33. The post-processing determined by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling. It may be understood that the post-processing unit 32 may be an optional component.

[0150] A decoder network 34 is used to receive the decoded feature data 31 or the post-processed decoded feature data 33 and to provide reconstructed picture data 35 .

[0151] The post-processor 36 is configured to perform post-processing on the reconstructed picture data 35 to obtain post-processed reconstructed picture data 37. The post-processing determined by the post-processing unit 36 ​​may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling. It may be understood that the post-processing unit 36 ​​may be an optional component.

[0152] Display device 38 is configured to receive reconstructed picture data 35 or post-processed picture data 37 for displaying the picture to a user, viewer, or the like. Display device 38 may be or include any type of player or display for presenting the reconstructed audio or picture, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of another display screen.

[0153] 1A depicts source device 12 and destination device 14 as separate devices, embodiments of the devices may alternatively include both source device 12 and destination device 14, or may include the functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In these embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0154] Based on the description, the existence and (exact) division of different units or functions of source device 12 and / or destination device 14 depicted in Figure 1A may vary with actual devices and applications, as will be apparent to those skilled in the art.

[0155] The feature data encoder 20A (e.g., a picture feature map encoder or an audio feature variable encoder), the feature data decoder 30A (e.g., a picture feature map decoder or an audio feature variable decoder), or both the feature data encoder 20A and the feature data decoder 30A may be implemented using the processing circuitry depicted in FIG. 1B , such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated processors for picture encoding, or any combination thereof. The feature data encoder 20A may be implemented using a processing circuitry 56, and the feature data decoder 30A may be implemented using a processing circuitry 56. The processing circuitry 56 may be configured to determine various operations as follows: If the techniques are implemented partially in software, the device may store instructions for the software on a suitable non-transitory computer-readable storage medium or may determine the instructions in hardware by using one or more processors to determine the techniques of the present invention. As shown in FIG. 1B, one of the feature data encoder 20A and the feature data decoder 30A may be integrated into a single device as part of a combined encoder / decoder (CODEC).

[0156] Source device 12 and destination device 14 may include any one of a variety of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, and the like, and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0157] In some cases, the coding system 10 depicted in FIG. 1A is merely an example. The techniques provided in this application may be applicable to a picture feature map or audio feature variable coding configuration (e.g., picture feature map encoding or picture feature map decoding), where the configuration does not necessarily include any data communication between the encoding device and the decoding device. In another example, data may be retrieved from local memory, transmitted over a network, and the like. A picture feature map or audio feature variable encoding device may encode data and store the data in memory, and / or a picture feature map or audio feature variable decoding device may retrieve data from memory and decode the data. In some examples, the encoding and decoding are determined by devices that do not communicate with each other but encode data in memory and / or retrieve data from memory and decode data.

[0158] 1B is a diagram of an example coding system 50 including the feature data encoder 20A of FIG. 1A and / or the feature data decoder 30A of FIG. 1B, according to an exemplary embodiment. The coding system 50 may include an imaging (or audio generation) device 51, the encoder 20A and the decoder 30A (and / or the feature data encoder / decoder implemented by using processing circuitry 56), an antenna 52, one or more processors 53, one or more memory stores 54, and / or a display (or audio playback) device 55.

[0159] 1B, imaging (or audio generation) device 51, antenna 52, processing circuitry 56, encoder 20A, decoder 30A, processor 53, memory storage 54, and / or display (or audio playback) device 55 can be in communication with one another. In different examples, coding system 50 may include only encoder 20A or only decoder 30A.

[0160] In some examples, antenna 52 may be configured to transmit or receive an encoded bitstream of feature data. Additionally, in some examples, display (or audio playback) device 55 may be configured to present picture (or audio) data. Processing circuitry 56 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, and the like. Coding system 50 may also include optional processor 53. Similarly, optional processor 53 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, an audio processor, a general-purpose processor, and the like. Additionally, memory storage 54 may be any type of memory, for example, volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)), or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 54 may be implemented by using cache memory. In another example, processing circuitry 56 may include memory (e.g., a cache) configured to implement a picture buffer.

[0161] In some examples, encoder 20A implemented using logic circuitry may include a picture buffer (e.g., implemented using processing circuitry 56 or memory storage 54) and a graphics processing unit (e.g., implemented using processing circuitry 56). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include encoder 20A implemented using processing circuitry 56. The logic circuitry may be configured to determine various operations in the specification.

[0162] 1B and / or any other decoder system or subsystem described herein. In some examples, decoder 30A implemented using logic circuitry may include a picture buffer (e.g., implemented using processing circuitry 56 or memory storage 54) and a graphics processing unit (e.g., implemented using processing circuitry 56). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include picture decoder 30A implemented using processing circuitry 56.

[0163] In some examples, antenna 52 may be configured to receive an encoded bitstream of picture data. As described above, the encoded bitstream may include data, indicators, index values, mode selection data, and the like, described in the specification, related to audio or video frame coding, e.g., data related to coding partitions. Coding system 50 may also include a decoder 30A coupled to antenna 52 and configured to decode the encoded bitstream. Display (or audio playback) device 55 may be configured to present the pictures (or audio).

[0164] In this embodiment of the application, for the examples described with reference to encoder 20A, it should be understood that decoder 30A may be configured to determine the inverse process. For signaling syntax elements, decoder 30A may be configured to receive and parse the syntax elements and, correspondingly, decode the associated picture data. In some examples, encoder 20A may perform entropy coding on the syntax elements to obtain an encoded bitstream. In examples, decoder 30A may parse the syntax elements and, correspondingly, decode the associated picture data.

[0165] 1C is a schematic diagram of a coding device 400 according to an embodiment of the present invention. The coding device 400 is suitable for implementing the disclosed embodiments described in the specification. In an embodiment, the coding device 400 may be a decoder, such as the picture feature map decoder 30A of FIG. 1A, or an encoder, such as the picture feature map encoder 20A of FIG. 1A.

[0166] The picture coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 configured to receive data, a processor, logic unit, or central processing unit (CPU) 430 configured to process the data (e.g., the processor 430 may be a neural network processing unit 430), a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) configured to transmit the data, and a memory 460 configured to store the data. The picture (or audio) coding device 400 may further include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 for the egress or ingress of optical or electrical signals.

[0167] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a coding module 470 based on a neural network NN). The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 determines, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 significantly improves the functionality of the coding device 400 and affects the switching of the coding device 400 to different states. Alternatively, the coding module 470 is implemented by using instructions stored in the memory 460 and determined by the processor 430.

[0168] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store programs when such programs are selected for determination and to store instructions and data read during program determination. Memory 460 may be volatile and / or nonvolatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0169] FIG. 1D is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 of FIG. 1A, according to an embodiment.

[0170] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, existing or later developed, that can manipulate or process information. While the disclosed implementations may be implemented with a single processor, such as the processor 502 depicted in the figure, advantages in speed and efficiency can be achieved by using more than one processor.

[0171] In implementation, the memory 504 in the apparatus 500 can be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 through a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to determine the methods described herein. For example, the application programs 510 may include applications 1 through N and further include a picture coding application that determines the methods described herein.

[0172] Apparatus 500 may further include one or more output devices, such as a display 518. In an example, display 518 may be a touch-sensitive display that combines a display with touch-sensing elements that may be configured to sense touch input. Display 518 may be coupled to processor 502 through bus 512.

[0173] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Additionally, secondary memory may be directly coupled to another component of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Accordingly, device 500 may have a variety of configurations.

[0174] FIG. 2A illustrates a possible system architecture 1800 in a picture feature map or audio feature variable encoding and decoding scenario, including: Capture Device 1801: A video capture device completes the capture of the original video (or audio); Pre-capture processing 1802: A series of pre-processing steps are performed on the original video (or audio) capture to obtain the video (or audio) data. Encoding 1803: Video (audio) encoding is used to reduce coding redundancy and reduce the amount of data transmission in the picture feature map or audio feature variable compression process; Transmitting 1804: The compressed and encoded bitstream data obtained through encoding is transmitted by using a transmitting module; Receiving 1805: The compressed and encoded bitstream data is received by a receiving module through a network transmission; Bitstream decoding 1806: Bitstream decoding is performed on the bitstream data; and Rendering display (or playback) 1807: Rendering display (or playback) is performed on the decoded data.

[0175] FIG. 2B illustrates a possible system architecture 1900 for a scenario of machine-oriented tasks for picture feature maps (or audio feature variables), including: Feature Extraction 1901: Feature extraction is performed on the picture (or audio) source; Side information extraction 1902: Side information extraction is performed on the data obtained through feature extraction; Probability estimation 1903: The side information is used as an input for probability estimation, and probability estimation is performed on the feature maps (or feature variables) to obtain a probability estimation result; Encoding 1904: Entropy coding is performed on the data obtained through feature extraction with reference to the probability estimation result to obtain a bitstream; Optionally, before the encoding is performed, a quantization or rounding operation is performed on the data obtained through feature extraction, and the quantized or rounded data obtained through feature extraction is encoded; Optionally, entropy coding is performed on the side information so that the bitstream includes side information data. Decoding 1905: Entropy decoding is performed on the bitstream with reference to the probability estimation result to obtain a picture feature map (or audio feature variable); Optionally, if the bitstream includes side information coded data, entropy decoding is performed on the side information coded data, and the decoded side information data is used as an input of probability estimation to obtain a probability estimation result; It should be noted that when only side information is used as the input of probability estimation, the probability estimation results of the feature elements may be output in parallel; when the input of probability estimation includes context information, the probability estimation results of the feature elements need to be output in series; the side information is feature information that is further extracted by inputting picture feature maps or audio feature variables into a neural network, and the quantity of feature elements included in the side information is less than the quantity of feature elements in the picture feature maps or audio feature variables; and optionally, the side information of the picture feature maps or audio feature variables may be coded into the bitstream; and Machine Vision Task 1906: A machine vision (or auditory) task is performed on the decoded feature maps (or feature variables).

[0176] Specifically, the decoded feature data is input into a machine vision (or auditory) task network, which outputs one-dimensional, two-dimensional, or multi-dimensional data related to the vision (or auditory) task, such as classification, target recognition, and semantic segmentation.

[0177] In a possible implementation, in the implementation process of system architecture 1900, the feature extraction and encoding processes are performed on the terminal, and the decoding and machine vision tasks are performed in the cloud.

[0178] Encoder 20A may be configured to receive picture (or picture data) or audio (or audio data) 17 through input 202 or the like. The received picture, picture data, audio, and audio data may alternatively be preprocessed picture (or preprocessed picture data) or audio (or preprocessed audio data) 19. For ease of brevity, the following description uses picture (or audio) 17. Picture (or audio) 17 may alternatively be referred to as the current picture or the picture to be encoded (especially when the current picture is distinguished from other pictures in video encoding, e.g., the other pictures are in the same video sequence, i.e., include previous coded and / or decoded pictures in the video sequence of the current picture), or current audio or the audio to be coded.

[0179] A (digital) picture is, or can be considered as, a two-dimensional array or matrix of samples with intensity values. The samples in the array may also be called pixels (or pels) (an abbreviation for picture element). The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, three color components are usually employed. Specifically, a picture may be represented as or contain three sample arrays. In an RGB format or color space, the picture contains corresponding red, green, and blue sample arrays. Similarly, each pixel may be represented in a luma or chroma format or color space, e.g., YCbCr, which contains a luma component denoted by Y (sometimes L is used instead) and two chroma components denoted by Cb and Cr. The luma component Y represents brightness or gray-level intensity (e.g., these two are the same in a grayscale picture), while the two chrominance (abbreviated as chroma) components Cb and Cr represent chromaticity or color information components. Correspondingly, a picture in YCbCr format includes a luma sample array of luma sample values ​​(Y) and two chroma sample arrays of chroma values ​​(Cb and Cr). A picture in RGB format may be converted or transformed to YCbCr format or vice versa, a process also known as color transformation or conversion. If a picture is monochrome, the picture may include only a luma sample array. Correspondingly, a picture may be, for example, an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats. Picture encoder 20A does not limit the color space of a picture.

[0180] Possibly, embodiments of encoder 20A may include a picture (or audio) partitioning unit (not depicted in FIG. 1A or 1B ) configured to partition picture (or audio) 17 into multiple (usually non-overlapping) picture blocks 203 or audio segments. These picture blocks may also be called root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The partitioning unit may be configured to use the same block size for all pictures of a video sequence and a corresponding grid defining the block sizes, or to vary the block size among pictures, picture subsets, or groups of pictures, and partition each picture into corresponding blocks.

[0181] In another possibility, the encoder may be configured to directly receive blocks 203 of picture 17, e.g., one, some, or all of the blocks forming picture 17. Picture blocks 203 may also be referred to as current picture blocks or picture blocks to be coded.

[0182] Like picture 17, picture block 203 is also, or may be considered to be, a two-dimensional array or matrix of samples having intensity values ​​(sample values), albeit with smaller dimensions than picture 17. In other words, block 203 may include, for example, one sample array (e.g., a luma array for a monochrome picture 17, or a luma or chroma array for a color picture), three sample arrays (e.g., one luma array and two chroma arrays for a color picture 17), or any other quantity and / or type of array depending on the applied color format. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Correspondingly, a block may be, for example, an array of M×N (M columns×N rows) samples or an array of M×N transform coefficients.

[0183] In another possibility, the encoder 20A depicted in FIGS. 1A and 1B or 3A to 3D is configured to encode the picture 17 block by block.

[0184] In another possibility, the encoder 20A depicted in FIGS. 1A and 1B or 3A to 3D is configured to encode the picture 17.

[0185] 1A and 1B or 3A-3D may be further configured to partition or encode a picture by using slices (also called video slices), where a picture may be partitioned or encoded by using one or more (usually non-overlapping) slices. Each slice may include one or more blocks (e.g., coding tree units CTU), or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standard or subpictures in the VVC standard).

[0186] In another possibility, the encoder 20A depicted in Figures 1A and 1B or 3A-3D may be further configured to partition and / or encode a picture by using slice / tile groups (also called video tile groups) and / or tiles (also called video tiles), where a picture may be partitioned or encoded by using one or more slice / tile groups (typically non-overlapping), each of which may include one or more blocks (e.g., CTUs) or one or more tiles. Each tile may be rectangular in shape and may include one or more full or fractional blocks (e.g., CTUs).

[0187] Encoder Network 20

[0188] The encoder network 20 is configured to obtain a picture feature map or audio feature variables based on input data by using the encoder network.

[0189] 4A includes multiple network layers, any of which may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like.

[0190] Possibly, the input of the encoder network 20 is at least one picture to be coded or at least one picture block to be coded. The picture to be coded can be an original picture, a lossy picture, or a residual picture.

[0191] 4B shows an example of the network structure of the encoder network 20. In this example, it can be seen that the encoder network includes five network layers, specifically, three convolutional layers and two nonlinear activation layers.

[0192] Rounding 24

[0193] Rounding is used to round the picture feature map or audio feature variable, for example by using scalar quantization or vector quantization, to obtain a rounded picture feature map or audio feature variable.

[0194] Possibly, the encoder 20A may be configured to output a quantization parameter (QP), e.g., output the quantization parameter directly or output the quantization parameter after it has been encoded or compressed by an encoding decision implementation unit, so that, for example, the decoder 30A may receive and apply the quantization parameter for decoding.

[0195] Possibly, the output feature map or feature audio feature variables are pre-processed before rounding, where the pre-processing may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, noise removal, or the like.

[0196] Probability estimate: 40

[0197] The probability estimation result of the picture feature map or audio feature variable is obtained based on the input feature map or feature variable information through probability estimation.

[0198] The probability estimation is used to perform probability estimation on the rounded picture feature maps or audio feature variables.

[0199] The probability estimation may be a probability estimation network, which is a convolutional network, including a convolutional layer and a nonlinear activation layer. Figure 4B is used as an example. The probability estimation network includes five network layers, specifically, three convolutional layers and two nonlinear activation layers. The probability estimation may be achieved by using a convolutional non-network probability estimation method. The probability estimation method includes, but is not limited to, maximum likelihood estimation, maximum a posteriori probability estimation, maximum likelihood estimation, and other statistical methods.

[0200] Encoding decision implementation 26

[0201] As shown in Figure 5, the coding decision implementation includes coding element determination and entropy coding. Picture feature maps or audio feature variables are one-dimensional, two-dimensional, or multi-dimensional data output by the encoder network, each of which is a feature element.

[0202] Coding element determination 261

[0203] The coding element determination is to determine each feature element of the picture feature map or audio feature variable based on the probability estimation result information of the probability estimation, and to determine the specific feature element on which entropy coding is performed based on the determination result.

[0204] After the element determination process of the Pth feature element of the picture feature map or audio feature variable is completed, the element determination process of the (P+1)th feature element of the picture feature map is started, where P is a positive integer and P is less than M.

[0205] Entropy Coding 262

[0206] The entropy encoding may use various disclosed entropy encoding algorithms to perform entropy encoding, such as a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an entropy coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or another entropy encoding method or technique. Encoded picture data 25 is obtained, which may be output in the form of an encoded bitstream 25 or the like via output 212, so that a decoder 30A or the like may receive and use parameters for decoding. The encoded bitstream 25 may be transmitted to the decoder 30A or stored in a memory for subsequent transmission or retrieval by the decoder 30A.

[0207] In another possibility, the entropy coding may be performed by using an entropy coding network, for example implemented by using a convolutional network.

[0208] In a possibility, since the entropy coding does not know the actual character probabilities of the rounded feature maps, the actual character probabilities of the rounded feature maps or related information may be collected and added to the entropy coding, and the information is transmitted to the decoder side.

[0209] Joint Network 44

[0210] The joint network obtains probability estimation results and decision information of picture feature maps or audio feature variables based on the input side information. The joint network is a multi-layer network, and the joint network may be a convolutional network, which includes a convolutional layer and a nonlinear activation layer. Any network layer of the joint network may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like.

[0211] The judgment information may be one-dimensional, two-dimensional, or multi-dimensional data, and the size of the judgment information may match the size of the picture feature map.

[0212] The decision information may be output after any network layer of the joint network.

[0213] The probability estimation results can be output after any network layer of the joint network.

[0214] Figure 6 shows an example of the output of a joint network structure. The network structure includes four network layers. The decision information is output after the fourth network layer, and the probability estimation result is output after the second network layer.

[0215] Generative Networks 46

[0216] The generating network obtains judgment information of the feature elements of the picture feature map based on the input probability estimation result. The generating network is a multi-layer network, and the generating network may be a convolutional network, which includes a convolutional layer and a nonlinear activation layer. Any network layer of the generating network may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like.

[0217] The decision information can be output after any network layer of the generative network. The decision information can be one-dimensional, two-dimensional, or multi-dimensional data.

[0218] Figure 7 shows an example of outputting judgment information based on the network structure of a generative network. The network structure includes four network layers.

[0219] Decryption decision implementation 30

[0220] As shown in Figure 8, the decoding decision implementation includes element determination and entropy decoding. The picture feature map or audio feature variable is one-dimensional, two-dimensional, or multi-dimensional data output by the decoding decision implementation, and each of the data is a feature element.

[0221] Decoding element determination 301

[0222] The decoding element determination is to determine each feature element of the picture feature map or audio feature variable based on the probability estimation result of the probability estimation, and to determine a specific feature element on which entropy decoding is performed based on the determination result. The decoding element determination is to determine each feature element of the picture feature map or audio feature variable based on the determination result, and to determine a specific feature element on which entropy decoding is performed based on the determination result. It can be considered as the reverse process of the coding element determination, which is to determine each feature element of the picture feature map based on the determination result, and to determine a specific feature element on which entropy coding is performed based on the determination result.

[0223] Entropy Decoding 302

[0224] The entropy decoding may use various disclosed entropy decoding algorithms to perform entropy decoding, such as a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an entropy decoding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. Encoded picture (or audio) data 25 is obtained, which may be output in the form of an encoded bitstream 25 or the like via output 212, thereby allowing decoder 30A or the like to receive and use parameters for decoding. The encoded bitstream 25 may be transmitted to decoder 30A or stored in memory for subsequent transmission or retrieval by decoder 30A.

[0225] In another possibility, the entropy decoding may be performed by using an entropy decoding network, for example implemented by using a convolutional network.

[0226] Decoder Network 34

[0227] The decoder network is used to pass the decoded picture feature map or audio feature variable 31 or the post-processed decoded picture feature map or audio feature variable 33 through a decoder network 34 to obtain reconstructed picture (or audio) data 35 or machine-oriented task data in the pixel domain.

[0228] The decoder network includes multiple network layers. Any of the network layers may be a convolutional layer, a normalization layer, a nonlinear activation layer, or the like. Operations such as concat, addition, and subtraction may be present in the decoder network unit 306.

[0229] Possibly, the structure of the network layers of the decoder networks may be the same or different from each other.

[0230] An example of the structure of the decoder network is shown in Figure 9. In this example, it can be seen that the decoder network includes five network layers, specifically, one normalization layer, two convolutional layers, and two nonlinear activation layers.

[0231] The decoder network outputs a reconstructed picture (or audio) or the acquired machine-oriented task data. Specifically, the decoder network may include a target recognition network, a classification network, or a semantic segmentation network.

[0232] It should be understood that in the encoder 20A and the decoder 30A, the processing result of the current step may be further processed and then output to the next step. For example, after the encoder unit or the decoder unit, further operations or processes, such as a clip or shift operation or a filtering operation, may be performed on the processing result of the encoder unit or the decoder unit.

[0233] Based on the foregoing description, the following provides several picture feature map or audio feature variable encoding and decoding methods according to embodiments of this application. For ease of description, the method embodiments described below are expressed as a combination of a series of operation steps. However, it should be understood by those skilled in the art that the specific implementation of the technical solution of this application is not limited to the order of the series of operation steps described.

[0234] The following describes the procedure of this application in detail with reference to the accompanying drawings. It should be noted that the encoder-side process in the flowchart may be specifically executed by the encoder 20A, and the decoder-side process in the flowchart may be specifically executed by the decoder 30A.

[0235] In the first to fifth embodiments, the first feature element or the second feature element is a feature element to be currently encoded or a feature element to be currently decoded, for example:

number

[0236] In embodiment 1 of this application, Figure 10A shows a specific implementation process 1400. The execution steps are as follows:

[0237] Encoder side: Step 1401: Obtain a picture feature map.

[0238] This step is specifically implemented by the encoder network 204 of FIG. 3A. For details, please refer to the above description of the encoder network 20. A picture is input to a feature extraction module to output a picture feature map y, where the feature map y may be three-dimensional data whose dimensions are w, x, h, x, and c. Specifically, the feature extraction module may be implemented by using an existing neural network, which is not limited here. This step is an existing technology.

[0239] The feature quantization module quantizes each feature value of the feature map y, rounds the floating-point feature values, and obtains integer feature values ​​through rounding to form the quantized feature map

number

[0240] Step 1402: Feature Map

number

number

number

[0241] The parameters x, y, and i are positive integers, and the coordinate (x, y, i) indicates the position of the current feature element to be coded. Specifically, the coordinate (x, y, i) indicates the position of the current feature element to be coded in the current 3D feature map relative to the feature element at the top left vertex. This step is specifically performed by the probability estimation 210 in FIG. 3A. For details, please refer to the above description of the probability estimation 40. Specifically, a probability distribution model can be used to obtain the probability distribution. For example, a Gaussian single model (GSM) or a Gaussian mixture model (GMM) can be used for modeling. First, side information

number

number

number

number

[0242] Step 1403: Feature map is used to obtain a compressed bitstream.

number

[0243] This step is specifically implemented by the encoding determination implementation 208 in FIG. 3A. For details, refer to the above description of the encoding determination implementation 26. The feature element that will be encoded currently

Number

Number

Number

Number

[0244] Step 1404: The encoder transmits or stores the compressed bitstream.

[0245] Decoder side: Step 1411: Obtain the bitstream of the decoded picture feature map.

[0246] Step 1412: Perform probability estimation based on the bitstream to obtain the probability estimation result of the feature element.

[0247] This step is specifically performed by probability estimation 302 in Figure 10B. For details, see the above description of probability estimation 40. The probability estimation is performed based on the feature map to be decoded.

number

number

number

number

[0248] The structural diagram of the probability estimation network used by the decoder side is the same as that of the encoder side probability estimation network in this embodiment.

[0249] Step 1413: The feature map to be decoded

number

[0250] This step is specifically implemented by the decoding decision implementation 304 in Fig. 10B. For details, please refer to the above description of the decoding decision implementation 30. The probability P that the value of the currently to-be-decoded feature is k, i.e., the probability estimation result P of the currently to-be-decoded feature, is obtained based on the probability distribution of the currently to-be-decoded feature. If the probability estimation result P does not satisfy the preset condition, i.e., P is greater than the first threshold T0, entropy decoding does not need to be performed on the currently to-be-decoded feature, and the value of the currently to-be-decoded feature is set to k. Otherwise, if the currently to-be-decoded feature satisfies the preset condition, i.e., P is less than or equal to the first threshold T0, entropy decoding is performed on the bitstream to obtain the value of the currently to-be-decoded feature.

[0251] By analyzing the bitstream and based on the first threshold T0, the index number can be obtained from the bitstream. The decoder side builds a threshold candidate list in the same manner as the encoder, and then obtains the corresponding threshold according to the correspondence between the threshold and the index number in the preset threshold candidate list. The index number is obtained from the bitstream, that is, the index number is obtained from the sequence header, picture header, slice header, or SEI.

[0252] Alternatively, the bitstream may be directly parsed and the threshold value is obtained from the bitstream, specifically, from the sequence header, picture header, slice header, or SEI.

[0253] Alternatively, a fixed threshold is set directly according to a threshold policy consistent with the decoding.

[0254] Step 1414: Decoded feature map

number

[0255] Case 1: Feature maps obtained through entropy decoding

number

[0256] Case 2: Feature maps obtained through entropy decoding

number

[0257] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0258] 11A shows a specific implementation process 1500 according to Embodiment 2 of this application. The execution steps are as follows:

[0259] It should be noted that in Methods 1 to 6 of this embodiment, the probability estimation result includes a first parameter and a second parameter. When the probability distribution is a Gaussian distribution, the first parameter is the mean value μ and the second parameter is the variance σ. When the probability distribution is a Laplace distribution, the first parameter is the location parameter μ and the second parameter is the scale parameter b.

[0260] Encoder side:

[0261] Step 1501: Obtain a picture feature map.

[0262] This step is specifically implemented by the encoder network 204 of FIG. 3B. For details, please refer to the above description of the encoder network 20. A picture is input to a feature extraction module to output a picture feature map y, which may be three-dimensional data whose dimensions are w, x, h, x, and c. Specifically, the feature extraction module may be implemented by using an existing neural network, which is not limited here. This step is an existing technology.

[0263] The feature quantization module quantizes each feature value in the feature map y, rounds the floating-point feature values ​​to obtain integer feature values, and then converts the quantized feature map

number

[0264] Step 1502: Picture feature map

number

number

[0265] This step is specifically performed by the side information extraction unit 214 of Fig. 3B. The side information extraction module may be implemented by using the network depicted in Fig. 12. Side Information

number

number

number

number

number

[0266] Entropy coding is a method to reduce side information

number

number

number

number

[0267] Step 1503: Feature Map

number

[0268] This step is specifically performed by the probability estimation 210 in FIG. 3B. For details, see the above description of the probability estimation 40. A probability distribution model can be used to obtain the probability estimation results and the probability distribution. The probability distribution model can be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mix model (GMM), or a Laplace distribution model.

[0269] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixed Gaussian model), first, side information

number

number

number

[0270] When the probability distribution model is a Laplace distribution model, first, the side information

number

number

number

[0271] Alternatively, side information

number

number

number

number

number

number

[0272] The probability estimation network may use a network based on deep learning, such as a recurrent neural network (RNN) and a convolutional neural network (CNN), which is not limited here.

[0273] Step 1504: Based on the probability estimation result, entropy coding is performed on the feature element to be coded.

number

number

[0274] This step is specifically performed by the encoding decision implementation 208 of Fig. 3B. For details, please refer to the above description of the encoding decision implementation 26. One or more of the following methods may be used to determine whether entropy coding is performed on the current feature to be encoded based on the probability estimation results:

number

[0275] Method 1: When the probability distribution model is a Gaussian distribution, whether to perform entropy encoding for the feature element to be currently encoded is determined based on the probability estimation result of the first feature element. When the values of the mean parameter μ and variance σ of the Gaussian distribution of the feature element to be currently encoded do not satisfy the preset conditions, that is, the absolute value of the difference between the mean μ and k is less than the second threshold T1, and the variance σ is less than the third threshold T2, the entropy encoding process does not need to be performed for the feature element to be currently encoded

Number

Number

Number

[0276] ​Specifically, when the value of k is 0, it is the optimal value. When the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, the entropy encoding process for the feature element to be currently encoded

Number

Number

Number

[0277] Method 2: When the probability distribution is a Gaussian distribution, the mean parameter μ and the variance σ values of the Gaussian distribution of the feature element to be currently encoded

Number

Number

Number

Number

[0278] Method 3: When the probability distribution is a Laplace distribution, the values of the location parameter μ and the scale parameter b of the Laplace distribution of the current feature element to be encoded

Number

Number

Number

number

[0279] Method 4: When the probability distribution is Laplace distribution, the current feature to be coded

number

number

number

number

[0280] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the position parameter μ is smaller than T5 and the scale parameter b is smaller than T6, the current feature element to be coded is

number

number

number

[0281] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature to be coded

number

number

number

number

[0282] Method 6: Current feature to be encoded

number

[0283] In actual applications, it should be noted that in order to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 may be rounded, that is, shifted to an integer and scaled.

[0284] It should be noted that the method for obtaining the threshold may alternatively use one of the following methods. This is not limited here.

[0285] Method 1: Threshold T1 is used as an example, any value within the range of the value of T1 is used as the threshold T1, and the threshold T1 is written into the bitstream. Specifically, the threshold is written into the bitstream, stored in the sequence header, picture header, slice / slice header, or SEI, and can be transmitted to the decoder side. Alternatively, another method can be used. This is not limited here. A similar method can also be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0286] Method 2: The encoder uses a fixed threshold that matches the decoder. The fixed threshold does not need to be written into the bitstream or transmitted to the decoder. For example, threshold T1 is used as an example, and any value within the range of T1 is directly used as the value of T1. A similar method can be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0287] Method 3: A threshold candidate list is constructed, and the most likely value within the range of T1's value is entered into the threshold candidate list. Each threshold corresponds to a threshold index number, and the optimal threshold is determined and used as the value of T1. The index number of the optimal threshold is used as the threshold index number of T1, and the threshold index number of T1 is written into the bitstream. Specifically, the threshold may be written into the bitstream, stored in the sequence header, picture header, slice header, or SEI, and transmitted to the decoder side. Alternatively, another method may be used, which is not limited here. A similar method may also be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0288] Step 1505: The encoder transmits or stores the compressed bitstream.

[0289] Decoder side:

[0290] Step 1511: Obtain the bitstream of picture feature maps to be decoded.

[0291] Step 1512: Obtain the probability estimation results of the feature elements.

[0292] This step is specifically performed by the probability estimation unit 302 of Figure 11A. For details, see the above description of the probability estimation 40. Entropy decoding is performed to obtain the side information

number

number

number

number

number

number

[0293] Correspondingly, it should be noted that the probability estimation method used by the decoder side is the same as that used by the encoder side in this embodiment, and the structural diagram of the probability estimation network used by the decoder side is the same as that of the encoder side probability estimation network in this embodiment, and the details will not be described again here.

[0294] Step 1513: This step is specifically implemented by the decoding decision implementation 304 of Figure 11A. For details, please refer to the above description of the decoding decision implementation 30.

number

number

[0295] One or more of the following methods may be used to estimate the probability of entropy decoding of the feature to be decoded:

number

[0296] Method 1: When the probability distribution model is Gaussian, the current feature to be decoded

number

number

number

number

number

[0297] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, the feature element that will be currently decoded

Number

Number

Number

Number

[0298] Method 2: When the probability distribution is a Gaussian distribution, the feature element that will be currently decoded

Number

Number

Number

Number

Number

[0299] Method 3: When the probability distribution is a Laplace distribution, the values of the location parameter μ and the scale parameter b are obtained based on the probability estimation result. When the relationship between the location parameter μ, the scale parameter b, and k satisfies abs(μ - k)+σ < T4 (a preset condition is not satisfied), where T4 is the fourth threshold, the value of the feature element to be currently decoded is set to k, and the execution of the entropy decoding process for the feature element to be currently decoded is skipped. Otherwise, when the probability estimation result of the feature element to be currently decoded satisfies abs(μ - k)+σ≧T4 (a preset condition), entropy decoding is performed on the feature element to be currently decoded

Number

Number

number

number

number

number

[0301] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the position parameter μ is smaller than T5 and the scale parameter b is smaller than T6, the current feature element to be decoded is

number

number

number

number

[0302] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature to be decoded

number

number

number

number

number

[0303] Method 6: The probability P that the value of the currently to-be-decoded feature is k, i.e., the probability estimation result P of the currently to-be-decoded feature, is obtained based on the probability distribution of the currently to-be-decoded feature. If the probability estimation result P does not satisfy the preset condition, i.e., P is greater than the first threshold T0, then entropy decoding does not need to be performed on the currently to-be-decoded feature, and the value of the currently to-be-decoded feature is set to k. Otherwise, if the currently to-be-decoded feature satisfies the preset condition, i.e., P is less than or equal to the first threshold T0, then entropy decoding is performed on the bitstream to obtain the value of the currently to-be-decoded feature.

[0304] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0305] The method for obtaining the thresholds T0, T1, T2, T3, T4, T5, T6, and T7 corresponds to the encoder-side method, and one of the following methods may be used:

[0306] Method 1: The threshold is obtained from the bitstream. Specifically, the threshold is obtained from the sequence header, picture header, slice header, or SEI.

[0307] Method 2: The decoder uses a fixed threshold that matches the encoder.

[0308] Method 3: The threshold index number is obtained from the bitstream. Specifically, the threshold index number is obtained from the sequence header, picture header, slice header, or SEI. Then, the decoder side constructs a threshold candidate list in the same manner as the encoder, and obtains the corresponding threshold in the threshold candidate list based on the threshold index number.

[0309] It should be noted that in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 may be rounded, i.e., shifted and scaled to integers.

[0310] Step 1514 is the same as step 1414.

[0311] 13A shows a specific implementation process 1600 according to Embodiment 3 of this application. The execution steps are as follows:

[0312] Encoder side:

[0313] Step 1601 is the same as step 1501. This step is specifically performed by the encoder network 204 of Figure 3C. See the above description of the encoder network 20 for details.

[0314] Step 1602 is the same as step 1502. This step is specifically performed by side information extraction 214 in Figure 3C.

[0315] Step 1603: Feature Map

number

[0316] This step may be specifically performed by the probability estimation 210 of FIG. 3C. For details, see the above description of the probability estimation 40. A probability distribution model may be used to obtain the probability estimation result. The probability distribution model may be a single Gaussian model, an asymmetric Gaussian model, a mixture of Gaussian models, or a Laplace distribution model.

[0317] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixed Gaussian model), first, side information

number

number

number

[0318] When the probability distribution model is a Laplace distribution model, first, the side information

number

number

number

[0319] Furthermore, the probability estimation results are input into a probability distribution model that is used to obtain a probability distribution.

[0320] Alternatively, side information

number

number

number

number

number

[0321] The probability estimation network may use a network based on deep learning, such as a recurrent neural network and a convolutional neural network, which is not limited here.

[0322] Step 1604: Based on the probability estimation result, determine whether to perform entropy coding on the current feature to be coded. Based on the determination result, entropy coding is performed on the current feature to be coded, and the current feature to be coded is written into the coded bitstream, or entropy coding is not performed. Only when it is determined that entropy coding needs to be performed on the current feature to be coded, entropy coding is performed on the current feature to be coded.

[0323] This step is specifically performed by the generative network 216 and the encoding decision implementation 208 of Figure 3C. For more details, see the above description of the generative network 46 and the encoding decision implementation 26. The probability estimation results 211 are input to the decision module, and their dimensions are represented as feature maps.

number

number

number

number

number

number

number

number

[0324] In a possible implementation, the probability estimation result or probability distribution of the currently encoded feature is input to a decision module, which directly outputs decision information indicating whether entropy coding needs to be performed on the currently encoded feature. For example, when the decision information output by the decision module is a preset value, it indicates that entropy coding needs to be performed on the currently encoded feature. When the decision information output by the decision module is not a preset value, it indicates that entropy coding does not need to be performed on the currently encoded feature. The decision module can be implemented by using a network method. Specifically, the probability estimation result or probability distribution is input to a generation network shown in FIG. 7, and the network outputs decision information, i.e., a preset value.

[0325] Method 1: The decision information is a feature map whose dimensions are

number

number

number

number

number

number

number

[0326] Method 2: The decision information is a feature map whose dimensions are

number

number

number

number

[0327] Method 3: Alternatively, the decision information may be an identifier or an identifier value directly output by the joint network. When the decision information is a preset value, it indicates that entropy coding needs to be performed on the current feature to be coded. When the decision information output by the decision module is not a preset value, it indicates that entropy coding does not need to be performed on the current feature to be coded. For example, when the alternative values ​​of the identifier or identifier value are 0 and 1, the corresponding preset value is 0 or 1. When the identifier or identifier value can alternatively have multiple alternative values, the preset value is some specific value. For example, when the alternative values ​​of the identifier or identifier value are 0 to 255, the preset value is a proper subset of 0 to 255.

[0328] High probability indicates that the current feature to be coded

number

[0329] Step 1605: The encoder transmits or stores the compressed bitstream.

[0330] Steps 1601 to 1604 are for the feature map

number

[0331] Decoder side:

[0332] Step 1611: Obtain the compressed bitstream to be decoded.

[0333] Step 1612: The feature map to be decoded

number

[0334] This step may be specifically performed by probability estimation 302 of Figure 13B. For more details, see the above description of probability estimation 40. Side Information

number

[0335] Step 1613: Obtain the determination information, and determine whether to perform entropy decoding based on the determination information.

[0336] This step may be specifically implemented by the generating network 310 and the decoding decision implementation 304 in FIG. 13B. For details, please refer to the above description of the generating network 46 and the decoding decision implementation 30. The decision information 311 is obtained by using the same method as the encoder-side method in this embodiment. When the decision map map[x][y][i] is a preset value, it means that the entropy decoding is performed on the current feature element to be decoded at the corresponding position.

number

number

number

[0337] In a possible implementation, the probability estimation result or probability distribution of the currently to-be-decoded feature is input to a decision module, which directly outputs decision information indicating whether entropy decoding needs to be performed on the currently to-be-decoded feature. For example, when the decision information output by the decision module is a preset value, it indicates that entropy decoding needs to be performed on the currently to-be-decoded feature. When the decision information output by the decision module is not a preset value, it indicates that entropy decoding does not need to be performed on the currently to-be-decoded feature, and the value of the currently to-be-decoded feature is set to k. The decision module can be implemented by using a network method. Specifically, the probability estimation result or probability distribution is input to a generative network shown in FIG. 8, and the network outputs decision information, i.e., a preset value. The decision information indicates whether entropy decoding should be performed on the currently to-be-decoded feature, and the decision information can include a decision map.

[0338] Step 1614 is the same as step 1414.

[0339] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0340] 14 shows a specific implementation process 1700 according to Embodiment 4 of this application. The execution steps are as follows:

[0341] Encoder side:

[0342] Step 1701 is the same as step 1501. This step may be specifically performed by the encoder network 204 of Figure 3D. For details, see the above description of the encoder network 20.

[0343] Step 1702 is the same as step 1502. This step is specifically performed by side information extraction 214 in Figure 3D.

[0344] Step 1703: Feature Map

number

[0345] This step may be specifically performed by the joint network 218 of FIG. 3D. For details, see the above description of the joint network 34. Specifically, the side information

number

number

number

number

number

[0346] It should be noted that the specific structure of the joint network is not limited in this embodiment.

[0347] It should be noted that the decision information, probability distribution, and / or probability estimation results can all be output from different layers of the joint network. For example, in case (1), the middle layer of the network outputs the decision information, and the last layer outputs the probability distribution and / or probability estimation results. In case (2), the middle layer of the network outputs the probability distribution and / or probability estimation results, and the last layer outputs the decision information. In case (3), the last layer of the network outputs the decision information, probability distribution, and / or probability estimation results together.

[0348] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixed Gaussian model), first, side information

number

[0349] When the probability distribution model is a Laplace distribution model, first, the side information

number

[0350] Alternatively, side information

number

number

number

[0351] Step 1704: Based on the determination information, determine whether to perform entropy coding, and based on the determination result, perform entropy coding and write the current to-be-coded feature into a compressed bitstream (encoded bitstream), or skip performing entropy coding. Only when it is determined that entropy coding needs to be performed on the current to-be-coded feature, entropy coding is performed on the current to-be-coded feature. This step may be specifically implemented by the encoding decision implementation 208 in FIG. 3D. For details, please refer to the above description of the encoding decision implementation 26.

[0352] Method 1: The decision information is a feature map whose dimensions are

number

number

number

number

number

number

number

[0353] Method 2: The decision information is a feature map whose dimensions are

number

number

number

number

[0354] Method 3: Alternatively, the decision information may be an identifier or an identifier value directly output by the joint network. When the decision information is a preset value, it indicates that entropy coding needs to be performed on the current feature to be coded. When the decision information output by the decision module is not a preset value, it indicates that entropy coding does not need to be performed on the current feature to be coded. When there are only two alternative values ​​for the current feature to be coded in the decision map output by the joint network, the preset value is a specific value. For example, when the alternative values ​​for the current feature to be coded are 0 and 1, the preset value is 0 or 1. When there are multiple alternative values ​​for the current feature to be coded in the decision map output by the joint network, the preset value is some specific value. For example, when the alternative values ​​for the current feature to be coded are 0 to 255, the preset value is a proper subset of 0 to 255.

[0355] High probability indicates that the current feature to be coded

number

[0356] Step 1705: The encoder transmits or stores the compressed bitstream.

[0357] Decoder side:

[0358] Step 1711: Obtain a bitstream of picture feature maps to be decoded, and extract side information from the bitstream.

number

[0359] Step 1712: Feature Map

number

[0360] This step may be specifically performed by the joint network 312 of Figure 16. For more details, see the above description of the joint network 34. Feature Map

number

[0361] Step 1713: Determine whether to perform entropy decoding based on the determination information, and perform or skip entropy decoding based on the determination result. This step may be specifically implemented by the decoding determination implementation 304 in Figure 16. For details, please refer to the above description of the decoding determination implementation 30.

[0362] Method 1: The decision information is a decision map, and when the decision map map[x][y][i] is a preset value, it means that the entropy decoding is performed on the current feature element to be decoded at the corresponding position.

number

number

number

[0363] Method 2: The decision information is a feature map whose dimensions are

number

number

number

number

number

[0364] Method 3: Alternatively, the decision information may be an identifier or an identifier value directly output by the joint network. When the decision information is a preset value, it indicates that entropy decoding needs to be performed on the current feature to be decoded. When the decision information output by the decision module is not a preset value, it indicates that entropy decoding does not need to be performed on the current feature to be decoded, and the value of the current feature to be decoded is set to k. When there are only two alternative values ​​for the current feature to be decoded in the decision map output by the joint network, the preset value is a specific value. For example, when the alternative values ​​for the current feature to be decoded are 0 and 1, the preset value is 0 or 1. When there are multiple alternative values ​​for the current feature to be decoded in the decision map output by the joint network, the preset value is some specific value. For example, when the alternative values ​​for the current feature to be decoded are 0 to 255, the preset value is a proper subset of 0 to 255.

[0365] Step 1714 is the same as step 1414. This step may be specifically implemented by the decoder network unit 306 of the decoder 9C in the above embodiment. For details, please refer to the description of the decoder network unit 306 in the above embodiment.

[0366] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0367] 17 shows a specific implementation process 1800 according to Embodiment 5 of this application. The execution steps are as follows:

[0368] Step 1801: Obtain feature variables of the audio data to be coded.

[0369] The audio signal to be coded may be a time-domain audio signal. The audio signal to be coded may be a frequency-domain signal obtained after a time-frequency transform is performed on the time-domain signal. For example, the frequency-domain signal may be a frequency-domain signal obtained after an MDCT transform is performed on the time-domain audio signal, and the time-domain audio signal is a frequency-domain signal obtained through an FFT transform. Alternatively, the signal to be coded may be a signal obtained through QMF filtering. Alternatively, the signal to be coded may be a residual signal, for example, another coded residual signal or a residual signal obtained through LPC filtering.

[0370] Obtaining the feature variables of the audio data to be encoded may be extracting feature vectors based on the audio signal to be encoded, for example extracting mel-cepstral coefficients based on the audio signal to be encoded, quantizing the extracted feature vectors, and using the quantized feature vectors as feature variables of the audio data to be encoded.

[0371] Alternatively, obtaining feature variables of the audio data to be encoded may be implemented by using an existing neural network. For example, the audio signal to be encoded is processed by an encoding neural network to obtain latent variables, the latent variables output by the neural network are quantized, and the quantized latent variables are used as feature variables of the audio data to be encoded. The encoding neural network process is pre-trained, and the specific network structure and training method of the encoding neural network are not limited by the present invention. For example, a fully connected network or a CNN network may be selected for the encoding neural network. The number of layers included in the encoding neural network and the number of nodes in each layer are not limited by the present invention.

[0372] The format of the latent variables output by encoding neural networks with different structures can be different. For example, the encoding neural network is a fully connected network. The output latent variable is a vector, and the dimension M of the vector is the size of the latent variable, for example, y=[y(0), y(1), ..., y(M-1)]. The encoding neural network is a CNN network. The output latent variable is a matrix with N*M dimensions. N is the number of channels in the CNN network, and M is the size of the latent variable of each channel in the CNN network, for example,

number

[0373] Both the quantized feature vectors or the quantized latent variables are

number

[0374] Step 1802: Feature variables of the audio data to be coded

number

number

[0375] The side information extraction module may be implemented by using the network depicted in Figure 12.

number

number

number

number

number

[0376] Entropy coding is a method to reduce side information

number

number

number

number

[0377] Step 1803: Feature variables

number

[0378] A probability distribution model can be used to obtain the probability estimation results and the probability distribution, and the probability distribution model can be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model.

[0379] The following are the feature variables for explanation:

number

number

number

[0380] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixed Gaussian model), first, side information

number

number

number

[0381] Alternatively, the variance may be estimated. For example, when the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixture of Gaussian models), first, side information

number

number

number

[0382] When the probability distribution model is a Laplace distribution model, first, the side information

number

number

number

[0383] Alternatively, side information

number

number

number

number

number

number

[0384] The probability estimation network may use a network based on deep learning, such as a recurrent neural network (RNN) and a convolutional neural network (CNN), which is not limited here.

[0385] Step 1804: Based on the probability estimation result, determine whether entropy coding needs to be performed on the current feature to be coded, and based on the determination result, perform entropy coding and write the current feature to be coded into a compressed bitstream (coded bitstream), or skip performing entropy coding.

[0386] One or more of the following methods may be used to estimate the entropy coding of the feature to be coded based on the probability estimation result:

number

number

[0387] Entropy coding is the current feature to be coded.

number

number

[0388] Method 1: When the probability distribution model is a Gaussian distribution, whether to perform entropy encoding on the feature element to be currently encoded is determined based on the probability estimation result of the first feature element. When the values of the mean parameter μ and the variance σ of the Gaussian distribution of the feature element to be currently encoded satisfy the second condition, that is, the absolute value of the difference between the mean μ and k is smaller than the second threshold T1, and the variance σ is smaller than the third threshold T2, the entropy encoding process does not need to be performed on the feature element

Number

Number

Number

[0389]

Number

Number

Number

[0390] is written into the bit stream. The value of T2 is any number satisfying 0 < T2 < 1, for example, 0.2, 0.3, 0.4, or the like, and T1 is a number greater than or equal to 0 and less than 1, such as 0.01, 0.02, 0.001, and 0.002.

[0390] Method 2: When the probability distribution is a Gaussian distribution, the values of the mean parameter μ and variance σ of the Gaussian distribution of the feature element

Number

Number

Number

Number

[0391] When the probability distribution is a Gaussian distribution, the probability estimation is for each characteristic element of the feature variable

Number

Number

Number

Number

Number

Number

[0392] Method 3: When the probability distribution is a Laplace distribution, the feature element that will currently be encoded

Number

Number

Number

Number

[0393] Method 4: When the probability distribution is a Laplace distribution, the feature element to be currently encoded

Number

Number

number

number

[0394] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the position parameter μ is smaller than T5 and the scale parameter b is smaller than T6, the current feature element to be coded is

number

number

number

[0395] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature to be coded

number

number

number

number

[0396] Method 6: Current feature to be encoded

number

[0397] In actual applications, it should be noted that in order to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 may be rounded, that is, shifted to an integer and scaled.

[0398] It should be noted that one of the following methods may be used as an alternative to obtain the threshold. This is not limited here.

[0399] Method 1: Threshold T1 is used as an example, any value within the range of the value of T1 is used as the threshold T1, and the threshold T1 is written into the bitstream. Specifically, the threshold is written into the bitstream, stored in the sequence header, picture header, slice / slice header, or SEI, and can be transmitted to the decoder side. Alternatively, another method may be used. This is not limited here. Similar methods can also be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0400] Method 2: The encoder uses a fixed threshold that matches the decoder, and the fixed threshold does not need to be written into the bitstream or transmitted to the decoder. For example, threshold T1 is used as an example, and any value within the range of T1 is directly used as the value of T1. A similar method can be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0401] Method 3: A threshold candidate list is constructed, and the most likely value within the range of T1's value is entered into the threshold candidate list. Each threshold corresponds to a threshold index number, and the optimal threshold is determined and used as the value of T1. The index number of the optimal threshold is used as the threshold index number of T1, and the threshold index number of T1 is written into the bitstream. Specifically, the threshold may be written into the bitstream, stored in the sequence header, picture header, slice header, or SEI, and transmitted to the decoder side. Alternatively, another method may be used, which is not limited here. A similar method may also be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0402] Step 1805: The encoder transmits or stores the compressed bitstream.

[0403] Decoder side:

[0404] Step 1811: Obtain the bitstream of audio feature variables to be decoded.

[0405] Step 1812: Obtain the probability estimation results of the feature elements.

[0406] Entropy decoding is side information

number

number

number

number

number

number

number

number

number

number

number

[0407] It should be noted that the probability estimation method used by the decoder side is the same as that used by the encoder side in this embodiment, and the structural diagram of the probability estimation network used by the decoder side is the same as that of the encoder side probability estimation network in this embodiment, and the details will not be described again here.

[0408] 1813: Whether entropy decoding needs to be performed on the current feature element to be decoded is determined based on the probability estimation result, and the entropy decoding is performed or not based on the determination result, and the decoded feature variable is

number

[0409] One or more of the following methods may be used to estimate the probability of entropy decoding of the feature to be decoded:

number

number

[0410] Entropy decoding is the current feature to be decoded.

number

number

[0411] Method 1: When the probability distribution model is Gaussian, the current feature to be decoded

number

number

number

number

number

[0412] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the mean parameter μ of the Gaussian distribution is smaller than T1 and the variance σ of the Gaussian distribution is smaller than T2, the current feature element to be decoded is

number

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0414] Method 3: When the probability distribution is a Laplace distribution, the values of the location parameter μ and the scale parameter b are obtained based on the probability estimation result. When the relationship between the location parameter μ, the scale parameter b, and k satisfies abs(μ - k)+σ < T4 (the second condition), T4 is the fourth threshold value, and the feature element that will be decoded currently

Number

Number

number

number

[0415] Method 4: When the probability distribution is a Laplace distribution, the values ​​of the location parameter μ and the scale parameter b are obtained based on the probability estimation result. When the absolute value of the difference between the location parameter μ and k is smaller than the second threshold T5 and the scale parameter b is smaller than the third threshold T6 (second condition), the current feature to be decoded is

number

number

number

number

[0416] In particular, when the value of k is 0, it is the optimal value. When the absolute value of the position parameter μ is smaller than T5 and the scale parameter b is smaller than T6, the current feature element to be decoded is

number

number

number

number

[0417] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature to be decoded

number

number

number

number

number

[0418] Method 6: The probability P that the value of the current to-be-decoded feature is k, i.e., the probability estimation result P of the current to-be-decoded feature, is obtained based on the probability distribution of the current to-be-decoded feature. When the probability estimation result P satisfies the second condition, i.e., P is greater than the first threshold T0, entropy decoding does not need to be performed on the first feature, and the value of the current to-be-decoded feature is set to k. Otherwise, when the current to-be-decoded feature satisfies the first condition, i.e., P is less than or equal to the first threshold T0, entropy decoding is performed on the bitstream to obtain the value of the first feature.

[0419] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0420] The method for obtaining the thresholds T0, T1, T2, T3, T4, T5, T6, and T7 corresponds to the encoder-side method, and one of the following methods may be used:

[0421] Method 1: The threshold is obtained from the bitstream. Specifically, the threshold is obtained from the sequence header, picture header, slice header, or SEI.

[0422] Method 2: The decoder uses a fixed threshold that matches the encoder.

[0423] Method 3: The threshold index number is obtained from the bitstream. Specifically, the threshold index number is obtained from the sequence header, picture header, slice header, or SEI. Then, the decoder side constructs a threshold candidate list in the same manner as the encoder, and obtains the corresponding threshold in the threshold candidate list based on the threshold index number.

[0424] It should be noted that in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 may be rounded, i.e., shifted and scaled to integers.

[0425] Step 1814: Decoded feature variables

number

[0426] Case 1: Feature variables obtained through entropy decoding

number

[0427] Case 2: Feature variables obtained through entropy decoding

number

[0428] The aforementioned decoder-side value k is set to the corresponding encoder-side value k.

[0429] 18 is a schematic diagram of an exemplary structure of an encoding device according to the present application. As shown in FIG. 18, the device in this example may correspond to the encoder 20A. The device may include an acquisition module 2001 and an encoding module 2002. The acquisition module 2001 may include the encoder network 204, the rounding 206 (optional), the probability estimation 210, the side information extraction 214, the generative network 216 (optional), and the joint network 218 (optional) in the above-mentioned embodiment. The encoding module 2002 includes the encoding decision implementation 208 in the above-mentioned embodiment.

[0430] The obtaining module 2001 is configured to obtain feature data to be encoded, the feature data to be encoded including a plurality of feature elements, the plurality of feature elements including a first feature element, and to obtain a probability estimation result of the first feature element. The encoding module 2002 is configured to determine whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element, and to perform entropy encoding on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element.

[0431] In a possible implementation, determining whether to perform entropy coding on the first feature element of the feature data includes: when a probability estimation result of the first feature element of the feature data satisfies a preset condition, entropy coding needs to be performed on the first feature element of the feature data; and when a probability estimation result of the first feature element of the feature data does not satisfy the preset condition, entropy coding does not need to be performed on the first feature element of the feature data.

[0432] In a possible implementation, the encoding module is further configured to determine, based on a probability estimation result of the feature data, that the probability estimation result of the feature data be input to a generative network, and the network outputs decision information. When the value of the decision information of the first feature element is 1, the first feature element of the feature data needs to be coded. When the value of the decision information of the first feature element is not 1, the first feature element of the feature data does not need to be coded.

[0433] In a possible implementation, the preset condition is that the probability value of the first feature element having a value k is less than or equal to a first threshold, where k is an integer.

[0434] In a possible implementation, the preset condition is that the absolute value of the difference between the mean value of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a second threshold, or that the variance of the first feature element is greater than or equal to a third threshold, where k is an integer.

[0435] In another possible implementation, the preset condition is that the sum of the variance of the probability distribution of the first feature element and the absolute value of the difference between the mean value of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a fourth threshold, where k is an integer.

[0436] In a possible implementation, the probability value of the first feature element having the value k is the maximum probability value among the probability values ​​of all possible values ​​of the first feature element.

[0437] In a possible implementation, probability estimation is performed on the feature data to obtain probability estimation results for feature elements of the feature data, where the probability estimation result for a first feature element includes a probability value for the first feature element and / or first and second parameters of a probability distribution.

[0438] In a possible implementation, the probability estimation result of the feature data is input to a generative network to obtain decision information for the first feature element, and whether to perform entropy coding on the first feature element is determined based on the decision information for the first feature element.

[0439] In a possible implementation, when the decision information of the feature data is a decision map and a value corresponding to the position where the first feature element is located in the decision map is a preset value, it is determined that entropy coding needs to be performed on the first feature element, and when the value corresponding to the position where the first feature element is located in the decision map is not the preset value, it is determined that entropy coding does not need to be performed on the first feature element.

[0440] In a possible implementation, when the determination information of the feature data is a preset value, it is determined that entropy coding needs to be performed on the first feature element. When the determination information is not a preset value, it is determined that entropy coding does not need to be performed on the first feature element. In a possible implementation, the encoding module is further configured to construct a threshold candidate list for the first threshold and enter the first threshold into the threshold candidate list for the first threshold, where the first threshold has a corresponding index number, and is further configured to write the index number of the first threshold into the encoded bitstream, where the length of the threshold candidate list for the first threshold may be set to T, where T is an integer greater than or equal to 1.

[0441] The device in this embodiment can be used in the technical solution implemented by the encoder in the method embodiment shown in Figures 3A to 3D. The implementation principle and its technical effect are similar. The details will not be described again here.

[0442] 19 is a schematic diagram of an exemplary structure of a decoding device according to the present application. As shown in FIG. 19, the device in this example may correspond to the decoder 30. The device may include an acquisition module 2101 and a decoding module 2102. The acquisition module 2101 may include the probability estimation 302, the generative network 310 (optional), and the joint network 312 in the above-mentioned embodiment. The decoding module 2102 includes the decoding decision implementation 304 and the decoder network 306 in the above-mentioned embodiment.

[0443] The obtaining module 2101 is configured to obtain a bitstream of feature data to be decoded, where the feature data to be decoded includes a plurality of feature elements, where the plurality of feature elements includes a first feature element, and is configured to obtain a probability estimation result of the first feature element. The decoding module 2102 is configured to determine whether to perform entropy decoding on the first feature element based on the probability estimation result of the first feature element, and to perform entropy decoding on the first feature element only when it is determined that entropy decoding needs to be performed on the first feature element.

[0444] In a possible implementation, determining whether to perform entropy decoding on the first feature element of the feature data includes: when a probability estimation result of the first feature element of the feature data satisfies a preset condition, the first feature element of the feature data needs to be decoded; alternatively, when the probability estimation result of the first feature element of the feature data does not satisfy the preset condition, the first feature element of the feature data does not need to be decoded, and the feature value of the first feature element is set to k, where k is an integer.

[0445] In a possible implementation, the decoding module is further configured to determine, based on the probability estimation result of the feature data, that the probability estimation result of the feature data is input to the decision network module, and the network outputs decision information. A first feature element of the feature data is decoded when a value of a position in the decision information corresponding to the first feature element of the feature data is 1. The first feature element of the feature data is not decoded when a value of a position in the decision information corresponding to the first feature element of the feature data is not 1, and the feature value of the first feature element is set to k, where k is an integer.

[0446] In a possible implementation, the preset condition is that the probability value of the first feature element having a value k is less than or equal to a first threshold, where k is an integer.

[0447] In another possible implementation, the preset condition is that the absolute value of the difference between the mean value of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a second threshold, or that the variance of the probability distribution of the first feature element is greater than or equal to a third threshold.

[0448] In another possible implementation, the preset condition is that the sum of the variance of the probability distribution of the first feature element and the absolute value of the difference between the mean value of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a fourth threshold.

[0449] In a possible implementation, probability estimation is performed on the feature data to obtain probability estimation results for feature elements of the feature data, where the probability estimation result for a first feature element includes a probability value for the first feature element and / or first and second parameters of a probability distribution.

[0450] In a possible implementation, the probability value of the first feature element having the value k is the maximum probability value among the probability values ​​of all possible values ​​of the first feature element.

[0451] In a possible implementation, the probability estimation result of the Nth feature element includes at least one of the following: a probability value of the Nth feature element, a first parameter and a second parameter of the probability distribution, and decision information. The first feature element of the feature data is decoded when the value of a position in the decision information corresponding to the first feature element of the feature data is 1. The first feature element of the feature data is not decoded when the value of a position in the decision information corresponding to the first feature element of the feature data is not 1, and the feature value of the first feature element is set to k, where k is an integer.

[0452] In a possible implementation, the probability estimation result of the feature data is input to a generative network to obtain decision information of a first feature element. When the value of the decision information of the first feature element is a preset value, it is determined that entropy decoding needs to be performed on the first feature element. When the value of the decision information of the first feature element is not the preset value, it is determined that entropy decoding does not need to be performed on the first feature element, and the feature value of the first feature element is set to k, where k is an integer and k is one of multiple candidate values ​​of the first feature element.

[0453] In a possible implementation, the acquisition module is further configured to: construct a threshold candidate list of the first threshold; acquire an index number of the threshold candidate list of the first threshold by decoding the bitstream; and use a position value of the threshold candidate list of the first threshold corresponding to the index number of the first threshold as the value of the first threshold. The length of the threshold candidate list of the first threshold may be set to T, where T is an integer greater than or equal to 1.

[0454] The device in this embodiment can be used in the technical solution implemented by the decoder in the method embodiments shown in Figure 10B, Figure 13B, and Figure 16. The implementation principle and its technical effect are similar. The details will not be described again here.

[0455] Those skilled in the art will understand that the functions described with reference to the various illustrative logical blocks, modules, and algorithm steps disclosed and described herein may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the functions described with reference to the illustrative logical blocks, modules, and steps may be stored on or transmitted over a computer-readable medium as one or more instructions or code and determined by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media, such as data storage media, or any communication medium that facilitates the transmission of a computer program from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) a communication medium, such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0456] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, flash memory, or any other medium that can be accessed by a computer and that can store the required program code in the form of instructions or data structures. Additionally, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or another remote source over coaxial cable, fiber optic, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave, the coaxial cable, fiber optic, twisted pair, DSL, or wireless technologies such as infrared, radio, or microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but actually refer to non-transitory tangible storage media. As used in this specification, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), and Blu-ray discs. Disks typically reproduce data magnetically, and discs reproduce data optically by using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0457] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or equivalent integrated circuits or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to the foregoing structure or any other structure that may be applied to implement the techniques described herein. Additionally, in some aspects, the functionality described with reference to the illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. Additionally, the techniques may be implemented entirely in one or more circuits or logic elements.

[0458] The techniques in this application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices configured to determine the disclosed techniques, but do not necessarily require implementation by different hardware units. In practice, as described above, the various units may be combined into a codec hardware unit in combination with appropriate software and / or firmware, or may be provided by interoperable hardware units (including one or more processors described above).

[0459] The above description is merely an exemplary specific implementation of this application, but is not intended to limit the protection scope of this application. Any variations or substitutions that are easily understood by those skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims. [Explanation of symbols]

[0460] 12 Source Devices 14 Destination Device 17 Source Data 18 Preprocessors 19 Preprocessed Source Data 20 Encoder Network 21 Feature data to be encoded 22 Preprocessor 23 Preprocessed feature data to be coded 24 Entropy Decoding 25 Encoded Bitstream 26 Communication Interface 27 Communication Channels 28 Communication Interface 29 Decoded Bitstream 30 Entropy Decoding 31 Decoded feature data 32 Post Processors 33 Post-processed decoded feature data 34 Decoder Network 35 Reconstructed Data 36 Post Processors 37 Post-processed reconstructed data 38 Display Devices 40 Probability Estimation 41 Probability estimation results 42 Probability estimation results 44 Memory Storage 50 Coding Systems 51 Imaging Device 52 Antenna 53 processors 55 Display Devices 56 Processing Unit 202 Input 204 Encoder Network 206 Rounding 208 Encoding decision implementation 210 Probability Estimation 212 Output 214 Side Information Extraction 216 Generative Networks 218 Joint Network 261 Coding Element Determination 262 Entropy Coding 301 Decryption element determination 302 Entropy Decoding 306 Decoder Network 310 Generative Networks 400 Coding Devices 410 Inlet Port 420 receiver unit 430 processor 440 transmitter unit 450 outlet port 460 memory 470 Coding Module 502 processor 504 memory 506 Data 508 Operating Systems 510 Application Program 512 Bus 518 Display 1801 Capture Device 1802 Pre-import processing 1803 Video Coding 1804 Sent 1805 Received 1806 Bitstream Decoding 1807 Rendering and Display 1901 Feature Extraction 1902 Side Information Extraction 1903 Probability Estimation 1904 encoding 1905 Decryption 1906 Machine Vision (Auditory) Task 2001 Acquisition Module 2002 Encoding Module 2101 Acquisition Module 2102 Decryption Module

Claims

1. A feature data encoding method, comprising: obtaining feature data to be encoded, the feature data to be encoded comprising a plurality of feature elements, the plurality of feature elements comprising a first feature element; obtaining a probability estimation result for the first feature, the probability estimation result for the first feature including parameters of a probability distribution of the first feature; determining whether to perform entropy coding on the first feature based on the probability estimation result of the first feature; performing entropy coding on the first feature only when it is determined that entropy coding needs to be performed on the first feature; Including, determining whether to perform entropy coding on the first feature based on the probability estimation result of the first feature, When the probability estimation result of the first feature element does not satisfy a predetermined condition, determining that entropy decoding does not need to be performed on the first feature element of the feature data, and setting a feature value of the first feature element to k, where k is an integer and k is one of a plurality of candidate values ​​of the first feature element.

2. A feature data decoding method, comprising: obtaining a bitstream of feature data to be decoded, the feature data to be decoded comprising a plurality of feature elements, the plurality of feature elements comprising a first feature element; obtaining a probability estimation result for the first feature, the probability estimation result for the first feature including parameters of a probability distribution of the first feature; determining whether to perform entropy decoding on the first feature based on the probability estimation result of the first feature; performing entropy decoding on the first feature only when it is determined that entropy decoding needs to be performed on the first feature; Including, determining whether to perform entropy decoding on the first feature based on the probability estimation result of the first feature, When the probability estimation result of the first feature element does not satisfy a predetermined condition, determining that entropy decoding does not need to be performed on the first feature element of the feature data, and setting a feature value of the first feature element to k, where k is an integer and k is one of a plurality of candidate values ​​of the first feature element.

3. The method of claim 2, wherein the step of determining whether to perform entropy decoding on the first feature element based on the probability estimation result of the first feature element comprises:

3. The method of claim 2, further comprising determining that entropy decoding needs to be performed on the first feature element of the feature data when the probability estimation result of the first feature element satisfies the preset condition.

4. The method described in claim 2, wherein when the probability estimation result of the first feature element is a probability value that the feature value of the first feature element is k, the predetermined condition is that the probability value that the feature value of the first feature element is k is less than or equal to a first threshold, k is an integer, and k is one of the multiple candidate values ​​of the first feature element.

5. When the probability estimation result of the first feature element includes a first parameter and a second parameter of a probability distribution of the first feature element, the predetermined condition is an absolute value of a difference between the first parameter of the probability distribution of the first feature element and the feature value k of the first feature element is equal to or greater than a second threshold; the second parameter of the probability distribution of the first feature is greater than or equal to a third threshold; or the sum of the second parameter of the probability distribution of the first feature element and the absolute value of the difference between the first parameter of the probability distribution of the first feature element and the feature value k of the first feature element is equal to or greater than a fourth threshold value; 3. The method of claim 2, wherein k is an integer and k is one of the plurality of candidate values ​​of the first feature element.

6. When the probability distribution is a Gaussian distribution, the first parameter of the probability distribution of the first feature element is a mean value of the Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is a variance of the Gaussian distribution of the first feature element, or 6. The method of claim 5, wherein when the probability distribution is a Laplace distribution, the first parameter of the probability distribution of the first feature element is a location parameter of the Laplace distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is a scale parameter of the Laplace distribution of the first feature element.

7. When the probability estimation result of the first feature element is obtained through a Gaussian mixture distribution, the predetermined condition is: a sum of any variance of the Gaussian mixture distribution of the first feature element and a sum of absolute values ​​of differences between all mean values ​​of the Gaussian mixture distribution of the first feature element and the feature value k of the first feature element is equal to or greater than a fifth threshold; The difference between any mean value of the Gaussian mixture distribution of the first feature element and the feature value k of the first feature element is equal to or greater than a sixth threshold; or Any variance of the Gaussian mixture distribution of the first feature element is equal to or greater than a seventh threshold.

3. The method of claim 2, wherein k is an integer and k is one of the plurality of candidate values ​​of the first feature element.

8. When the probability estimation result of the first feature element is obtained through an asymmetric Gaussian distribution, the predetermined condition is: an absolute value of a difference between a mean value of the asymmetric Gaussian distribution of the first feature element and the feature value k of the first feature element is equal to or greater than an eighth threshold; a first variance of the asymmetric Gaussian distribution of the first feature is greater than or equal to a ninth threshold; or a second variance of the asymmetric Gaussian distribution of the first feature element is equal to or greater than a tenth threshold; 3. The method of claim 2, wherein k is an integer and k is one of the plurality of candidate values ​​of the first feature element.

9. The method of claim 8, wherein the step of determining whether to perform entropy decoding on the first feature based on the probability estimation result of the first feature comprises: inputting the probability estimation result of the feature data into a generative network to obtain determination information of the first feature element; determining whether to perform entropy decoding on the first feature based on the determination information of the first feature; 3. The method of claim 2, comprising:

10. An encoder comprising processing circuitry configured to perform the method of claim 1.

11. A decoder comprising a processing circuit configured to perform the method of any one of claims 2 to 9.

Images