Feature data encoding and decoding method and device

Abstract

The invention provides an image or audio encoding and decoding method and device, and relates to the technical field of image or audio encoding and decoding based on artificial intelligence (AI), in particular to the technical field of encoding and decoding of image feature maps or audio feature variables based on a neural network. The coding method comprises the steps that a to-be-coded target is acquired, the to-be-coded target comprises a plurality of feature elements, and the plurality of feature elements comprise a first feature element; the method further comprises the steps of obtaining a probability estimation result of the first feature element, and judging whether entropy coding is executed on the first feature element or not according to the probability estimation result of the first feature element; only when it is judged that entropy coding needs to be carried out on the first feature element, entropy coding is carried out on the first feature element. Whether the feature elements are coded or not is judged according to the probability estimation result. In this way, the coding and decoding complexity can be reduced under the condition that the coding and decoding performance is not affected, and the target to be coded comprises the image feature map or the audio feature variable.

Description

[0001] This application is a divisional application, the original application number is 202111091143.4, the original application date is September 17, 2021, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of image or audio compression based on artificial intelligence (AI), and in particular to a feature data encoding and decoding method and device. BACKGROUND

[0003] Image or audio encoding and decoding (referred to as coding) is widely used in digital image or audio applications, such as broadcast digital television, image or audio transmission on the Internet and mobile networks, real-time conversation applications such as video or voice chat, and video or voice conference, DVD and Blu-ray disc, image or audio content acquisition and editing system, and security applications of camcorders. Video is composed of multiple images, so the image in the present application can be a single image or an image in a video.

[0004] A large amount of video data needs to be described even in a short movie, which can cause difficulty when the data is to be transmitted in a network with limited bandwidth capacity or otherwise transferred. Thus, image (or audio) data is typically compressed before being transmitted in modern telecommunication networks. The size of an image (or audio) can also be an issue when the video is stored on a storage device, as memory resources can be limited. Image (or audio) compression devices typically use software and / or hardware at the source side to encode the image (or audio) data before transmission or storage, thereby reducing the amount of data needed to represent the digital image (or audio). The compressed data is then received by an image (or audio) decompression device at the destination side. With limited network resources and the growing demand for higher image (or audio) quality, it is necessary to improve the compression and decompression techniques that can improve the compression ratio without affecting the image (or audio) quality.

[0005] In recent years, applying deep learning to image (or audio) encoding and decoding has gradually become a trend. For example, Google has organized the CLIC (Challenge on Learned Image Compression) competition at the CVPR (IEEE Conference on Computer Vision and Pattern Recognition) for several consecutive years. CLIC focuses on using deep neural networks to improve image compression efficiency, and in 2020, an image challenge category was added to CLIC. Based on performance evaluations of the competition solutions, the overall compression efficiency of current deep learning-based image encoding and decoding schemes is comparable to the latest generation video image encoding and decoding standard VVC (Versatile Video Coding), and it has unique advantages in improving user-perceived quality.

[0006] The VVC video standard was finalized in June 2020, encompassing almost all technologies and algorithms that can significantly improve compression efficiency. Therefore, continuing research into new compression coding algorithms along the traditional signal processing path is unlikely to yield major technological breakthroughs in the short term. Unlike traditional image algorithms that manually design and optimize each module of image compression, end-to-end AI image compression optimizes the entire system, resulting in better compression performance. Variational Autoencoder (AE) is currently the mainstream AI image lossy compression technology. The current mainstream approach involves obtaining image feature maps from the image to be encoded through an encoding network, and then performing entropy coding on these feature maps. However, the entropy coding process suffers from excessive complexity. Summary of the Invention

[0007] This application provides a method and apparatus for encoding and decoding feature data, which can reduce encoding and decoding complexity without affecting encoding and decoding performance.

[0008] Firstly, a method for encoding feature data is provided, including: Acquire feature data to be encoded, the feature data to be encoded including multiple feature elements, the multiple feature elements including a first feature element; Obtain the probability estimation result of the first feature element; Based on the probability estimation result of the first feature element, determine whether to perform entropy encoding on the first feature element; Entropy encoding is performed on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element.

[0009] The feature data includes image feature maps, audio feature variables, or both. It can be one-dimensional, two-dimensional, or multi-dimensional data output by the coding network, where each data point is a feature element. It should be noted that in this application, feature points and feature elements have the same meaning.

[0010] Specifically, the first feature element is any one of the feature elements to be encoded in the feature data to be encoded.

[0011] In one possibility, the probability estimation process for obtaining the probability estimation result of the first feature element can be implemented by a probability estimation network; in another possibility, the probability estimation process can use a traditional non-network probability estimation method to perform probability estimation on the feature data.

[0012] It should be noted that when only edge information is used as input for probability estimation, the probability estimation results of each feature element can be output in parallel; when the input for probability estimation includes context information, the probability estimation results of each feature element need to be output serially. The edge information refers to the feature information further extracted from the feature data input to the neural network, and the number of feature elements contained in the edge information is less than the number of feature elements in the feature data. Optionally, the edge information of the feature data can be encoded into the bitstream.

[0013] In one possibility, when the first feature element of the feature data does not meet the preset conditions, it is not necessary to perform entropy encoding on the first feature element of the feature data.

[0014] Specifically, if the current first feature element is the Pth feature element of the feature data, after completing the judgment of the Pth feature element and performing or not performing entropy encoding based on the judgment result, the judgment of the (P+1)th feature element of the feature data and the performance of entropy encoding based on the judgment result begin, where P is a positive integer and P is less than M, and M is the number of feature elements in the entire feature data. For example, for the second feature element, if it is determined that entropy encoding is not required for the second feature element, then the entropy encoding process for the second feature element is skipped.

[0015] In the above technical solution, by determining whether entropy encoding needs to be performed on each feature element to be encoded, the entropy encoding process of some feature elements can be skipped, which can significantly reduce the number of elements that need to be entropy encoded. In this way, the complexity of entropy encoding can be reduced.

[0016] In one possible implementation, determining whether to perform entropy encoding on the first feature element includes: when the probability estimation result of the first feature element meets a preset condition, determining that entropy encoding needs to be performed on the first feature element; or when the probability estimation result of the first feature element does not meet the preset condition, determining that entropy encoding does not need to be performed on the first feature element.

[0017] In one possible implementation, the probability estimation result of the first feature element is the probability value of the first feature element taking the value k, and the preset condition is that the probability value of the first feature element taking the value k is less than or equal to a first threshold, where k is an integer.

[0018] k is one of the possible values ​​of the first feature element. For example, the first feature element can take values ​​in the range of [-255, 255]. k can be set to 0, in which case entropy coding is performed on the first feature element with a probability value less than or equal to 0.5. Entropy coding is not performed on the first feature element with a probability value greater than 0.5.

[0019] In one possible implementation, the probability value of the first feature element being k is the maximum probability value among all possible values ​​of the first feature element.

[0020] Specifically, the first threshold selected for the encoded bitstream at low bitrates is less than the first threshold selected for the encoded bitstream at high bitrates. The specific bitrate is related to the image resolution and image content. Taking the publicly available Kodak dataset as an example, a bitrate below 0.5 bpp is considered low, and vice versa.

[0021] For a given bitrate, the first threshold can be configured according to actual needs, and no limitation is made here.

[0022] In the above technical solution, the flexible setting of the first threshold allows for a flexible reduction in entropy coding complexity as required.

[0023] In one possible implementation, the probability estimation result of the first feature element includes a first parameter and a second parameter of the probability distribution of the first feature element.

[0024] Then, when the probability distribution is a Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean of the Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian distribution of the first feature element; or when the probability distribution is a Laplace distribution, the first parameter of the probability distribution of the first feature element is the location parameter of the Laplace distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the scale parameter of the Laplace distribution of the first feature element. The preset condition can be any of the following: The absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to the second threshold; or The second parameter of the probability distribution of the first feature element is greater than or equal to the third threshold; or The sum of the absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element and the second parameter of the probability distribution of the first feature element is greater than or equal to the fourth threshold.

[0025] When the probability distribution is a Gaussian mixture distribution, the first parameter of the probability distribution of the first feature element is the mean of the Gaussian mixture distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian mixture distribution of the first feature element. The preset condition can be any of the following: The sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution of the first feature element and the values ​​of the first feature element with the value k, and the sum of any variance of the Gaussian mixture distribution of the first feature element, is greater than or equal to the fifth threshold; or The difference between any mean of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is greater than or equal to the sixth threshold; or The variance of any of the Gaussian mixture distributions of the first feature element is greater than or equal to the seventh threshold.

[0026] When the probability distribution is an asymmetric Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean of the asymmetric Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the first variance and the second variance of the asymmetric Gaussian distribution of the first feature element. The preset condition can be any one of the following: The absolute value of the difference between the mean of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is greater than or equal to the eighth threshold; or The first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the ninth threshold. The second variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the tenth threshold.

[0027] The probability distribution of the first feature element is a Gaussian mixture distribution. The range of values ​​for the first feature element is determined when multiple means of the probability distribution of the first feature element are not within the range of values ​​for the first feature element.

[0028] The probability distribution of the first feature element is a Gaussian distribution. The range of values ​​for the first feature element is determined when the mean of the probability distribution of the first feature element is not within the range of values ​​for the first feature element.

[0029] The probability distribution of the first feature element is a Gaussian distribution. The judgment value range of the first feature element is determined. The judgment value range includes multiple possible values ​​of the first feature element. When the absolute value of the difference between the mean parameter of the Gaussian distribution of the first feature element and each value in the judgment value range of the first feature element is greater than or equal to the eleventh threshold, or the variance of the probability distribution of the first feature element is greater than or equal to the twelfth threshold.

[0030] The value of the first feature element is not within the range of values ​​that can be determined for the first feature element.

[0031] The probability value corresponding to the value of the first feature element is less than or equal to the thirteenth threshold.

[0032] In one possible implementation, the method further includes: constructing a threshold candidate list for a first threshold, placing the first threshold into the threshold candidate list and correspondingly assigning an index number to the first threshold, and writing the index number of the first threshold into the encoded bitstream, wherein the length of the threshold candidate list for the first threshold can be set to T; T is an integer greater than or equal to 1. It is understood that the other thresholds can be constructed using the same method as the threshold candidate list for the first threshold, and have corresponding index numbers and be written into the encoded bitstream.

[0033] Specifically, the index number is written into the bitstream. It can be stored in the sequence header, picture header, slice header, or SEI (supplementary enhancement information) and transmitted to the decoding end. Other methods can also be used, which are not limited here. The method of constructing the candidate list is not limited.

[0034] In another possibility, the decision information is obtained by inputting the probability estimation result into the generator network. The generator network can be a convolutional network, which may include multiple network layers, and any one of these layers can be a convolutional layer, a normalization layer, a nonlinear activation layer, etc.

[0035] In one possible implementation, the probability estimation result of the feature data is input into a generator network to obtain decision information for the first feature element, the decision information being used to indicate whether to perform entropy encoding on the first feature element.

[0036] In one possible implementation, the decision information of the feature data is a decision graph, also known as a decision map. Preferably, the decision graph is a binary graph, also known as a binary map. The decision information values ​​of the feature elements in the binary graph are typically 0 or 1. Therefore, when the value corresponding to the position of the first feature element in the decision graph is a preset value, entropy encoding needs to be performed on the first feature element; when the value corresponding to the position of the first feature element in the decision graph is not a preset value, entropy encoding does not need to be performed on the first feature element.

[0037] In one possible implementation, the decision information of the feature elements in the feature data is a preset value. The preset value of the decision information is typically 1. Therefore, when the decision information is a preset value, entropy encoding needs to be performed on the first feature element; when the decision information is not a preset value, entropy encoding is not required. The decision information can be an identifier or the value of the identifier. Whether to perform entropy encoding on the first feature element depends on whether the identifier or the value of the identifier is a preset value. If it is a preset value, entropy encoding needs to be performed on the first feature element; if it is not a preset value, entropy encoding is not required. The set of decision information for each feature element in the feature data can also be a floating-point number, meaning the value can be any value other than 0 and 1. In this case, by setting a preset value, when the value of the decision information of the first feature element is equal to or greater than the preset value, it is determined that entropy encoding needs to be performed on the first feature element; or when the value of the decision information of the first feature element is less than the preset value, it is determined that entropy encoding is not required.

[0038] In one possible implementation, the method further includes: obtaining the feature data by passing the image to be encoded through an encoding network; obtaining the feature data by rounding the image to be encoded after passing through the encoding network; or obtaining the feature data by quantization and rounding the image to be encoded after passing through the encoding network.

[0039] The encoding network can employ an autoencoder structure. The encoding network can be a convolutional neural network. The encoding network can include multiple sub-networks, each containing one or more convolutional layers. The network structures of the sub-networks can be the same or different from each other.

[0040] The image to be encoded can be either the original image or the residual image.

[0041] It should be understood that the image to be encoded can be in RGB format or YUV, RAW, or other representation formats. The image to be encoded can be preprocessed before being input into the encoding network. Preprocessing operations can include transformation, block partitioning, filtering, pruning, and other operations.

[0042] It should be understood that multiple images to be encoded or multiple blocks of images to be encoded are allowed to be input into the encoding / decoding network for processing to obtain feature data within the same timestamp or at the same moment.

[0043] Secondly, a method for decoding feature data is provided, including: Obtain the bitstream of the feature data to be decoded; The feature data to be decoded includes multiple feature elements, and the multiple feature elements include a first feature element; Obtain the probability estimation result of the first feature element; Based on the probability estimation result of the first feature element, determine whether to perform entropy decoding on the first feature element; Entropy decoding is performed on the first feature element only when it is determined that entropy decoding is required on the first feature element.

[0044] It can be understood that the first feature element is any feature element in the feature data to be decoded. When all feature elements in the feature data to be decoded have completed the judgment and entropy decoding is performed or not performed according to the judgment result, the decoded feature data is obtained.

[0045] The feature data to be decoded can be one-dimensional, two-dimensional, or multi-dimensional, where each data point is a feature element. It should be noted that in this application, feature points and feature elements have the same meaning.

[0046] Specifically, the first feature element is any one of the feature elements to be decoded in the feature data to be decoded.

[0047] In one possibility, the probability estimation process for obtaining the probability estimation result of the first feature element can be implemented by a probability estimation network; in another possibility, the probability estimation process can use a traditional non-network probability estimation method to perform probability estimation on the feature data.

[0048] It should be noted that when only edge information is used as input for probability estimation, the probability estimation results of each feature element can be output in parallel; when the input for probability estimation includes context information, the probability estimation results of each feature element need to be output serially. The edge information contains fewer feature elements than the feature data.

[0049] In one possibility, the bitstream contains side information, and the decoding process requires decoding the side information.

[0050] Specifically, the judgment process for each feature element in the feature data includes condition judgment and determining whether to perform entropy decoding based on the condition judgment result.

[0051] One possibility is that entropy decoding can be achieved through neural networks.

[0052] In another possibility, entropy decoding can be achieved through traditional entropy decoding methods.

[0053] Specifically, if the current first feature element is the Pth feature element of the feature data, after completing the judgment of the Pth feature element and performing or not performing entropy decoding based on the judgment result, the judgment of the (P+1)th feature element of the feature data and the performance of or not performing entropy decoding based on the judgment result begin, where P is a positive integer and P is less than M, and M is the number of feature elements in the entire feature data. For example, for the second feature element, if it is determined that entropy decoding is not required for the second feature element, then the entropy decoding for the second feature element is skipped.

[0054] In the above technical solution, by determining whether entropy decoding needs to be performed for each feature element to be decoded, the entropy decoding process of some feature elements can be skipped, which can significantly reduce the number of elements that need to be entropy decoded. In this way, the complexity of entropy decoding can be reduced.

[0055] In one possible implementation, determining whether to perform entropy decoding on the first feature element of the feature data includes: when the probability estimation result of the first feature element of the feature data meets a preset condition, determining that entropy decoding needs to be performed on the first feature element; or when the probability estimation result of the first feature element does not meet the preset condition, determining that entropy decoding does not need to be performed on the first feature element, and setting the feature value of the first feature element to k; where k is an integer.

[0056] In one possible implementation, the probability estimation result of the first feature element is the probability value of the first feature element taking the value k, and the preset condition is that the probability value of the first feature element taking the value k is less than or equal to a first threshold, where k is an integer.

[0057] In one possibility, the first feature element is set to k if the preset condition is not met. For example, the first feature element can take values ​​in the range of [-255, 255]. k can be set to 0, in which case entropy encoding is performed on the first feature element with a probability value less than or equal to 0.5. Entropy encoding is not performed on the first feature element with a probability value greater than 0.5.

[0058] In another possibility, the first feature element may be determined by a list if the preset conditions are not met.

[0059] In another possibility, the first feature element is set to a fixed integer value if the preset conditions are not met.

[0060] k is one of the possible values ​​of the first characteristic element mentioned above.

[0061] In one possibility, k is the value with the highest probability among all possible values ​​of the first feature element mentioned above.

[0062] Specifically, the first threshold selected for the decoded bitstream at low bitrates is less than the first threshold selected for the decoded bitstream at high bitrates. The specific bitrate is related to the image resolution and image content. Taking the publicly available Kodak dataset as an example, a bitrate below 0.5 bpp is considered low, and vice versa.

[0063] For a given bitrate, the first threshold can be configured according to actual needs, and no limitation is made here.

[0064] In the above technical solution, the flexible setting of the first threshold allows for a flexible reduction in entropy decoding complexity as required.

[0065] In one possible implementation, the probability estimation result of the first feature element includes a first parameter and a second parameter of the probability distribution of the first feature element.

[0066] Then, when the probability distribution is a Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean of the Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian distribution of the first feature element; or when the probability distribution is a Laplace distribution, the first parameter of the probability distribution of the first feature element is the location parameter of the Laplace distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the scale parameter of the Laplace distribution of the first feature element. The preset condition can be any of the following: The absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to the second threshold; or The second parameter of the first feature element is greater than or equal to the third threshold; or The sum of the absolute value of the difference between the first parameter of the probability distribution of the first feature element and the value k of the first feature element and the second parameter of the probability distribution of the first feature element is greater than or equal to the fourth threshold.

[0067] When the probability distribution is a Gaussian mixture distribution, the first parameter of the probability distribution of the first feature element is the mean of the Gaussian mixture distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the variance of the Gaussian mixture distribution of the first feature element. The preset condition can be any of the following: The sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution of the first feature element and the values ​​of the first feature element with the value k, and the sum of any variance of the Gaussian mixture distribution of the first feature element, is greater than or equal to the fifth threshold; or The difference between any mean of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is greater than the sixth threshold; or The variance of any of the Gaussian mixture distributions of the first feature element is greater than or equal to the seventh threshold.

[0068] When the probability distribution is an asymmetric Gaussian distribution, the first parameter of the probability distribution of the first feature element is the mean of the asymmetric Gaussian distribution of the first feature element, and the second parameter of the probability distribution of the first feature element is the first variance and the second variance of the asymmetric Gaussian distribution of the first feature element. The preset condition can be any one of the following: The absolute value of the difference between the mean parameter of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is greater than the eighth threshold; or The first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the ninth threshold. The second variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the tenth threshold.

[0069] The probability distribution of the first feature element is a Gaussian mixture distribution. The range of values ​​for the first feature element is determined when multiple means of the probability distribution of the first feature element are not within the range of values ​​for the first feature element.

[0070] The probability distribution of the first feature element is a Gaussian distribution. The range of values ​​for the first feature element is determined when the mean of the probability distribution of the first feature element is not within the range of values ​​for the first feature element.

[0071] The probability distribution of the first feature element is a Gaussian distribution. The judgment value range of the first feature element is determined. The judgment value range includes multiple possible values ​​of the first feature element. When the absolute value of the difference between the mean parameter of the Gaussian distribution of the first feature element and each value in the judgment value range of the first feature element is greater than or equal to the eleventh threshold, or the variance of the probability distribution of the first feature element is greater than or equal to the twelfth threshold.

[0072] The value k of the first feature element is not within the range of values ​​for the first feature element.

[0073] The probability value corresponding to the first feature element being k is less than or equal to the thirteenth threshold.

[0074] In one possible implementation, a threshold candidate list for a first threshold is constructed. The bitstream is decoded to obtain the index number of the threshold candidate list for the first threshold. The value at the position in the threshold candidate list corresponding to the index number of the first threshold is used as the value of the first threshold. The length of the threshold candidate list for the first threshold can be set to T, where T is an integer greater than or equal to 1. It is understood that any other arbitrary threshold can be constructed using the same method as the threshold candidate list for the first threshold, and the index number of the corresponding threshold can be decoded, and the value in the constructed list can be selected as the threshold based on the index number.

[0075] In another possibility, the decision information is obtained by inputting the probability estimation result into the generator network. The generator network can be a convolutional network, which may include multiple network layers, and any one of these layers can be a convolutional layer, a normalization layer, a nonlinear activation layer, etc.

[0076] In one possible implementation, the probability estimation result of the feature data is input into a generator network to obtain decision information for the first feature element, the decision information being used to indicate whether to perform entropy decoding on the first feature element.

[0077] In one possible implementation, the decision information of each feature element in the feature data is a decision graph, also known as a decision map. Preferably, the decision graph is a binary graph, also known as a binary map. The decision information values ​​of feature elements in a binary graph are typically 0 or 1. Therefore, when the value corresponding to the position of the first feature element in the decision graph is a preset value, entropy decoding needs to be performed on the first feature element; when the value corresponding to the position of the first feature element in the decision graph is not a preset value, entropy decoding is not required for the first feature element.

[0078] The set of decision information for each feature element in the feature data can also be a floating-point number, meaning the value can be any value other than 0 and 1. In this case, a preset value can be set. When the value of the decision information of the first feature element is equal to or greater than the preset value, it is determined that entropy decoding needs to be performed on the first feature element; or when the value of the decision information of the first feature element is less than the preset value, it is determined that entropy decoding does not need to be performed on the first feature element.

[0079] In one possible implementation, the feature data is passed through a decoding network to obtain a reconstructed image.

[0080] In another possible implementation, the feature data is processed by a decoding network to obtain machine-oriented task data. Specifically, the feature data is processed by a machine-oriented task module to obtain machine-oriented task data, and the machine-oriented module includes an object recognition network, a classification network, or a semantic segmentation network.

[0081] Thirdly, a feature data encoding device is provided, comprising: The module is used to acquire feature data to be encoded, the feature data to be encoded including multiple feature elements, the multiple feature elements including a first feature element, and to acquire the probability estimation result of the first feature element; The encoding module is used to determine whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element; and to perform entropy encoding on the first feature element only when it is determined that entropy encoding is required.

[0082] The further implementation functions of the above-mentioned acquisition module and encoding module can refer to the first aspect or any of the implementation methods of the first aspect, and will not be elaborated here.

[0083] Fourthly, a feature data decoding apparatus is provided, comprising: The acquisition module is used to acquire the bitstream of the feature data to be decoded, the feature data to be decoded including multiple feature elements, the multiple feature elements including a first feature element; and to acquire the probability estimation result of the first feature element. The decoding module is used to determine whether to perform entropy decoding on the first feature element based on the probability estimation result of the first feature element; and to perform entropy decoding on the first feature element only when it is determined that entropy decoding is required.

[0084] The further implementation functions of the above-mentioned acquisition module and decoding module can be referred to the second aspect or any of the implementation methods of the second aspect, and will not be elaborated here.

[0085] Fifthly, this application provides an encoder, including a processing circuit, for determining the method described in accordance with the first aspect and any one of the first aspects.

[0086] In a sixth aspect, this application provides a decoder, including processing circuitry, for determining the method described in the second aspect and any one of the second aspects.

[0087] In a seventh aspect, this application provides a computer program product, including program code, which, when determined on a computer or processor, is used to determine the methods described in the first aspect and any one of the first aspects, and the second aspect and any one of the second aspects.

[0088] Eighthly, this application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program determined by the processor, wherein the program, when determined by the processor, causes the decoder to determine the method described in the first aspect and any one of the first aspects.

[0089] Ninthly, this application provides a decoder, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program determined by the processor, wherein the program, when determined by the processor, causes the encoder to determine the method described in the second aspect and any one of the second aspects.

[0090] In a tenth aspect, this application provides a non-transitory computer-readable storage medium, including program code, which, when determined by a computer device, is used to determine the methods described in the first aspect and any one of the first aspects, and the second aspect and any one of the second aspects.

[0091] Eleventhly, the present invention relates to an encoding apparatus having the function of implementing the behavior in the method embodiments of the first aspect or any of the first aspects described above. The function can be implemented in hardware or by hardware determining corresponding software implementation. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the encoding apparatus includes: an acquisition module, used to transform the original image or residual image to a feature space through an encoding network, and extract feature data for compression. Additionally, probability estimation is performed on the feature data to obtain probability estimation results for each feature element of the feature data; an encoding module, using the probability estimation results for each feature element of the feature data, used to determine, under certain conditions, whether entropy encoding is performed on each feature element in the feature data and to complete the encoding process of all feature elements in the feature data to obtain the encoded bitstream of the feature data. These modules can determine the corresponding functions in the method examples of the first aspect or any of the first aspects described above, as detailed in the method examples, and will not be repeated here.

[0092] In a twelfth aspect, the present invention relates to a decoding apparatus having the function of implementing the behavior in the method embodiments of the second aspect or any of the second aspects described above. The function can be implemented in hardware or by hardware determining corresponding software implementation. The hardware or software includes one or more modules corresponding to the above-described functions. In one possible design, the decoding apparatus includes: an acquisition module, configured to acquire the bitstream of feature data to be decoded, and perform probability estimation based on the bitstream of the feature data to obtain probability estimation results for each feature element of the feature data; and a decoding module, which, using the probability estimation results for each feature element of the feature data, determines, under certain conditions, whether entropy decoding is performed on each feature element in the feature data and completes the decoding process of all feature elements in the feature data to obtain the feature data, and decodes the feature data to obtain a reconstructed image or machine-oriented task data. These modules can determine the corresponding functions in the method examples of the second aspect or any of the second aspects described above, as detailed in the method examples, and will not be repeated here.

[0093] In the thirteenth aspect, a method for encoding feature data is provided, including: Acquire feature data to be encoded, the feature data including multiple feature elements, the multiple feature elements including a first feature element; Obtain the side information of the feature data, and input the side information of the feature data into the joint network to obtain the decision information of the first feature element; Based on the decision information of the first feature element, determine whether to perform entropy encoding on the first feature element; Entropy encoding is performed on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element.

[0094] The feature data is one-dimensional, two-dimensional, or multi-dimensional data output by the coding network, where each data element is a feature element.

[0095] In one possibility, the side information of the feature data is encoded into the bitstream. This side information is feature information obtained by further extraction from the feature data input into a neural network, and the number of feature elements contained in the side information is less than the number of feature elements in the feature data.

[0096] The first feature element is any feature element in the feature data.

[0097] In one possibility, the set of decision information for each feature element of the feature data can be represented by a decision graph or similar method. The decision graph is one-dimensional, two-dimensional, or multi-dimensional image data and has the same size as the feature data.

[0098] In one possibility, the joint network also outputs a probability estimate of the first feature element, which includes the probability value of the first feature element and / or a first parameter and a second parameter of the probability distribution.

[0099] In the above technical solution, by determining whether entropy encoding needs to be performed on each feature element to be encoded, the entropy encoding process of some feature elements can be skipped, which can significantly reduce the number of elements that need to be entropy encoded. In this way, the complexity of entropy encoding can be reduced.

[0100] In one possibility, when the value at the position corresponding to the first feature element in the decision graph is a preset value, entropy encoding needs to be performed on the first feature element; when the value at the position corresponding to the first feature element in the decision graph is not a preset value, entropy encoding does not need to be performed on the first feature element.

[0101] In the fourteenth aspect, a method for decoding feature data is provided, including: Obtain the bitstream of the feature data to be decoded and the side information of the feature data to be decoded; The feature data to be decoded includes multiple feature elements, and the multiple feature elements include a first feature element; The side information of the feature data to be decoded is input into the joint network to obtain the decision information of the first feature element; Based on the decision information of the first feature element, determine whether to perform entropy decoding on the first feature element; Entropy decoding is performed on the first feature element only when it is determined that entropy decoding is required on the first feature element.

[0102] In one possibility, the bitstream of the feature data to be decoded is decoded to obtain side information. The side information contains fewer feature elements than the feature data.

[0103] The first feature element is any feature element in the feature data.

[0104] In one possibility, the decision information of each feature element of the feature data can be represented by a decision graph or similar method. The decision graph is one-dimensional, two-dimensional, or multi-dimensional image data and has the same size as the feature data.

[0105] In one possibility, the joint network also outputs a probability estimate of the first feature element, which includes the probability value of the first feature element and / or a first parameter and a second parameter of the probability distribution.

[0106] In one possibility, when the value at the position corresponding to the first feature element in the decision graph is a preset value, entropy decoding needs to be performed on the first feature element; when the value at the position corresponding to the first feature element in the decision graph is not a preset value, entropy decoding does not need to be performed on the first feature element, and the feature value of the first feature element is set to k, where k is an integer.

[0107] In the above technical solution, by determining whether entropy decoding needs to be performed on each feature element to be encoded, the entropy decoding process of some feature elements can be skipped, which can significantly reduce the number of elements that need to be entropy decoded. In this way, the complexity of entropy decoding can be reduced.

[0108] In existing mainstream end-to-end feature data encoding and decoding schemes, the entropy encoding / decoding or arithmetic encoding / decoding processes suffer from excessive complexity. This application utilizes information related to the probability distribution of feature points in the feature data to be encoded to determine whether each feature element in the data requires entropy encoding / decoding, thereby skipping the entropy encoding / decoding process for certain feature elements. This significantly reduces the number of elements requiring encoding / decoding and lowers the encoding / decoding complexity. Furthermore, the threshold value can be flexibly set to control the bitrate of the generated bitstream based on the actual bitrate requirements of the bitstream.

[0109] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0110] The accompanying drawings used in the embodiments of this application are described below.

[0111] FIG. 1A This is an exemplary block diagram of an image decoding system; FIG. 1B This is implemented for the processing circuit of the image decoding system. FIG. 1C A schematic block diagram of an image decoding device; FIG. 1D This is a diagram illustrating the device implementation of an embodiment of this application; FIG. 2A This is a system architecture diagram for one possible scenario of this application; FIG. 2B This is a system architecture diagram for one possible scenario of this application; FIG. 3A-3D This is a schematic block diagram of an encoder; FIG. 4A This is a schematic diagram of a coding network unit; FIG. 4B This is a schematic diagram of the network structure of an encoding network; FIG. 5 A schematic diagram of the unit structure for implementing coding decisions; FIG. 6 Output an example diagram for the joint network; FIG. 7 To generate an example diagram of the network output; FIG. 8 This is a schematic diagram illustrating the implementation of the decoding decision. FIG. 9 Example diagram of the network structure for decoding networks; FIG. 10A This is an example diagram of the decoding method according to an embodiment of this application; FIG. 10B This is a schematic block diagram of an image feature map decoder according to an embodiment of this application; FIG. 11A This is an example diagram of the decoding method according to an embodiment of this application; FIG. 12 Example diagram of the network structure for the edge information extraction module; FIG. 13A This is an example diagram of the decoding method according to an embodiment of this application; FIG. 13B This is a schematic block diagram of an image feature map decoder according to an embodiment of this application; FIG. 14 This is an example diagram of the decoding method according to an embodiment of this application; FIG. 15 Example diagram of the network structure of a federated network; FIG. 16 This is a schematic block diagram of an image feature map decoder according to an embodiment of this application; FIG. 17 This is an example diagram of the decoding method according to an embodiment of this application; FIG. 18 This is an exemplary structural diagram of the encoding device of this application; FIG. 19 This is an exemplary structural diagram of the decoding device of this application. Detailed Implementation

[0112] The terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0113] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0114] This application provides an AI-based feature data encoding and decoding technology, particularly an image feature map and / or audio feature variable encoding and decoding technology based on neural networks, specifically an end-to-end image feature map and / or audio feature variable encoding and decoding system.

[0115] In the field of image coding, the terms "picture" or "image" are used synonymously. Image coding (or often simply called coding) comprises two parts: image encoding and image decoding. Video is composed of multiple images and is a representation of consecutive images. Image encoding is determined at the source side and typically involves processing (e.g., compressing) the raw video images to reduce the amount of data required to represent them (thus enabling more efficient storage and / or transmission). Image decoding is determined at the destination side and typically involves inverse processing relative to the encoder to reconstruct the image. The "decoding" of images or audio in the embodiments should be understood as the "encoding" or "decoding" of images or audio. The encoding and decoding parts are also collectively referred to as codec (encoding and decoding, CODEC).

[0116] In lossless image coding, the original image can be reconstructed, meaning the reconstructed image has the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In traditional lossy image coding, further compression is performed through quantization and other methods to reduce the amount of data required to represent the video image. However, the decoder cannot completely reconstruct the video image, meaning the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0117] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0118] (1) Neural Network A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be: (1-1) Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0119] (2) Deep Neural Networks A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0120] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: ,in, It is the input vector. It is the output vector. It is an offset vector. It is the weight matrix (also called coefficients). It's an activation function. Each layer simply applies the input vector... The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients... and offset vector The number of parameters is also quite large. These parameters are defined in DNNs as follows: [as coefficients] For example: Suppose in a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 represents the coefficient. The index corresponds to the third-level index 2 in the output and the second-level index 4 in the input.

[0121] In summary, the Lth The coefficients from the k-th neuron in layer 1 to the j-th neuron in layer L are defined as follows: .

[0122] It should be noted that the input layer does not have... In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers in the trained deep neural network (composed of vectors from many layers). The resulting weight matrix.

[0123] (3) Convolutional Neural Network A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0124] (4) Entropy coding Entropy coding is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization coefficients and other syntax elements, resulting in encoded data that can be output as an encoded bitstream, allowing decoders to receive and use the parameters for decoding. The encoded bitstream can be transmitted to the decoder or stored in memory for later transmission or retrieval by the decoder.

[0125] In the following embodiment of the decoding system 10, the encoder 20A and the decoder 30A are based on FIGS. 1A-15 Describe it.

[0126] FIG. 1A This is a schematic block diagram of an exemplary decoding system 10, such as an image (or audio) decoding system 10 (or simply decoding system 10) that can utilize the technology described in this application. The encoder 20A and decoder 30A in the image decoding system 10 represent devices, etc., that can be used to determine the various technologies according to the various examples described in this application.

[0127] like FIG. 1A As shown, the decoding system 10 includes a source device 12, which provides an encoded bitstream 21, such as an encoded image (or audio), to a destination device 14 for decoding the encoded bitstream 21.

[0128] The source device 12 includes an encoder 20A, and optionally, an image source 16, a preprocessor (or preprocessing unit) 18, a communication interface (or communication unit) 26, and a probability estimator (or probability estimating unit) 40.

[0129] The image (or audio) source 16 may include or may be any type of image capture device for capturing real-world images (or audio), and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The audio or image source may be any type of memory or storage device for storing any of the aforementioned audio or images.

[0130] To distinguish the processing determined by the preprocessor (or preprocessing unit) 18, the image or audio (image or audio data) 17 may also be referred to as the raw image or audio (raw image data or audio data) 17.

[0131] The preprocessor 18 receives (raw) image (or audio) data 17 and preprocesses the image (or audio) data 17 to obtain a preprocessed image or audio (or preprocessed image or audio data) 19. For example, the preprocessing determined by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0132] The encoder 20A includes an encoding network 20, an entropy code 24, and optionally a preprocessor 22.

[0133] Image (or audio) encoding network (or encoding network) 20 is used to receive preprocessed image (or audio) data 19 and provide encoded image (or audio) data 21.

[0134] The preprocessor 22 receives the feature data 21 to be encoded and preprocesses it to obtain preprocessed feature data 23 to be encoded. For example, the preprocessing determined by the preprocessor 22 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or noise reduction. It is understood that the preprocessing unit 22 may be an optional component.

[0135] Entropy coding 24 is used to receive the feature data to be encoded (or preprocess the feature data to be encoded) 23 and generate the encoded bitstream 25 based on the probability estimation result 41 provided by probability estimation 40.

[0136] The communication interface 26 in the source device 12 can be used to: receive the encoded bitstream 25 and send the encoded bitstream 25 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 27 for storage or direct reconstruction.

[0137] The target device 14 includes a decoder 30A, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 36 and a display device 38.

[0138] The communication interface 28 in the destination device 14 is used to receive the encoded bitstream 25 (or other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded bitstream storage device, and to provide the encoded bitstream 25 to the decoder 30A.

[0139] Communication interfaces 26 and 28 can be used to send or receive encoded streams (or encoded stream data) 25 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0140] For example, the communication interface 26 can be used to encapsulate the encoded stream 25 into a suitable format such as a message, and / or process the encoded stream using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0141] Communication interface 28 corresponds to communication interface 26. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded bitstream 25.

[0142] Both communication interface 26 and communication interface 28 can be configured as follows: FIG. 1A The arrow pointing from the source device 12 to the corresponding communication channel 27 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0143] Decoder 30A includes decoding network 34, entropy decoding 30, and optionally, post-processor 32.

[0144] Entropy decoding 30 is used to receive the encoded bitstream 25 and provide decoding feature data 31 based on the probability estimation result 42 provided by probability estimation 40.

[0145] The post-processor 32 is used to post-process the decoded feature data 31 to obtain post-processed decoded feature data 33. The post-processing determined by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming, or resampling. It is understood that the post-processing unit 32 may be an optional component.

[0146] The decoding network 34 is used to receive the decoded feature data 31 or the post-processed decoded feature data 33 and provide the reconstructed image data 35.

[0147] The post-processor 36 is used to post-process the reconstructed image data 35 to obtain post-processed reconstructed image data 37. The post-processing determined by the post-processing unit 36 ​​may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming, or resampling. It is understood that the post-processing unit 36 ​​may be an optional component.

[0148] Display device 38 is used to receive reconstructed image data 35 or post-processed reconstructed image data 37 to display the image to a user or viewer. Display device 38 can be or includes any type of player or display for representing reconstructed audio or images, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0149] although FIG. 1A The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0150] According to the description, FIG. 1A The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0151] Feature data encoder 20A (e.g., image feature map encoder or audio feature variable encoder) or feature data decoder 30A (e.g., image feature map decoder or audio feature variable decoder), or both, can be transmitted via, for example, FIG. 1BThe processing circuitry shown can be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, image encoding dedicated processors, or any combination thereof. Feature data encoder 20A and feature data decoder 30A can be implemented by processing circuitry 56. Processing circuitry 56 can be used to determine various operations discussed below. If some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and use one or more processors to determine the instructions in hardware, thereby determining the technology of the invention. One of the feature data encoder 20A and feature data decoder 30A can be integrated into a single device as part of a combined encoder / decoder (CODEC), such as... FIG. 1B As shown.

[0152] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or notebook computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiving device, broadcast transmitting device, etc., and may or may not use an operating system of any type. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0153] In some cases, FIG. 1A The decoding system 10 shown is merely exemplary. The technology provided in this application can be applied to image feature map or audio feature variable encoding settings (e.g., image feature map encoding or image feature map decoding), which do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. The image feature map or audio feature variable encoding device can encode data and store it in memory, and / or the image feature map or audio feature variable decoding device can retrieve data from memory and decode it. In some examples, encoding and decoding are determined by devices that do not communicate with each other but only encode data to memory and / or retrieve and decode data from memory.

[0154] FIG. 1B It includes, according to an exemplary embodiment FIG. 1A Feature data encoder 20A and / or FIG. 1B An illustrative diagram illustrating an example of a decoding system 50 for a feature data decoder 30A. The decoding system 50 may include an imaging (or audio generation) device 51, an encoder 20A, a decoder 30A (and / or a feature data encoder / decoder implemented by processing circuitry 56), an antenna 52, one or more processors 53, one or more memory storage devices 54, and / or a display (or audio playback) device 55.

[0155] like FIG. 1B As shown, the imaging (or audio generation) device 51, antenna 52, processing circuitry 56, encoder 20A, decoder 30A, processor 53, memory storage 54, and / or display (or audio playback) device 55 are capable of communicating with each other. In different instances, the decoding system 50 may contain only encoder 20A or only decoder 30A.

[0156] In some instances, antenna 52 can be used to transmit or receive encoded bitstreams of feature data. Additionally, in some instances, display (or audio playback) device 55 can be used to present image (or audio) data. Processing circuitry 56 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Decoding system 50 can also include an optional processor 53, which similarly can include application-specific integrated circuit (ASIC) logic, a graphics processor, an audio processor, a general-purpose processor, etc. Furthermore, memory 54 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 54 can be implemented using high-speed cache memory. In other instances, processing circuitry 56 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0157] In some instances, the encoder 20A, implemented via logic circuitry, may include (e.g., implemented via processing circuitry 56 or memory storage 54) an image buffer and (e.g., implemented via processing circuitry 56) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the encoder 20A implemented via processing circuitry 56. The logic circuitry may be used to determine the various operations discussed herein.

[0158] In some instances, decoder 30A can be implemented in a similar manner via processing circuitry 56 to implement reference. FIG. 1B The decoder 30 and / or any other decoder system or subsystem described herein may be represented by various modules. In some instances, the logic circuit-implemented decoder 30A may include an image buffer (implemented via processing circuitry 56 or memory storage 54) and a graphics processing unit (e.g., implemented via processing circuitry 56). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the image decoder 30A implemented via processing circuitry 56.

[0159] In some instances, antenna 52 can be used to receive encoded bitstreams of image data. As discussed herein, the encoded bitstream may contain data related to encoded audio or video frames, indicators, index values, mode selection data, etc., such as data related to encoded segmentation. Decoding system 50 may also include decoder 30A coupled to antenna 52 for decoding the encoded bitstream. Display (or audio playback) device 55 is used to present the image (or audio).

[0160] It should be understood that, in the embodiments of this application, for the example described with reference to encoder 20A, decoder 30A can be used to determine the reverse process. Regarding signaling syntax elements, decoder 30A can be used to receive and parse such syntax elements, and accordingly decode the associated image data. In some examples, encoder 20A can entropy-encode syntax elements into an encoded bitstream. In such instances, decoder 30A can parse such syntax elements and accordingly decode the associated image data.

[0161] FIG. 1C This is a schematic diagram of a decoding device 400 provided in an embodiment of the present invention. The decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the decoding device 400 may be a decoder, for example... FIG. 1A The image feature map decoder 30A in the image can also be an encoder, for example... FIG. 1A Image feature map encoder 20A.

[0162] Image decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, the processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Image (or audio) decoding device 400 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the entry or exit of optical or electrical signals.

[0163] Processor 430 is implemented in both hardware and software. Processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470 (e.g., a neural network (NN) based decoding module 470). Decoding module 470 implements the embodiments disclosed above. For example, decoding module 470 determines, processes, prepares, or provides various encoding operations. Therefore, decoding module 470 provides a substantial improvement to the functionality of decoding device 400 and affects the switching of decoding device 400 to different states. Alternatively, decoding module 470 may be implemented with instructions stored in memory 460 and determined by processor 430.

[0164] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store such programs during program selection and to store instructions and data read during program selection. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0165] FIG. 1D A simplified block diagram of the apparatus 500 provided for an exemplary embodiment; the apparatus 500 can be used as...FIG. 1A The source device 12 and the destination device 14 are either or both of them.

[0166] The processor 502 in device 500 can be a central processing unit. Alternatively, processor 502 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation can be implemented using a single processor, such as processor 502 as shown in the figure, using more than one processor is faster and more efficient.

[0167] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, which includes at least one program that allows processor 502 to determine the methods described herein. For example, application program 510 may include applications 1 to N, and also includes an image decoding application that determines the methods described herein.

[0168] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0169] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, auxiliary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 500 can have a wide variety of configurations.

[0170] FIG. 2A The system architecture 1800 is shown as a possible image feature map or audio feature variable encoding / decoding scenario, including: Acquisition Device 1801: The video acquisition device completes the acquisition of raw video (or audio); Pre-processing for data acquisition 1802: Raw video (or audio) acquisition undergoes a series of pre-processing steps to obtain video (or audio) data; Encoding 1803: Video (or audio) encoding is used to reduce coding redundancy and reduce the amount of data transmitted during the compression of image feature maps or audio feature variables; Send 1804: The encoded compressed bitstream data is sent through the sending module; Receive 1805: Compressed encoded bitstream data is transmitted over the network and received by the receiving module; Bitstream Decoding 1806: Decodes the bitstream data; Render Display (or Playback) 1807: Render and display (or play back) the decoded data; FIG. 2B This illustrates a possible system architecture 1900 for machine-oriented task scenarios, including image feature maps (or audio feature variables), comprising: Feature Extraction 1901: Extracting features from an image (or audio) source; Edge Information Extraction 1902: Extracting edge information from feature extraction data; Probability Estimation 1903: Edge information is used as input for probability estimation to perform probability estimation on the feature map (or feature variables) to obtain the probability estimation result; Encoding 1904: Entropy encoding is performed on the feature extraction data based on the probability estimation results to obtain the bitstream; Optionally, the feature extraction data can be quantized or rounded before encoding, and then the quantized or rounded feature extraction data can be encoded.

[0171] Optionally, entropy encoding is performed on the edge information so that the bitstream includes the edge information data.

[0172] Decoding 1905: Combine the probability estimation results to perform entropy decoding on the bitstream to obtain image feature maps (or audio feature variables); Optionally, if the bitstream includes side information encoded data, entropy decoding is performed on the side information encoded data, and the decoded side information data is used as input for probability estimation to obtain the probability estimation result.

[0173] It should be noted that when only edge information is used as input for probability estimation, the probability estimation results of each feature element can be output in parallel; when the input for probability estimation includes context information, the probability estimation results of each feature element need to be output serially. The edge information refers to feature information further extracted from image feature maps or audio feature variables input into a neural network. The number of feature elements contained in the edge information is less than the number of feature elements in the image feature map or audio feature variable. Optionally, the edge information of the image feature map or audio feature variable can be encoded into the bitstream.

[0174] Machine Vision Task 1906: Perform machine vision (or auditory) tasks on decoded feature maps (or feature variables).

[0175] Specifically, the decoded feature data is input into the machine vision (or hearing) task network, and the network output is one-dimensional, two-dimensional, or multi-dimensional data related to the vision (or hearing) task, such as classification, object recognition, semantic segmentation, etc.

[0176] In one possible implementation, during the implementation of system architecture 1900, feature extraction and encoding are performed on the terminal, while decoding and execution of machine vision tasks are performed in the cloud.

[0177] Encoder 20A can be used to receive images (or image data) or audio (or audio data) 17 via input 202, etc. The received images, image data, audio, and audio data can also be pre-processed images (or pre-processed image data) or audio (or pre-processed audio data) 19. For simplicity, the following description uses the term image (or audio) 17. Image (or audio) 17 can also be referred to as the current image or the image to be encoded (especially in video encoding where the current image is distinguished from other images, such as those in the same video sequence, i.e., including previously encoded and / or decoded images in the video sequence of the current image) or the current audio or the audio to be encoded.

[0178] A (digital) image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. Similarly, each pixel can be represented in a luma / chroma format or color space, such as YCbCr, including the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in a grayscale image), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image includes a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also called color transformation or conversion. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays. The image encoder 20A does not restrict the color space of the image.

[0179] In one possibility, embodiments of encoder 20A may include an image (or audio) segmentation unit. FIG. 1A or FIG. 1B(Not shown in the image) 17 is used to segment an image (or audio) 17 into multiple (typically non-overlapping) image blocks 203 or audio segments. These image blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. Segmentation units can be used to apply the same block size to all images in a video sequence and to a corresponding grid with a defined block size, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0180] In another possibility, the encoder can be used to directly receive blocks 203 of image 17, for example, one, several, or all of the blocks that make up image 17. Image block 203 can also be referred to as the current image block or the image block to be encoded.

[0181] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a single pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0182] In another possibility, FIGS. 1A-1B or FIGS. 3A-3D The encoder 20A shown is used to encode the image 17 block by block.

[0183] In another possibility, FIGS. 1A-1B or FIGS. 3A-3D The encoder 20A shown is used to encode image 17.

[0184] In another possibility, FIGS. 1A-1B or FIGS. 3A-3D The encoder 20A shown can also be used to segment encoded images using slices (also known as video slices), where an image can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and subpictures in the VVC standard).

[0185] In another possibility, FIGS. 1A-1B or FIGS. 3A-3D The encoder 20A shown can also be used to segment and / or encode images using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the images can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0186] Coding Network 20 The encoding network 20 is used to obtain image feature maps or audio feature variables based on the input data.

[0187] In one possibility, coding network 20, such as FIG. 4A As shown, the encoding network 20 contains multiple network layers, any of which can be a convolutional layer, a normalization layer, a non-linear activation layer, etc.

[0188] In one possibility, the input to the encoding network 20 is at least one image to be encoded or at least one block of images to be encoded. The image to be encoded can be a raw image, a lossy image, or a residual image.

[0189] In one possibility, an example of the network structure of the coding network in coding network 20 is as follows: FIG. 4B As shown, the encoding network in the example contains five network layers, specifically three convolutional layers and two non-linear activation layers.

[0190] Round to 24 Rounding is used to round image feature maps or audio feature variables through, for example, scalar quantization or vector quantization, to obtain rounded image feature maps or audio feature variables.

[0191] In one possibility, encoder 20A can be used to output a quantization parameter (QP), for example, directly outputting it or outputting it after encoding or compression by the encoding decision implementation unit, such that decoder 30A can receive it and decode it using the quantization parameter.

[0192] In one possibility, the output feature map or feature audio feature variables are preprocessed before rounding. Preprocessing may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction.

[0193] Probability estimate 40 Probability estimation is based on input feature maps or feature variable information to obtain probability estimation results for image feature maps or audio feature variables.

[0194] Probability estimation is used to estimate the probability of image feature maps or audio feature variables after rounding.

[0195] Probability estimation can be achieved through probability estimation networks, which are convolutional networks. Convolutional networks include convolutional layers and non-linear activation layers. FIG. 4B For example, the probabilistic estimation network consists of five layers: three convolutional layers and two non-linear activation layers. Probabilistic estimation can be implemented using traditional non-network probabilistic estimation methods. These methods include, but are not limited to, statistical methods such as equal maximum likelihood estimation, maximum a posteriori estimation, and maximum likelihood estimation.

[0196] Encoding decision implementation 26 like FIG. 5 As shown, the encoding decision implementation includes encoding element judgment and entropy encoding. The image feature map or audio feature variable is one-dimensional, two-dimensional, or multi-dimensional data output by the encoding network, where each data point is a feature element. Encoding element judgment 261 Encoding element judgment is based on the probability estimation results to judge each feature element in the image feature map or audio feature variable, and determines which feature elements to perform entropy coding based on the judgment results.

[0197] After the element judgment process of the Pth feature element of the image feature map or audio feature variable is completed, the element judgment process of the (P+1)th feature element of the image feature map begins, where P is a positive integer and P is less than M.

[0198] Entropy coding 262 Entropy coding can employ various publicly available entropy coding algorithms, such as variable length coding (VLC), context adaptive VLC (CAVLC), entropy coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques. This results in encoded image data 25, which can be output as an encoded bitstream 25 through output terminal 212, allowing decoder 30A to receive and use parameters for decoding. The encoded bitstream 25 can be transmitted to decoder 30A or stored in memory for later transmission or retrieval by decoder 30A.

[0199] In another possibility, entropy coding can be performed using entropy coding networks, such as convolutional networks.

[0200] In one possibility, since entropy coding does not know the true character probability of the rounded feature map, it can statistically add this or related information to the entropy coding and then transmit this information to the decoding end.

[0201] United Network 44 A joint network obtains probability estimates and decision information for image feature maps or audio feature variables based on input side information. A joint network is a multi-layered network, which can be a convolutional network, including convolutional layers and non-linear activation layers. Any layer in a joint network can be a convolutional layer, a normalization layer, a non-linear activation layer, etc.

[0202] The decision information can be one-dimensional, two-dimensional, or multi-dimensional data, and the size of the decision information can be consistent with the size of the image feature map.

[0203] The decision information can be output after any network layer in the joint network.

[0204] The probability estimation result can be output after any network layer in the joint network.

[0205] FIG. 6The example output of the network structure for the joint network consists of four network layers, where decision information is output after the fourth network layer and probability estimation results are output after the second network layer.

[0206] Generative Network 46 Generative networks (GNNs) generate decision information for each feature element in an image feature map based on input probability estimation results. GNNs are multi-layered networks, and can be convolutional networks, which include convolutional layers and non-linear activation layers. Any layer in a GNN can be a convolutional layer, a normalization layer, a non-linear activation layer, etc.

[0207] The decision information can be output after any network layer in the generative network. The decision information can be one-dimensional, two-dimensional, or multi-dimensional data.

[0208] FIG. 7 Example of output decision information for generating network structure, which includes 4 network layers.

[0209] Decoding decision-making achievement 30 like FIG. 8 As shown, the decoding decision implementation includes element judgment and entropy decoding. The image feature map or audio feature variable is one-dimensional, two-dimensional, or multi-dimensional data output by the decoding decision implementation, where each data point is a feature element. Decoding element judgment 301 The decoding element judgment process assesses each feature element in the image feature map or audio feature variable based on the probability estimation results, and determines which feature elements to perform entropy decoding based on the assessment results. This decoding element judgment process, which assesses each feature element in the image feature map and determines which feature elements to perform entropy encoding based on the assessment results, can be seen as the inverse process of the encoding element judgment process, which assesses each feature element in the image feature map and determines which feature elements to perform entropy encoding based on the assessment results.

[0210] Entropy Decoding 302 Entropy decoding can employ various publicly available entropy decoding algorithms, such as variable-length coding (VLC), context-adaptive VLC (CAVLC), entropy decoding schemes, binarization algorithms, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques. This results in encoded image (or audio) data 25 that can be output as an encoded bitstream 25 through output terminal 212, allowing decoder 30A to receive and use the parameters for decoding. The encoded bitstream 25 can be transmitted to decoder 30A or stored in memory for later transmission or retrieval by decoder 30A.

[0211] In another possibility, entropy decoding can be performed using an entropy decoding network, such as a convolutional network.

[0212] Decoding Network 34 The decoding network is used to pass the decoded image feature map or audio feature variable 31 or the post-processed decoded image feature map or audio feature variable 33 through the decoding network 34 to obtain reconstructed image (or audio) data 35 or machine-oriented task data in the pixel domain.

[0213] The decoding network comprises multiple network layers, any of which can be a convolutional layer, a normalization layer, a non-linear activation layer, etc. The decoding network unit 306 can contain operations including concatenation, addition, and subtraction.

[0214] In one possibility, the structures of the network layers in the decoding network can be the same or different from each other.

[0215] Example of a decoding network structure: FIG. 9 As shown, the decoding network in the example contains 5 network layers, specifically including one normalization layer, two convolutional layers, and two non-linear activation layers.

[0216] The decoding network outputs a reconstructed image (or audio), or outputs machine-oriented task-oriented data. Specifically, the decoding network may include an object recognition network, a classification network, or a semantic segmentation network.

[0217] It should be understood that in encoder 20A and decoder 30A, the processing result of the current step can be further processed and then output to the next step. For example, after encoder unit or decoder unit, the processing result of encoder unit or decoder unit can be further calculated or processed, such as clipping or shifting operations or filtering.

[0218] Based on the above description, the following are some encoding and decoding methods for image feature maps or audio feature variables provided by embodiments of this application. For convenience, the various method embodiments described below are all described as a combination of a series of action steps. However, those skilled in the art should understand that the specific implementation of the technical solution of this application is not limited by the order of the described series of action steps.

[0219] The process of this application will now be described in detail with reference to the accompanying drawings. It should be noted that the encoding process in the flowchart can be specifically executed by the encoder 20A described above, and the decoding process in the flowchart can be specifically executed by the decoder 30A described above.

[0220] In Examples 1 to 5, the first feature element or the second feature element is either the current feature element to be encoded or the current feature element to be decoded, or, for example... A decision graph can also be called a decision map. A binary graph is preferred for decision graphs, and a binary graph can also be called a binary map.

[0221] In Embodiment 1 of this application, FIG. 10A The specific implementation process 1400 is shown, and the operation steps are as follows: Encoding end: Step 1401: Obtain the feature map of the image This step is specifically made by FIG. 3A This is implemented using encoding network 204, which can be found in the description of encoding network 20 above. The images are then input into the feature extraction module, which outputs feature maps of the images. Feature map The three-dimensional data of wxh xc. Specifically, the feature extraction module can be implemented using existing neural networks, which are not limited here. This step is based on existing technology.

[0222] The feature quantization module performs feature map quantization. Each feature value in the image is quantized, and the floating-point feature values ​​are rounded to obtain integer feature values, resulting in the quantized feature map. The description of rounding to 24 can be found in the above embodiments.

[0223] Step 1402: Process the feature map Probability estimation is performed to obtain the probability estimates of each feature element, i.e., the feature map. Each feature element in probability distribution: Where the parameters x, y, and i are positive integers, and the coordinates (x, y, i) represent the position of the current feature element to be encoded. Specifically, the coordinates (x, y, i) represent the position of the current feature element to be encoded relative to the top-left vertex of the current 3D feature map. This step is specifically... FIG. 3A The probability estimation in section 210 is implemented, and the details can be found in the description of probability estimation 40 above. Specifically, a probability distribution model can be used to obtain the probability distribution, such as using a Gaussian single model (GSM) or a Gaussian mixture model (GMM). First, the edge information... In conjunction with contextual information, a probability estimation network is used to process feature maps. Each feature element in Probability estimation is performed to obtain each feature element. The probability distribution is obtained. The probability estimation network can use deep learning-based networks, such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (PixelCNNs), etc., without limitation here. Substituting the model parameters into the probability distribution model yields the probability distribution.

[0224] Step 1403: Process the feature map Entropy encoding is performed to obtain a compressed bitstream, and a compressed bitstream is generated.

[0225] This step is specifically made by FIG. 3A The encoding decision implementation 208 is implemented in the above description of the encoding decision implementation 26. Based on the probability distribution, the current feature element to be encoded is obtained. The probability P that takes the value k is given when the current feature element to be encoded... If the probability estimation result P of the current feature element does not meet the preset condition: when P is greater than (or equal to) the first threshold T0, skip the entropy encoding process for the current feature element to be encoded; otherwise, if the probability estimation result P of the current feature element to be encoded meets the preset condition: when P is less than the first threshold T0, perform entropy encoding on the current feature element to be encoded and write it into the bitstream. Here, k can be any integer, such as 0, 1, -1, 2, 3, etc. The first threshold T0 is the condition that satisfies... Any number in the set, for example, 0.99, 0.98, 0.97, 0.95, etc. (It can be assumed that the threshold for each feature element is the same).

[0226] Step 1404: The encoder sends or stores the compressed bitstream.

[0227] Decoding end: Step 1411: Obtain the bitstream of the decoded image feature map Step 1412: Perform probability estimation based on the bitstream to obtain the probability estimation results of each feature element. This step is specifically made by FIG. 10B The probability estimation 302 is implemented in this way; for details, please refer to the description of probability estimation 40 above. This is for the feature map to be decoded. Each feature element in Probability estimation is performed to obtain the feature elements to be decoded. The probability distribution of the feature map to be decoded. It includes multiple feature elements, including the feature element to be decoded.

[0228] The probability estimation network structure used at the decoding end is the same as that at the encoding end in this embodiment.

[0229] Step 1413: Feature map to be decoded Execution entropy decoding This step is specifically made by FIG. 10B The decoding decision implementation 304 is implemented in the above description of the decoding decision implementation 30. Based on the probability distribution of the current feature element to be decoded, the probability P of the current feature element taking the value k is obtained, i.e., the probability estimation result P of the current feature element to be decoded. When the probability estimation result P does not meet the preset condition: P is greater than the first threshold T0, entropy decoding is not required for the current feature element to be decoded, and the value of the current feature element to be decoded is set to k. Otherwise, when the current feature element to be decoded meets the preset condition: P is less than or equal to the first threshold T0, entropy decoding is performed on the bitstream to obtain the value of the current feature element to be decoded.

[0230] The first threshold T0 can be obtained by parsing the bitstream, retrieving the index number from the bitstream, and constructing a threshold candidate list using the same method as the encoding end. Then, the corresponding threshold is obtained based on the pre-defined correspondence between the threshold candidate list and the index number. Retrieving the index number from the bitstream means retrieving the index number from the sequence header, image header, slice / band, or SEI. Alternatively, the bitstream can be directly parsed to obtain the threshold, specifically from the sequence header, image header, slice / band, or SEI.

[0231] Alternatively, a fixed threshold can be set directly based on the threshold agreement strategy with the decoding.

[0232] Step 1414: Decode the feature map Perform reconstruction, or input the corresponding machine task into the machine vision task module to execute the corresponding machine task. This step can be specifically performed by... FIG. 10B The decoding network 306 is used to implement this, and you can refer to the description of the decoding network 34 above for details.

[0233] Scenario 1: Feature map after entropy decoding The input image reconstruction module outputs a reconstructed image from the neural network. The neural network can employ any structure, such as a fully connected network, convolutional neural network, or recurrent neural network. A multi-layered deep neural network structure can be used to achieve better estimation results.

[0234] Scenario 2: The feature map after entropy decoding The input is directed to the machine vision task module to execute the corresponding machine task. For example, it can complete machine vision tasks such as object classification, recognition, and segmentation.

[0235] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0236] FIG. 11A The specific implementation flow 1500 of Embodiment 2 of this application is shown, and the operation steps are as follows: It should be noted that in methods one to six of this embodiment: the probability estimation result includes a first parameter and a second parameter; when the probability distribution is a Gaussian distribution, the first parameter is the mean μ and the second parameter is the variance σ; when the probability distribution is a Laplace distribution, the first parameter is the location parameter μ and the second parameter is the scale parameter b.

[0237] Encoding end: Step 1501: Obtain the feature map of the image This step is specifically made by FIG. 3B This is implemented using encoding network 204, which can be found in the description of encoding network 20 above. The images are then input into the feature extraction module, which outputs feature maps of the images. Specifically, the feature extraction module can be implemented using existing neural networks, and this is not a limitation. This step is based on existing technology.

[0238] The feature quantization module performs feature map quantization. Each feature value in the image is quantized, and the floating-point feature values ​​are rounded to obtain integer feature values, resulting in the quantized feature map. . Step 1502: Feature map of the image Input edge information extraction module, output edge information .

[0239] This step is specifically made by FIG. 3BThe edge information extraction unit 214 in the middle is used to implement this. The edge information extraction module can use... FIG. 12 The network shown is used to implement edge information. This can be understood as the feature map The feature map obtained by further extraction , The number of feature elements contained is greater than that of the feature map. few.

[0240] It should be noted that edge information can be obtained in this step. Entropy encoding is performed and written to the bitstream; edge information can also be processed in subsequent step 1504. Entropy encoding is performed and written to the bitstream; no specific restrictions are imposed here.

[0241] Step 1503: Process the feature map Probability estimation is performed to obtain the probability estimation results for each feature element. This step is specifically made by FIG. 3B The probability estimation 210 is implemented in the above description of probability estimation 40. The probability estimation results and probability distribution can be obtained using a probability distribution model. The probability distribution model can be: a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model.

[0242] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or Gaussian mixture model), the edge information is first... Alternatively, context information can be input into a probability estimation network to evaluate the feature map. Each feature element in Probability estimation is performed to obtain the values ​​of the mean parameter μ and the variance σ. Further, the mean parameter μ and the variance σ are input into the probability distribution model used to obtain the probability distribution. At this point, the probability estimation result is the mean parameter μ and the variance σ.

[0243] When using the Laplace distribution model for probability distribution, first consider the edge information. Alternatively, context information can be input into a probability estimation network to evaluate the feature map. Each feature element in Probability estimation is performed to obtain the values ​​of the position parameter μ and the scale parameter b. Further, the position parameter μ and the scale parameter b are input into the used probability distribution model to obtain the probability distribution. At this point, the probability estimation result is the position parameter μ and the scale parameter b.

[0244] You can also add edge information And / or contextual information is input into the probability estimation network to encode the feature map. Each feature element in Probability estimation is performed to obtain the current feature element to be encoded. The probability distribution is then used to obtain the current feature element to be encoded. The probability P is the value of m. The probability estimate at this point is the current feature element to be encoded. The probability P of taking the value m.

[0245] The probability estimation network can use deep learning-based networks, such as recurrent neural networks (RNNs) and convolutional neural networks (PixelCNNs), without any limitation.

[0246] Step 1504: Determine the current feature element to be encoded based on the probability estimation result. Whether entropy encoding needs to be performed, and whether to perform entropy encoding and write it into the compressed bitstream (encoded bitstream) based on the judgment result, or not to perform entropy encoding. Entropy encoding is only performed on the current feature element to be encoded if it is determined that entropy encoding needs to be performed on the current feature element to be encoded.

[0247] This step is specifically made by FIG. 3B The encoding decision implementation 208 is implemented in the code, and the details can be found in the description of the encoding decision implementation 26 above. The current feature element to be encoded is determined based on the probability estimation result. Whether entropy encoding needs to be performed can be determined using one or more of the following methods. Here, the parameters x, y, and i are positive integers, and the coordinates (x, y, i) represent the position of the current feature element to be encoded. Specifically, the coordinates (x, y, i) represent the position of the current feature element to be encoded relative to the top-left vertex of the current 3D feature map.

[0248] Method 1: When the probability distribution model is a Gaussian distribution, determine whether to perform entropy encoding on the current feature element based on the probability estimation result of the first feature element. If the mean parameter μ and variance σ of the Gaussian distribution of the current feature element do not meet the preset conditions: when the absolute value of the difference between the mean μ and k is less than the second threshold T1 and the variance σ is less than the third threshold T2, it is not necessary to encode the current feature element. Perform the entropy encoding process; otherwise, if the preset conditions are met—that is, when the absolute value of the difference between the mean μ and k is greater than or equal to the second threshold T1 or the variance σ is less than the third threshold T2—then the current feature element to be encoded is... Perform entropy coding and write to the bitstream. Here, k is an arbitrary integer, such as 0, 1, -1, 2, 3, etc. T2 takes any value satisfying 0 < T2 < 1, such as 0.2, 0.3, 0.4, etc. T1 is a number greater than or equal to 0 and less than 1, such as 0.01, 0.02, 0.001, 0.002.

[0249] Specifically, when k takes the value of 0, it is the optimal value. It can be directly determined that when the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, then skip the current feature element to be encoded Perform the entropy coding process; otherwise, for the current feature element to be encoded Perform entropy coding and write to the bitstream. Here, the value of T2 takes any value satisfying 0 < T2 < 1, such as 0.2, 0.3, 0.4, etc. T1 is a number greater than or equal to 0 and less than 1, such as 0.01, 0.02, 0.001, 0.002.

[0250] Method 2: When the probability distribution is a Gaussian distribution, according to the probability estimation result, obtain the values of the mean parameter μ and the variance σ of the Gaussian distribution of the current feature element to be encoded When the relationship between the mean μ, variance σ, and k satisfies (not satisfying the preset condition), skip the current feature element to be encoded Perform the entropy coding process, where abs(μ - k) represents calculating the absolute value of the difference between the mean μ and k; otherwise, when the probability estimation result of the current feature element to be encoded satisfies (preset condition), for the current feature element to be encoded Perform entropy coding and write to the bitstream. Here, k is an arbitrary integer, such as 0, 1, -1, -2, 3, etc. The fourth threshold T3 is a number greater than or equal to 0 and less than 1, such as 0.2, 0.3, 0.4, etc.

[0251] Method 3: When the probability distribution is a Laplace distribution, according to the probability estimation result, obtain the values of the location parameter μ and the scale parameter b of the Laplace distribution of the current feature element to be encoded When the relationship between the location parameter μ, scale parameter b, and k satisfies (not satisfying the preset condition), skip the current feature element to be encoded Perform the entropy coding process, where abs(μ - k) represents calculating the absolute value of the difference between the location parameter μ and k; otherwise, when the probability estimation result of the current feature element to be encoded satisfies (preset condition), for the current feature element to be encoded Entropy encoding is performed and written to the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. The fourth threshold T4 is a number greater than or equal to 0 and less than 0.5, for example, values ​​such as 0.05, 0.09, 0.17, etc.

[0252] Method 4: When the probability distribution is a Laplace distribution, the current feature element to be encoded is obtained based on the probability estimation result. The values ​​of the location parameter μ and scale parameter b of the Laplace distribution. When the absolute value of the difference between the location parameter μ and k is less than the second threshold T5 and the scale parameter b is less than the third threshold T6 (not meeting the preset condition), skip the current feature element to be encoded. Perform the entropy encoding process; otherwise, if the absolute value of the difference between the position parameter μ and k is less than the second threshold T5 or the scale parameter b is greater than or equal to the third threshold T6 (preset condition), then the current feature element to be encoded is... Perform entropy encoding and write the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. T5 takes the value 1e-2, and T6 takes any number that satisfies T6<0.5, such as 0.05, 0.09, 0.17, etc.

[0253] Specifically, when k is 0, it is the optimal value. It can be directly determined that if the absolute value of the position parameter μ is less than T5 and the scale parameter b is less than T6, then the current feature element to be encoded is skipped. Perform the entropy encoding process; otherwise, for the current feature element to be encoded... Entropy encoding is performed and written to the bitstream. The threshold T5 is 1e-2, and T2 is any number that satisfies T6 < 0.5, such as 0.05, 0.09, 0.17, etc.

[0254] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature element to be encoded is obtained based on the probability estimation result. All mean parameters μ of the mixture Gaussian distribution i and variance σ i The value of k. When the sum of the absolute values ​​of the differences between all the means of the Gaussian mixture distribution and k, and the sum of any variance of the Gaussian mixture distribution, is less than the fifth threshold T7 (the preset condition is not met), skip the current feature element to be encoded. Perform the entropy encoding process; otherwise, if the sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution and k, and the sum of any variance of the Gaussian mixture distribution, are greater than or equal to the fifth threshold T7 (preset condition), then for the current feature element to be encoded... Entropy encoding is performed and written to the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. T7 is a number greater than or equal to 0 and less than 1, for example, values ​​such as 0.2, 0.3, 0.4, etc. (It can be assumed that the threshold for each feature element is the same). Method 6: Obtain the current feature element to be encoded based on the probability distribution. The probability P is set to value k. If the probability estimate P of the current feature element to be encoded does not satisfy a preset condition: when P is greater than (or equal to) a first threshold T0, the entropy encoding process for the current feature element to be encoded is skipped; otherwise, if the probability estimate P of the current feature element to be encoded satisfies a preset condition: when P is less than the first threshold T0, entropy encoding is performed on the current feature element to be encoded and written into the bitstream. Here, k can be any integer, such as 0, 1, -1, 2, 3, etc. The first threshold T0 is the condition that satisfies... Any number in the set, for example, values ​​such as 0.99, 0.98, 0.97, 0.95, etc. (It can be assumed that the threshold for each feature element is the same). It should be noted that, in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 can be rounded up to integers.

[0255] It should be noted that the threshold can also be obtained using one of the following methods, which are not limited here: Method 1: Taking threshold T1 as an example, any value within the range of T1 is taken as threshold T1, and threshold T1 is written into the bitstream. Specifically, the threshold can be written into the bitstream and stored in the sequence header, image header, slice / band, or SEI before being transmitted to the decoding end. Other methods can also be used, which are not limited here. Similar methods can be used for the other thresholds T0, T2, T3, T4, T5, and T6.

[0256] Method 2: The encoding end uses a fixed threshold agreed upon with the decoding end, eliminating the need to write to the bitstream or transmit to the decoding end. For example, taking threshold T1 as an example, any value within the range of T1 can be directly used as the value of T1. Similar methods can be used for the other thresholds T0, T2, T3, T4, T5, and T6.

[0257] Method 3: Construct a threshold candidate list. The most likely values ​​within the range of T1 are placed in the threshold candidate list, with each threshold corresponding to a threshold index number. An optimal threshold is determined, and this optimal threshold is used as the value of T1. The index number of the optimal threshold is used as the threshold index number of T1, and the threshold index number of T1 is written into the bitstream. Specifically, the threshold is written into the bitstream and can be stored in the sequence header, image header, slice / strip, or SEI before being transmitted to the decoding end. Other methods can also be used, which are not limited here. Similar methods can be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0258] Step 1505: The encoder sends or stores the compressed bitstream.

[0259] Decoding end: Step 1511: Obtain the bitstream of the feature map of the image to be decoded. Step 1512: Obtain the probability estimation results for each feature element. This step is specifically made by FIG. 11A This is implemented using probability estimation unit 302, which can be referred to in the above description of probability estimation 40. Edge information. Perform entropy decoding to obtain edge information Combined with edge information Feature maps to be decoded Each feature element in Perform probability estimation to obtain the current feature element to be decoded. The probability estimation results.

[0260] It should be noted that the probability estimation method used at the decoding end is the same as the probability estimation method at the encoding end in this embodiment, and the probability estimation network structure diagram is the same as the probability estimation network structure at the encoding end in this embodiment, so it will not be described again here.

[0261] Step 1513: This step is specifically composed of... FIG. 11A The decoding decision implementation 304 is implemented in the code, and the details can be found in the description of the decoding decision implementation 30 above. The current feature element to be decoded is determined based on the probability estimation result. Determine whether entropy decoding is needed, and decide whether to perform entropy decoding based on the result, to obtain the decoded feature map. .

[0262] Determine the current feature element to be decoded based on the probability estimation result. Whether entropy decoding needs to be performed can be determined using one or more of the following methods.

[0263] Method 1: When the probability distribution model is a Gaussian distribution, the current feature element to be decoded is obtained based on the probability estimation result. The values of the mean parameter μ and the variance σ. When the absolute value of the difference between the mean μ and k is less than the second threshold T1 and the variance σ is less than the third threshold T2 (the preset condition is not satisfied), the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded is skipped is performed; otherwise, when the absolute value of the difference between the mean μ and k is less than the second threshold T1 or the variance σ is greater than or equal to the third threshold T2 (the preset condition), for the current feature element to be decoded entropy decoding is performed to obtain the value of the current feature element to be decoded .

[0264] Specifically, when k takes the value of 0, it is the optimal value. It can be directly determined that when the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, the value of the current feature element to be decoded is set to k, then the entropy decoding process for the current feature element to be decoded is skipped is performed, otherwise, for the current feature element to be decoded entropy decoding is performed to obtain the value of the current feature element to be decoded .

[0265] Method 2: When the probability distribution is a Gaussian distribution, based on the probability estimation result, the values of the mean parameter μ and the variance σ of the current feature element to be decoded are obtained. When the relationship between the mean μ, the variance σ and k satisfies abs(μ - k)+σ < T3 (the preset condition is not satisfied), where T3 is the fourth threshold, the value of the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded is skipped is performed, otherwise, when the probability estimation result of the current feature element to be decoded satisfies , for the current feature element to be decoded entropy decoding is performed to obtain the value of the current feature element to be decoded .

[0266] Method 3: When the probability distribution is a Laplace distribution, based on the probability estimation result, the values of the location parameter μ and the scale parameter b are obtained. When the relationship between the location parameter μ, the scale parameter b and k satisfies abs(μ - k)+σ < T4 (the preset condition is not satisfied), where T4 is the fourth threshold, the value of the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded is skipped is performed, otherwise, when the probability estimation result of the current feature element to be decoded satisfies (the preset condition), for the current feature element to be decoded entropy decoding is performed to obtain the value of the current feature element to be decoded The value of .

[0267] Method 4: When the probability distribution is a Laplace distribution, the values ​​of the position parameter μ and the scale parameter b are obtained based on the probability estimation results. When the absolute value of the difference between the position parameter μ and k is less than the second threshold T5 and the scale parameter b is less than the third threshold T6 (the preset condition is not met), the current feature element to be decoded is... The value is set to k, skipping the current feature element to be decoded. Perform the entropy decoding process; otherwise, if the absolute value of the difference between the position parameter μ and k is less than the second threshold T5 or the scale parameter b is greater than or equal to the third threshold T6 (preset condition), proceed with the current feature element to be decoded. Perform entropy decoding to obtain the current feature element to be decoded. The value of .

[0268] Specifically, k is optimal when it is 0. This allows us to directly determine when the absolute value of the position parameter μ is less than T5 and the scale parameter b is less than T6, and then move the current feature element to be decoded. The value is set to k, skipping the current feature element to be decoded. Execute the entropy decoding process; otherwise, perform the current feature element to be decoded. Perform entropy decoding to obtain the current feature element to be decoded. The value of .

[0269] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature element to be decoded is obtained based on the probability estimation result. All mean parameters μ of the mixture Gaussian distribution i and variance σ i The value of . When the sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution and the value k of the current feature element to be decoded, and the sum of any variance of the Gaussian mixture distribution, is less than the fifth threshold T7 (the preset condition is not met), the current feature element to be decoded is . The value is set to k, skipping the current feature element to be decoded. Perform the entropy decoding process; otherwise, if the sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution and the value k of the current feature element to be decoded, and the sum of any variance of the Gaussian mixture distribution, is greater than or equal to the fifth threshold T7 (preset condition), then for the current feature element to be decoded... Perform entropy decoding to obtain the current feature element to be decoded. The value of .

[0270] Method 6: Based on the probability distribution of the current feature element to be decoded, obtain the probability P of the current feature element to be decoded taking the value k, that is, the probability estimation result P of the current feature element to be decoded. When the probability estimation result P does not meet the preset condition: P is greater than the first threshold T0, there is no need to perform entropy decoding on the current feature element to be decoded, and the value of the current feature element to be decoded is set to k. Otherwise, when the current feature element to be decoded meets the preset condition: P is less than or equal to the first threshold T0, entropy decoding is performed on the bitstream to obtain the value of the current feature element to be decoded.

[0271] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0272] The methods for obtaining thresholds T0, T1, T2, T3, T4, T5, T6, and T7 correspond to those used at the encoding end, and one of the following methods can be used: Method 1: Obtain the threshold from the bitstream, specifically from the sequence header, image header, slice / band, or SEI.

[0273] Method 2: The decoding end uses a fixed threshold agreed upon with the encoding end.

[0274] Method 3: Obtain the threshold index number from the bitstream. Specifically, obtain the threshold index number from the sequence header, image header, slice / band, or SEI. Then, the decoder constructs a threshold candidate list in the same way as the encoder, and obtains the corresponding threshold from the threshold candidate list based on the threshold index number.

[0275] It should be noted that, in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 can be rounded up to integers.

[0276] Step 1514: Same as step 1414.

[0277] FIG. 13A The specific implementation flow 1600 provided in Embodiment 3 of this application is shown, and the operation steps are as follows: Encoding end: Step 1601: Same as step 1501, this step is specifically... FIG. 3C The coding network 204 is used to implement this, and you can refer to the description of coding network 20 above for details; Step 1602: Same as step 1502, this step is specifically... FIG. 3C Edge information extraction is achieved using 214; Step 1603: Process the feature map Probability estimation is performed to obtain the probability estimation results for each feature element; This step can be specifically described by FIG. 3CThe probability estimation is implemented using probability estimation 210, which can be referred to in the above description of probability estimation 40. The probability estimation result can be obtained using a probability distribution model. This probability distribution model can be: a single Gaussian model, an asymmetric Gaussian model, a Gaussian mixture model, or a Laplace distribution model.

[0278] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or Gaussian mixture model), the edge information is first... Alternatively, context information can be input into a probability estimation network to evaluate the feature map. Each feature element in Probability estimation is performed to obtain the values ​​of the model parameters, the mean parameter μ and the variance σ, which are the probability estimation results.

[0279] When using the Laplace distribution model for probability distribution, first consider the edge information. Alternatively, context information can be input into a probability estimation network to evaluate the feature map. Each feature element in Probability estimation is performed to obtain the values ​​of the model parameters, location parameter μ and scale parameter b, which are the probability estimation results.

[0280] Furthermore, the probability estimation result is input into the probability distribution model used to obtain the probability distribution. Alternatively, edge information And / or contextual information is input into the probability estimation network to encode the feature map. Each feature element in Probability estimation is performed to obtain the current feature element to be encoded. The probability distribution is then used to obtain the current feature element to be encoded. The probability P is the value m. Here, m is any integer, such as 0, 1, -1, -2, 3, etc.

[0281] The probability estimation network can use deep learning-based networks, such as recurrent neural networks and convolutional neural networks, without any restrictions.

[0282] Step 1604: Determine whether to perform entropy encoding on the current feature element to be encoded based on the probability estimation result. Depending on the determination result, perform entropy encoding on the current feature element to be encoded and write it into the encoded bitstream, or do not perform entropy encoding. Entropy encoding is performed only on the current feature element to be encoded if it is determined that entropy encoding is necessary.

[0283] This step is specifically made by FIG. 3CThe generation network 216 and encoding decision implementation 208 are used to implement this, and the specific implementation can be found in the description of the generation network 46 and encoding decision implementation 26 above. The probability estimation result 211 is input into the judgment module, and the output is compared with the feature map. Decision information 217 of the same dimension. In this embodiment, decision information 217 can be a three-dimensional decision map. The judgment module can be implemented using a network method, that is, by inputting the probability estimation result or probability distribution. FIG. 7 The generator network shown outputs a decision graph (map). The preset value represents the current feature element to be encoded at the corresponding position. Entropy encoding is required; entropy encoding is performed on the current feature element to be encoded based on the probability distribution. The preset value represents the current feature element to be encoded at the corresponding position. The high probability value is k. A value not specified by the preset value indicates the current feature element to be encoded at the corresponding position. Entropy encoding is not required; that is, the entropy encoding process is skipped. The decision information is related to the feature map. Decision graphs with the same dimensions. Indicate decision The value at coordinates (x, y, i) in the map. This refers to the current feature element to be encoded in the decision map output by the generating network. When there are only two possible values, the preset value is a specific numerical value. For example, when the possible values ​​for the current feature element to be encoded are 0 and 1, the preset value is 0 or 1. When generating the network output decision map map, the current feature element to be encoded... When there are multiple possible values, the preset values ​​are some specific numerical values, such as the current feature element to be encoded. When the element can be selected with a value of 0 to 255, the default value is a proper subset of 0 to 255.

[0284] In one possible implementation, the probability estimate or probability distribution of the current feature element to be encoded is input to a judgment module. The judgment module directly outputs decision information on whether the current feature element needs to undergo entropy encoding. For example, if the decision information output by the judgment module is a preset value, it indicates that the current feature element needs to undergo entropy encoding; if the decision information output by the judgment module is not a preset value, it indicates that the current feature element does not need to undergo entropy encoding. The judgment module can be implemented using a network method, that is, by inputting the probability estimate or probability distribution... FIG. 7 The generator network shown outputs decision information, i.e., preset values.

[0285] Method 1: Decision information is related to feature maps Decision graphs with the same dimensions The preset value represents the current feature element to be encoded at the corresponding position. Entropy encoding is required; entropy encoding is performed on the current feature element to be encoded based on the probability distribution. A value not specified by the preset value indicates the current feature element to be encoded at the corresponding position. The high probability value is k. A value of 0 indicates the current feature element to be encoded at the corresponding position. Entropy encoding is not required, meaning the entropy encoding process is skipped. When the feature elements in the decision graph (map) are... When there are only two possible values, the default value is a specific numerical value. For example, when the possible values ​​for a feature element are 0 and 1, the default value is 0 or 1. When the feature element in the decision graph map... When there are multiple possible values, the default values ​​are some specific numerical values, such as feature elements. When the element can be selected with a value of 0 to 255, the default value is a proper subset of 0 to 255.

[0286] Method 2: Decision information is related to feature maps Decision graphs with the same dimensions A value greater than or equal to the threshold T0 indicates the current feature element to be encoded at the corresponding position. Entropy encoding is required, which involves performing entropy encoding on the current feature element to be encoded based on the probability distribution. Less than the threshold T0 indicates the current feature element to be encoded at the corresponding position. The high probability value is k, which represents the current feature element to be encoded at the corresponding position. Entropy coding is not required; that is, the entropy coding process is skipped. Based on the numerical range of the decision graph, T0 can be the mean value within that range.

[0287] Method 3: The decision information can also be the identifier or identifier value directly output by the joint network. When the decision information is a preset value, it indicates that the current feature element to be encoded needs to undergo entropy encoding. When the decision information output by the judgment module is not a preset value, it indicates that the current feature element to be encoded does not need to undergo entropy encoding. For example, when the optional values ​​of the identifier or identifier value are 0 and 1, the corresponding preset value is 0 or 1. When the identifier or identifier value can also have multiple optional values, the preset value is some specific values. For example, when the optional values ​​of the identifier or identifier value are 0~255, the preset value is a proper subset of 0~255.

[0288] High probability refers to: the current feature element to be encoded. The probability of taking the value k is very high, exceeding the threshold P, where P can be a number greater than 0.9, such as 0.9, 0.95, or 0.98.

[0289] Step 1605: The encoder sends or stores the compressed bitstream.

[0290] For feature maps At least one feature element in the process performs steps 1601 to 1604 as described above to obtain a compressed bitstream and transmits the compressed bitstream to the decoding end.

[0291] Decoding end: Step 1611: Obtain the compressed bitstream to be decoded Step 1612: Feature map to be decoded Probability estimation is performed to obtain the probability estimation results for each feature element. This step can be specifically described by FIG. 13B This is achieved through probability estimation 302, which can be referred to in the above description of probability estimation 40. Side information is obtained from the bitstream. Use the method in step 1603 to obtain the probability estimation result of the current feature element to be decoded. Step 1613: Obtain decision information and determine whether to perform entropy decoding based on the decision information.

[0292] This step can be specifically described by FIG. 13B The generation network 310 and decoding decision implementation 304 are used to implement this, and the specific implementation can be found in the above description of the generation network 46 and decoding decision implementation 30. Decision information 311 is obtained using the same method as the encoding end in this embodiment. This indicates the current feature element to be decoded at the corresponding position. Entropy decoding is required; entropy decoding is performed on the current feature element to be decoded based on the probability distribution. This indicates the current feature element to be decoded at the corresponding position. No entropy decoding is required, meaning it represents the corresponding position. For a specific value k.

[0293] In one possible implementation, the probability estimate or probability distribution of the current feature element to be decoded is input to a judgment module. The judgment module directly outputs decision information on whether the current feature element needs to undergo entropy decoding. For example, if the decision information output by the judgment module is a preset value, it indicates that the current feature element needs to undergo entropy decoding; if the decision information output by the judgment module is not a preset value, it indicates that the current feature element does not need to undergo entropy decoding, and the value of the current feature element is set to k. The judgment module can be implemented using a network method, that is, by inputting the probability estimate or probability distribution... FIG. 8 The generator network shown outputs decision information, i.e., preset values. This decision information indicates whether to perform entropy decoding on the current feature element to be decoded, and the decision information may include a decision graph (map).

[0294] Step 1614: Same as Step 1414.

[0295] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0296] FIG. 14 The specific implementation flow 1700 of Embodiment 4 of this application is shown, and the operation steps are as follows: Encoding end: Step 1701: Same as step 1501, this step can be specifically performed by... FIG. 3D The coding network 204 is used to implement this, and you can refer to the description of coding network 20 above for details; Step 1702: Same as step 1502, this step is specifically... FIG. 3D This is achieved by extracting edge information from 214; Step 1703: Obtain feature maps The probability estimation results and decision information for each feature element; This step can be specifically described by FIG. 3D This is implemented using joint network 218, which can be referred to in the above description of joint network 34. Specifically, the edge information... Inputting contextual information into the joint network, the joint network outputs the feature map to be encoded. Each feature element in The probability distribution and / or probability estimation results, and their relationship with the feature map. Decision information with the same dimension. For example, when simultaneously including edge information. In conjunction with contextual information input, a network structure such as... FIG. 15 As shown.

[0297] It should be noted that this embodiment does not impose any restrictions on the specific structure of the joint network.

[0298] It should be noted that decision information, probability distributions, and / or probability estimation results can all be output from different layers of the joint network. For example: Case 1) The middle layers of the network output decision information, and the last layer outputs probability distributions and / or probability estimation results; Case 2) The middle layers of the network output probability distributions and / or probability estimation results, and the last layer outputs decision information; Case 3) The last layer of the network outputs decision information and probability distributions and / or probability estimation results together.

[0299] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or Gaussian mixture model), the edge information is first... Alternatively, contextual information can be input into the joint network to obtain the values ​​of the model parameters, mean μ and variance σ, i.e., the probability estimation results. Further, these probability estimation results are input into the Gaussian model to obtain the probability distribution.

[0300] When using the Laplace distribution model for probability distribution, first consider the edge information. Alternatively, contextual information can be input into the joint network to obtain the values ​​of the model parameters, location parameter μ and scale parameter b, i.e., the probability estimation results. Further, these probability estimation results are input into the Laplace distribution model to obtain the probability distribution.

[0301] Alternatively, the edge information Inputting contextual information into the joint network yields the current feature element to be encoded. The probability distribution is then used to obtain the current feature element to be encoded. The probability P, which takes the value m, is the probability estimate. Here, m is any integer, such as 0, 1, -1, -2, 3, etc.

[0302] Step 1704 involves determining whether to perform entropy encoding based on decision information; depending on the determination result, entropy encoding is performed and written to the compressed bitstream (encoded bitstream), or entropy encoding is not performed. Entropy encoding is only performed on the current feature element to be encoded when it is determined that entropy encoding is required. This step can be specifically performed by… FIG. 3D The coding decision implementation 208 is implemented in the above-mentioned coding decision implementation 26.

[0303] Method 1: Decision information is related to feature maps Decision graphs with the same dimensions The preset value represents the current feature element to be encoded at the corresponding position. Entropy encoding is required; entropy encoding is performed on the current feature element to be encoded based on the probability distribution. A value not specified by the preset value indicates the current feature element to be encoded at the corresponding position. The high probability value is k. A value of 0 indicates the current feature element to be encoded at the corresponding position. Entropy encoding is not required, meaning the entropy encoding process is skipped. When the current feature element to be encoded in the decision graph (map)... When there are only two possible values, the preset value is a specific numerical value. For example, when the possible values ​​for the current feature element to be encoded are 0 and 1, the preset value is 0 or 1. When the current feature element to be encoded in the decision graph map... When there are multiple possible values, the preset values ​​are some specific numerical values, such as the current feature element to be encoded. When the element can be selected with a value of 0 to 255, the default value is a proper subset of 0 to 255.

[0304] Method 2: Decision information is related to feature maps Decision graphs with the same dimensions A value greater than or equal to the threshold T0 indicates the current feature element to be encoded at the corresponding position. Entropy encoding is required, which involves performing entropy encoding on the current feature element to be encoded based on the probability distribution. Less than the threshold T0 indicates the current feature element to be encoded at the corresponding position. The high probability value is k, which represents the current feature element to be encoded at the corresponding position. Entropy coding is not required, meaning the entropy coding process is skipped. Based on the decision... The range of values ​​in map, where T0 can be the mean of the range.

[0305] Method 3: The decision information can also be the identifier or identifier value directly output by the joint network. When the decision information is a preset value, it indicates that the current feature element to be encoded needs to undergo entropy encoding. When the decision information output by the judgment module is not a preset value, it indicates that the current feature element to be encoded does not need to undergo entropy encoding. When the current feature element to be encoded in the joint network output decision graph map has only two possible values, the preset value is a specific numerical value. For example, when the possible values ​​of the current feature element to be encoded are 0 and 1, the preset value is 0 or 1. When the current feature element to be encoded in the joint network output decision graph map has multiple possible values, the preset value is some specific numerical values. For example, when the possible values ​​of the current feature element to be encoded are 0~255, the preset value is a proper subset of 0~255.

[0306] High probability refers to: the current feature element to be encoded. The probability of taking the value m is very high. For example, the probability of taking the value k is greater than the threshold P, where P can be a number greater than 0.9, such as 0.9, 0.95, or 0.98.

[0307] Step 1705: The encoder sends or stores the compressed bitstream.

[0308] Decoding end: Step 1711: Obtain the bitstream of the feature map of the image to be decoded, and extract the side information from the bitstream. .

[0309] Step 1712: Obtain feature maps The probability estimation results and decision information for each feature element. This step can be specifically described by FIG. 16 This is implemented using joint network 312, which can be referred to in the description of joint network 34 above. Feature map acquisition. The probability estimation results and decision information for each feature element are obtained using the same method as in step 1703.

[0310] Step 1713: Determine whether to perform entropy decoding based on the decision information; decide whether to perform entropy decoding based on the determination result. This step can be specifically defined by... FIG. 16The decoding decision implementation 304 is implemented in the above description of the decoding decision implementation 30.

[0311] Method 1: The decision information is a decision map. The preset value represents the current feature element to be decoded at the corresponding position. Entropy decoding is required; entropy decoding is performed on the current feature element to be decoded based on the probability distribution. A value not specified by the default indicates the current feature element to be decoded at the corresponding position. No entropy decoding is required, meaning it represents the corresponding position. Set to a specific value k.

[0312] Method 2: Decision information is related to feature maps Same dimensions map A value greater than or equal to the threshold T0 indicates the current feature element to be decoded at the corresponding position. Entropy decoding is required. Less than the threshold T0 indicates the current feature element to be decoded at the corresponding position. The high probability value is k, which represents the current feature element to be decoded at the corresponding position. No entropy decoding is required, meaning it represents the corresponding position. Set to a specific value k. The value of T0 is the same as that of the encoding end.

[0313] Method 3: The decision information can also be the identifier or identifier value directly output by the joint network. When the decision information is a preset value, it indicates that the current feature element to be decoded needs to undergo entropy decoding. When the decision information output by the judgment module is not a preset value, it indicates that the current feature element to be decoded does not need to undergo entropy decoding, and the value of the current feature element to be decoded is set to k. When the current feature element to be decoded in the decision graph map output by the joint network has only two possible values, the preset value is a specific value. For example, when the possible values ​​of the current feature element to be decoded are 0 and 1, the preset value is 0 or 1. When the current feature element to be decoded in the decision graph map output by the joint network has multiple possible values, the preset value is some specific values. For example, when the possible values ​​of the current feature element to be decoded are 0~255, the preset value is a proper subset of 0~255.

[0314] Step 1714: Similar to step 1414, this step can be implemented by the decoding network unit 306 in the decoder 9C of the above embodiment. For details, please refer to the description of the decoding network unit 306 in the above embodiment.

[0315] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0316] FIG. 17The specific implementation flow 1800 of Embodiment 5 of this application is shown, and the operation steps are as follows: Step 1801: Obtain the feature variables of the audio data to be encoded The audio signal to be encoded can be a time-domain audio signal; the audio signal to be encoded can be a frequency-domain signal obtained by time-frequency transformation of a time-domain signal, such as a frequency-domain signal obtained by MDCT transformation of a time-domain audio signal, or a frequency-domain signal obtained by FFT transformation of a time-domain audio signal; the signal to be encoded can also be a signal after QMF filtering; the signal to be encoded can also be a residual signal, such as a residual signal after other encoding or a residual signal after LPC filtering.

[0317] Obtaining feature variables of the audio data to be encoded: This can be done by extracting feature vectors from the audio signal to be encoded, such as extracting Mel-frequency cepstral coefficients from the audio signal to be encoded; quantizing the extracted feature vectors and using the quantized feature vectors as feature variables of the audio data to be encoded.

[0318] Obtaining feature variables from the audio data to be encoded: This can also be achieved using existing neural networks. For example, the audio signal to be encoded can be processed through an encoding neural network to obtain latent variables. The latent variables output by the neural network can then be quantized, and the quantized latent variables can be used as feature variables of the audio data to be encoded. The encoding neural network is pre-trained, and this invention does not limit the specific network structure or training method of the encoding neural network. For example, the encoding neural network can be a fully connected network or a CNN network. This invention also does not limit the number of layers in the encoding neural network or the number of nodes in each layer.

[0319] The latent variables output by different coding neural networks may have different forms. For example, a fully connected coding neural network outputs a vector as a latent variable, where the dimension M of the vector is the latent size, such as y = [y(0), y(1), ..., y(M-1)]. A CNN (Neural Network Conversion) network outputs an N*M dimensional matrix as a latent variable, where N is the number of channels in the CNN network and M is the latent size of each channel, such as...

[0320] A specific method for quantizing the latent variables output by a neural network can be to perform scalar quantization on each element of the latent variable. The quantization step size can be determined according to different encoding rates. Scalar quantization can also involve a bias; for example, the latent variable to be quantized may be biased before scalar quantization according to a predetermined quantization step size. Other existing quantization techniques can also be used to quantize latent variables, and this invention is not limited to any particular method.

[0321] The quantized feature vector or the quantized latent variable can both be denoted as , that is, the characteristic variables of the audio data to be encoded.

[0322] Step 1802: Feature variables of the audio data to be encoded Input edge information extraction module, output edge information .

[0323] Among them, the edge information extraction module can use FIG. 12 The network shown is used to implement edge information. This can be understood as the characteristic variable Further extraction of feature variables , The number of feature elements contained is greater than the number of feature variables. few.

[0324] It should be noted that edge information can be obtained in this step. Entropy encoding is performed and written to the bitstream; edge information can also be processed in subsequent step 1804. Entropy encoding is performed and written to the bitstream; no specific restrictions are imposed here.

[0325] Step 1803: For characteristic variables Probability estimation is performed to obtain the probability estimation results for each feature element. Probability distribution models can be used to obtain probability estimates and probability distributions. These models can be: Gaussian single model (GSM), asymmetric Gaussian model, Gaussian mixture model (GMM), or Laplace distribution.

[0326] The following uses characteristic variables Let's take an N*M dimensional matrix as an example for explanation. The current feature variable to be encoded... The characteristic elements in are denoted as , , .

[0327] When the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or Gaussian mixture model), the edge information is first... Alternatively, contextual information can be input into a probability estimation network for feature variables. Each feature element in Probability estimation is performed to obtain the values ​​of the mean parameter μ and the variance σ. Further, the mean parameter μ and the variance σ are input into the probability distribution model used to obtain the probability distribution. At this point, the probability estimation result is the mean parameter μ and the variance σ.

[0328] Variance can also be estimated, for example, when the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or Gaussian mixture model), first, the edge information... Alternatively, contextual information can be input into a probability estimation network for feature variables. Each feature element in The variance σ is obtained by performing probability estimation. Further, the variance σ is input into the probability distribution model used to obtain the probability distribution. At this point, the probability estimation result is the variance σ.

[0329] When using the Laplace distribution model for probability distribution, first consider the edge information. Alternatively, contextual information can be input into a probability estimation network to evaluate feature map variables. Each feature element in Probability estimation is performed to obtain the values ​​of the position parameter μ and the scale parameter b. Further, the position parameter μ and the scale parameter b are input into the used probability distribution model to obtain the probability distribution. At this point, the probability estimation result is the position parameter μ and the scale parameter b.

[0330] You can also add edge information And / or contextual information is input into the probability estimation network to encode the feature map. Each feature element in Probability estimation is performed to obtain the current feature element to be encoded. The probability distribution is then used to obtain the current feature element to be encoded. The probability P is the value of m. The probability estimate at this point is the current feature element to be encoded. The probability P of taking the value m.

[0331] The probability estimation network can use deep learning-based networks, such as recurrent neural networks (RNNs) and convolutional neural networks (PixelCNNs), without any limitation.

[0332] Step 1804: Determine whether the current feature element to be encoded needs to be entropy encoded based on the probability estimation result, and perform entropy encoding to write it into the compressed bitstream (encoded bitstream) or not perform entropy encoding based on the determination result.

[0333] The current feature element to be encoded is determined based on the probability estimation result. One or more of the following methods can be used to determine whether entropy encoding needs to be performed. Here, the parameters j and i are positive integers, and the coordinates (j, i) represent the position of the currently encoded feature element. Alternatively, based on the probability estimation result, it is determined whether the currently encoded feature element One or more of the following methods can be used to determine whether entropy encoding needs to be performed. Here, the parameter i is a positive integer, and the coordinate i represents the position of the currently encoded feature element.

[0334] The following uses the example of determining whether the currently encoded feature element needs to perform entropy encoding to illustrate. The method of determining whether the currently encoded feature element needs to perform entropy encoding is similar and will not be elaborated here.

[0335] Method 1: When the probability distribution model is a Gaussian distribution, based on the probability estimation result of the first special element, it is determined whether to perform entropy encoding on the currently encoded feature element. When the values of the mean parameter μ and variance σ of the Gaussian distribution of the currently encoded feature element satisfy the second condition: when the absolute value of the difference between the mean μ and k is less than the second threshold T1 and the variance σ is less than the third threshold T2, there is no need to perform entropy encoding on the currently encoded feature element during the entropy encoding process. Otherwise, when the first condition is satisfied: when the absolute value of the difference between the mean μ and k is greater than or equal to the second threshold T1 or the variance σ is less than the third threshold T2, entropy encoding is performed on the currently encoded feature element and written into the code stream. Here, k is an arbitrary integer, such as 0, 1, -1, 2, 3, etc. T2 takes any value that satisfies 0 < T2 < 1, such as 0.2, 0.3, 0.4, etc. T1 is a number greater than or equal to 0 and less than 1, such as 0.01, 0.02, 0.001, 0.002.

[0336] Specifically, when k takes the value of 0, it is the optimal value. It can be directly determined that when the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, the entropy encoding process for the currently encoded feature element is skipped Otherwise, entropy encoding is performed on the currently encoded feature element and written into the code stream. Here, the value of T2 takes any value that satisfies <0 < T2 < 1, such as 0.2, 0.3, 0.4, etc. T1 is a number greater than or equal to 0 and less than 1, such as 0.01, 0.02, 0.001, 0.002.

[0337] Method 2: When the probability distribution is a Gaussian distribution, based on the probability estimation result, the values of the mean parameter μ and variance σ of the Gaussian distribution of the currently encoded feature element are obtained. When the relationship between the mean μ, variance σ and k satisfies the When (second condition), skip the current feature element to be encoded. The entropy encoding process is performed, where abs(μ-k) represents the absolute value of the difference between the mean μ and k; otherwise, when the probability estimate of the current feature element to be encoded satisfies... Time (first condition), for the current feature element to be encoded Entropy encoding is performed and written into the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. The fourth threshold T3 is a number greater than or equal to 0 and less than 1, for example, values ​​of 0.2, 0.3, 0.4, etc.

[0338] When the probability distribution is Gaussian, if the feature variable Each feature element in Probability estimation only yields the current feature element to be encoded. The value of the variance σ of the Gaussian distribution, when the variance σ satisfies When (second condition), skip the current feature element to be encoded. Perform entropy encoding; otherwise, if the probability estimate of the current feature element to be encoded satisfies... Time (first condition), for the current feature element to be encoded Entropy encoding is performed and written into the bitstream. The fourth threshold T3 is a number greater than or equal to 0 and less than 1, such as 0.2, 0.3, 0.4, etc.

[0339] Method 3: When the probability distribution is a Laplace distribution, the current feature element to be encoded is obtained based on the probability estimation result. The values ​​of the location parameter μ and scale parameter b of the Laplace distribution. When the relationship between the location parameter μ, scale parameter b and k satisfies... When the second condition is met, skip the current feature element to be encoded. The entropy encoding process is performed, where abs(μ-k) represents the absolute value of the difference between the position parameters μ and k; otherwise, when the probability estimate of the current feature element to be encoded satisfies... (First condition) For the current feature element to be encoded Entropy encoding is performed and written to the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. The fourth threshold T4 is a number greater than or equal to 0 and less than 0.5, for example, values ​​such as 0.05, 0.09, 0.17, etc.

[0340] Method 4: When the probability distribution is a Laplace distribution, the current feature element to be encoded is obtained based on the probability estimation result. The values ​​of the location parameter μ and scale parameter b of the Laplace distribution. When the absolute value of the difference between the location parameter μ and k is less than the second threshold T5 and the scale parameter b is less than the third threshold T6 (second condition), skip the current feature element to be encoded. Perform entropy encoding; otherwise, if the absolute value of the difference between the position parameter μ and k is less than the second threshold T5 or the scale parameter b is greater than or equal to the third threshold T6 (first condition), then the current feature element to be encoded is... Entropy encoding is performed and written to the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. T5 takes the value 1e-2, and T6 takes any number satisfying T6 < 0.5, such as 0.05, 0.09, 0.17, etc.

[0341] Specifically, when k is 0, it is the optimal value. It can be directly determined that if the absolute value of the position parameter μ is less than T5 and the scale parameter b is less than T6, then the current feature element to be encoded is skipped. Perform entropy encoding; otherwise, for the current feature element to be encoded... Entropy encoding is performed and written into the bitstream. The threshold T5 is 1e-2, and T2 is any number that satisfies T6 < 0.5, such as 0.05, 0.09, 0.17, etc.

[0342] Method 5: When the probability distribution is a Gaussian mixture distribution, the current feature element to be encoded is obtained based on the probability estimation result. All mean parameters μ of the mixture Gaussian distribution i and variance σ i The value of k. When the sum of the absolute values ​​of the differences between all the means of the Gaussian mixture distribution and k, and the sum of any variance of the Gaussian mixture distribution, is less than the fifth threshold T7 (second condition), skip the current feature element to be encoded. Perform entropy encoding; otherwise, if the sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution and k, and the sum of any variance of the Gaussian mixture distribution, are greater than or equal to the fifth threshold T7 (first condition), then for the current feature element to be encoded... Entropy encoding is performed and written to the bitstream. Here, k is any integer, such as 0, 1, -1, -2, 3, etc. T7 is a number greater than or equal to 0 and less than 1, for example, values ​​such as 0.2, 0.3, 0.4, etc. (It can be assumed that the threshold for each feature element is the same). Method 6: Obtain the current feature element to be encoded based on the probability distribution. Let P be a probability of value k. If the probability estimate P of the current feature element to be encoded satisfies the second condition: when P is greater than (or equal to) the first threshold T0, skip the entropy encoding process for the current feature element; otherwise, if the probability estimate P of the current feature element to be encoded satisfies the first condition: when P is less than the first threshold T0, perform entropy encoding on the current feature element and write it into the bitstream. Here, k can be any integer, such as 0, 1, -1, 2, 3, etc. The first threshold T0 is... Any number in the set, for example, values ​​such as 0.99, 0.98, 0.97, 0.95, etc. (It can be assumed that the threshold for each feature element is the same). It should be noted that, in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 can be rounded up to integers.

[0343] It should be noted that the threshold can also be obtained using one of the following methods, which are not limited here: Method 1: Taking threshold T1 as an example, any value within the range of T1 is taken as threshold T1, and threshold T1 is written into the bitstream. Specifically, the threshold can be written into the bitstream and stored in the sequence header, image header, slice / band, or SEI before being transmitted to the decoding end. Other methods can also be used, which are not limited here. Similar methods can be used for the other thresholds T0, T2, T3, T4, T5, and T6.

[0344] Method 2: The encoding end uses a fixed threshold agreed upon with the decoding end, eliminating the need to write to the bitstream or transmit to the decoding end. For example, taking threshold T1 as an example, any value within the range of T1 can be directly used as the value of T1. Similar methods can be used for the other thresholds T0, T2, T3, T4, T5, and T6.

[0345] Method 3: Construct a threshold candidate list. The most likely values ​​within the range of T1 are placed in the threshold candidate list, with each threshold corresponding to a threshold index number. An optimal threshold is determined, and this optimal threshold is used as the value of T1. The index number of the optimal threshold is used as the threshold index number of T1, and the threshold index number of T1 is written into the bitstream. Specifically, the threshold is written into the bitstream and can be stored in the sequence header, image header, slice / strip, or SEI before being transmitted to the decoding end. Other methods can also be used, which are not limited here. Similar methods can be used for the remaining thresholds T0, T2, T3, T4, T5, and T6.

[0346] Step 1805: The encoder sends or stores the compressed bitstream.

[0347] Decoding end: Step 1811: Obtain the bitstream of the audio feature variables to be decoded. Step 1812: Obtain the probability estimation results for each feature element. Edge information Entropy decoding is performed to obtain edge information. Combined with edge information Audio feature variables to be decoded Each feature element in Perform probability estimation to obtain the current feature element to be decoded. The probability estimation result. Here, parameters j and i are positive integers, and the coordinates (j, i) represent the position of the current feature element to be decoded. Alternatively, edge information. Entropy decoding is performed to obtain edge information. Combined with edge information Audio feature variables to be decoded Each feature element in Perform probability estimation to obtain the current feature element to be decoded. The probability estimation result is given. Here, parameter i is a positive integer, and coordinate i represents the position of the current feature element to be decoded.

[0348] It should be noted that the probability estimation method used at the decoding end is the same as the probability estimation method at the encoding end in this embodiment, and the probability estimation network structure diagram is the same as the probability estimation network structure at the encoding end in this embodiment, so it will not be described again here.

[0349] Step 1813: Based on the probability estimation results, determine whether the current feature element to be decoded needs to undergo entropy decoding, and perform or not perform entropy decoding based on the determination result to obtain the decoded feature variables. .

[0350] Determine the current feature element to be decoded based on the probability estimation result. Whether entropy decoding needs to be performed can be determined using one or more of the following methods. Alternatively, the current feature element to be decoded can be determined based on the probability estimation result. Whether entropy decoding needs to be performed can be determined using one or more of the following methods.

[0351] The following section determines the current feature element to be decoded based on the probability estimation results. Taking the need for entropy decoding as an example, we will determine the current feature element to be decoded. The method for determining whether entropy decoding is required is similar and will not be elaborated here.

[0352] Method 1: When the probability distribution model is a Gaussian distribution, the current feature element to be decoded is obtained based on the probability estimation result. The values of the mean parameter μ and variance σ. When the absolute value of the difference between the mean μ and k is less than the second threshold T1 and the variance σ is less than the third threshold T2 (second condition), set the value of the current feature element to be decoded to k and skip the entropy decoding process for the current feature element to be decoded ; otherwise, when the absolute value of the difference between the mean μ and k is less than the second threshold T1 or the variance σ is greater than or equal to the third threshold T2 (first condition), perform entropy decoding on the current feature element to be decoded to obtain the value of the current feature element to be decoded .

[0353] Specifically, when k takes the value of 0, it is the optimal value. It can be directly determined that when the absolute value of the mean parameter μ of the Gaussian distribution is less than T1 and the variance σ of the Gaussian distribution is less than T2, set the value of the current feature element to be decoded to k, then skip the entropy decoding process for the current feature element to be decoded ; otherwise, perform entropy decoding on the current feature element to be decoded to obtain the value of the current feature element to be decoded .

[0354] Method 2: When the probability distribution is a Gaussian distribution, based on the probability estimation result, obtain the values of the mean parameter μ and variance σ of the current feature element to be decoded . When the relationship between the mean μ, variance σ and k satisfies abs(μ - k)+σ < T3 (second condition), where T3 is the fourth threshold, set the value of the current feature element to be decoded to k and skip the entropy decoding process for the current feature element to be decoded ; otherwise, when the probability estimation result of the current feature element to be decoded satisfies , perform entropy decoding on the current feature element to be decoded to obtain the value of the current feature element to be decoded . When the probability distribution is a Gaussian distribution, if only the variance σ value of the current feature element to be decoded is obtained based on the probability estimation result, when the variance σ relationship satisfies σ < T3 (second condition), where T3 is the fourth threshold, set the value of the current feature element to be decoded to 0 and skip the entropy decoding process for the current feature element to be decoded ; otherwise, when the probability estimation result of the current feature element to be decoded satisfies , perform entropy decoding on the current feature element to be decoded to obtain the value of the current feature element to be decoded .

[0355] Method 3: When the probability distribution is a Laplace distribution, values of the location parameter μ and the scale parameter b are obtained according to the probability estimation result. When the relationship between the location parameter μ, the scale parameter b, and k satisfies abs(μ - k) + σ < T4 (the second condition), where T4 is the fourth threshold, the value of the current feature element to be decoded is set to k, and the entropy decoding process for the feature element is skipped. Otherwise, when the probability estimation result of the current feature element to be decoded satisfies (the first condition), the feature element is entropy decoded to obtain the value of the feature element .

[0356] Method 4: When the probability distribution is a Laplace distribution, values of the location parameter μ and the scale parameter b are obtained according to the probability estimation result. When the absolute value of the difference between the location parameter μ and k is less than the second threshold T5 and the scale parameter b is less than the third threshold T6 (the second condition), the value of the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded is skipped. Otherwise, when the absolute value of the difference between the location parameter μ and k is less than the second threshold T5 or the scale parameter b is greater than or equal to the third threshold T6 (the first condition), the current feature element to be decoded is entropy decoded to obtain the value of the current feature element to be decoded .

[0357] Specifically, when k takes the value of 0, it is the optimal value. It can be directly determined that when the absolute value of the location parameter μ is less than T5 and the scale parameter b is less than T6, the value of the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded is skipped. Otherwise, the current feature element to be decoded is entropy decoded to obtain the value of the current feature element to be decoded .

[0358] Method 5: When the probability distribution is a mixture of Gaussian distributions, all mean parameters μ of the mixture of Gaussian distributions of the current feature element to be decoded and variances σ i and i values are obtained according to the probability estimation result. When the sum of the absolute values of the differences between all means of the mixture of Gaussian distributions and k and the sum of any variance of the mixture of Gaussian distributions is less than the fifth threshold T7 (the second condition), the value of the current feature element to be decoded is set to k, and the entropy decoding process for the current feature element to be decoded Perform entropy decoding; otherwise, if the sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution and k, and the sum of any variance of the Gaussian mixture distribution, are greater than or equal to the fifth threshold T7 (first condition), then for the current feature element to be decoded... Perform entropy decoding to obtain the current feature element to be decoded. The value of .

[0359] Method 6: Based on the probability distribution of the current feature element to be decoded, obtain the probability P of the current feature element to be decoded taking the value k, that is, the probability estimation result P of the current feature element to be decoded. When the probability estimation result P satisfies the second condition: P is greater than the first threshold T0, there is no need to perform entropy decoding on the first feature element, and the value of the current feature element to be decoded is set to k. Otherwise, when the current feature element to be decoded satisfies the first condition: P is less than or equal to the first threshold T0, entropy decoding is performed on the bitstream to obtain the value of the first feature element.

[0360] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0361] The methods for obtaining thresholds T0, T1, T2, T3, T4, T5, T6, and T7 correspond to those used at the encoding end, and one of the following methods can be used: Method 1: Obtain the threshold from the bitstream, specifically from the sequence header, image header, slice / band, or SEI.

[0362] Method 2: The decoding end uses a fixed threshold agreed upon with the encoding end.

[0363] Method 3: Obtain the threshold index number from the bitstream. Specifically, obtain the threshold index number from the sequence header, image header, slice / band, or SEI. Then, the decoder constructs a threshold candidate list in the same way as the encoder, and obtains the corresponding threshold from the threshold candidate list based on the threshold index number.

[0364] It should be noted that, in practical applications, to ensure platform consistency, the thresholds T1, T2, T3, T4, T5, and T6 can be rounded up to integers.

[0365] Step 1814: Decode the feature variables Perform reconstruction, or input the corresponding machine task into the machine-oriented auditory task module to execute the task. This step can be specifically performed by... FIG. 10B The decoding network 306 is used to implement this, and you can refer to the description of the decoding network 34 above for details.

[0366] Scenario 1: Feature variables after entropy decoding The input image reconstruction module outputs reconstructed audio from the neural network. The neural network can employ any structure, such as a fully connected network, convolutional neural network, or recurrent neural network. A multi-layered deep neural network structure can be used to achieve better estimation results.

[0367] Scenario 2: Feature variables after entropy decoding The input is directed to the machine hearing task module to execute the corresponding machine task. For example, it can complete machine hearing tasks such as audio classification and recognition.

[0368] The k-values ​​at the decoding end and the encoding end are set accordingly.

[0369] FIG. 18 This is an exemplary structural diagram of the encoding device of this application, as shown below. FIG. 18 As shown, the apparatus in this example can correspond to encoder 20A. This apparatus may include an acquisition module 2001 and an encoding module 2002. The acquisition module 2001 may include the encoding network 204, rounding 206 (optional), probability estimation 210, edge information extraction 214, generator network 216 (optional), and joint network 218 (optional) from the aforementioned embodiments. The encoding module 2002 includes the encoding decision implementation 208 from the aforementioned embodiments. The acquisition module 2001 is used to acquire feature data to be encoded, the feature data to be encoded including multiple feature elements, the multiple feature elements including a first feature element, and to acquire the probability estimation result of the first feature element; the encoding module 2002 is used to determine whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element; and to perform entropy encoding on the first feature element only when it is determined that entropy encoding needs to be performed on the first feature element.

[0370] In one possible implementation, determining whether to perform entropy encoding on the first feature element of the feature data includes: when the probability estimation result of the first feature element of the feature data meets a preset condition, entropy encoding of the first feature element of the feature data is required; when the probability estimation result of the first feature element of the feature data does not meet the preset condition, entropy encoding of the first feature element of the feature data is not required.

[0371] In one possible implementation, the encoding module is further configured to determine, based on the probability estimation result of the feature data: the probability estimation result of the feature data is input into the generator network, and the network outputs decision information. When the decision information of the first feature element is 1, the first feature element of the feature data needs to be encoded; when the decision information of the first feature element is not 1, the first feature element of the feature data does not need to be encoded.

[0372] In one possible implementation, the preset condition is that the probability value of the first feature element being k is less than or equal to a first threshold, where k is an integer.

[0373] In one possible implementation, the preset condition is that the absolute value of the difference between the mean of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a second threshold or the variance of the first feature element is greater than or equal to a third threshold, where k is an integer.

[0374] In another possible implementation, the preset condition is that the sum of the absolute value of the difference between the mean of the probability distribution of the first feature element and the value k of the first feature element and the variance of the probability distribution of the first feature element is greater than or equal to a fourth threshold, where k is an integer.

[0375] In one possible implementation, the probability value of the first feature element being k is the maximum probability value among all possible values ​​of the first feature element.

[0376] In one possible implementation, probability estimation is performed on the feature data to obtain probability estimation results for each feature element in the feature data, wherein the probability estimation result for the first feature element includes the probability value of the first feature element, and / or the first parameter and the second parameter of the probability distribution.

[0377] In one possible implementation, the probability estimation result of the feature data is input into a generator network to obtain decision information for the first feature element. Based on the decision information of the first feature element, it is determined whether to perform entropy encoding on the first feature element.

[0378] In one possible implementation, when the decision information of the feature data is a decision graph, if the value corresponding to the position of the first feature element in the decision graph is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; if the value corresponding to the position of the first feature element in the decision graph is not a preset value, it is determined that entropy encoding does not need to be performed on the first feature element.

[0379] In one possible implementation, when the decision information of the feature data is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; when the decision information is not a preset value, it is determined that entropy encoding does not need to be performed on the first feature element. In another possible implementation, the encoding module is further configured to construct a threshold candidate list for a first threshold, place the first threshold in the threshold candidate list and assign it an index number, and write the index number of the first threshold into the encoded bitstream, wherein the length of the threshold candidate list for the first threshold can be set to T; T is an integer greater than or equal to 1.

[0380] The device in this embodiment can be used for FIGS. 3A-3D The technical solution implemented by the encoder in the method embodiment shown has a similar implementation principle and technical effect, and will not be described again here.

[0381] FIG. 19 This is an exemplary structural diagram of the decoding device of this application, as shown below. FIG. 19 As shown, the apparatus in this example can correspond to decoder 30. This apparatus may include an acquisition module 2101 and a decoding module 2102. The acquisition module 2101 may include the probability estimation 302, the generator network 310 (optional), and the joint network 312 as described in the foregoing embodiments. The decoding module 2102 includes the decoding decision implementation 304 and the decoding network 306 as described in the foregoing embodiments. The acquisition module 2101 is used to acquire the bitstream of the feature data to be decoded, the feature data to be decoded includes multiple feature elements, the multiple feature elements include a first feature element; acquire the probability estimation result of the first feature element; the decoding module 2102 is used to determine whether to perform entropy decoding on the first feature element based on the probability estimation result of the first feature element; and perform entropy decoding on the first feature element only when it is determined that entropy decoding is required on the first feature element.

[0382] In one possible implementation, determining whether to entropy decode the first feature element of the feature data includes: if the probability estimation result of the first feature element of the feature data meets a preset condition, the first feature element of the feature data needs to be decoded; or if the probability estimation result of the first feature element of the feature data does not meet the preset condition, the first feature element of the feature data does not need to be decoded, and the feature value of the first feature element is set to k; where k is an integer.

[0383] In one possible implementation, the decoding module is further configured to determine, based on the probability estimation result of the feature data: the probability estimation result of the feature data is input into a judgment network module, and the network outputs decision information. When the value of the position corresponding to the first feature element of the feature data in the decision information is 1, the first feature element of the feature data is decoded; when the value of the position corresponding to the first feature element of the feature data in the decision information is not 1, the first feature element of the feature data is not decoded, and the feature value of the first feature element is set to k, where k is an integer.

[0384] In one possible implementation, the preset condition is that the probability value of the first feature element being k is less than or equal to a first threshold, where k is an integer.

[0385] In another possible implementation, the preset condition is that the absolute value of the difference between the mean of the probability distribution of the first feature element and the value k of the first feature element is greater than or equal to a second threshold, or the variance of the probability distribution of the first feature element is greater than or equal to a third threshold.

[0386] In another possible implementation, the preset condition is that the sum of the absolute value of the difference between the mean of the probability distribution of the first feature element and the value k of the first feature element and the variance of the probability distribution of the first feature element is greater than or equal to a fourth threshold.

[0387] In one possible implementation, probability estimation is performed on the feature data to obtain probability estimation results for each feature element in the feature data, wherein the probability estimation result for the first feature element includes the probability value of the first feature element, and / or the first parameter and the second parameter of the probability distribution. In one possible implementation, the probability value of the first feature element being k is the maximum probability value among all possible values ​​of the first feature element.

[0388] In one possible implementation, the probability estimation result of the Nth feature element includes at least one of the following: the probability value of the Nth feature element, a first parameter of the probability distribution, a second parameter of the probability distribution, and decision information. When the value corresponding to the position of the first feature element in the decision information is 1, the first feature element of the feature data is decoded; when the value corresponding to the position of the first feature element in the decision information is not 1, the first feature element of the feature data is not decoded, and the feature value of the first feature element is set to k, where k is an integer.

[0389] In one possible implementation, the probability estimation result of the feature data is input into a generator network to obtain the decision information of the first feature element; when the value of the decision information of the first feature element is a preset value, it is determined that entropy decoding needs to be performed on the first feature element; when the value of the decision information of the first feature element is not a preset value, it is determined that entropy decoding does not need to be performed on the first feature element, and the feature value of the first feature element is set to k, where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

[0390] In one possible implementation, the obtaining module is further configured to construct a threshold candidate list for the first threshold, obtain the index number of the threshold candidate list for the first threshold by decoding the bitstream, and take the value of the position of the threshold candidate list for the first threshold corresponding to the index number of the first threshold as the value of the first threshold, wherein the length of the threshold candidate list for the first threshold can be set to T; T is an integer greater than or equal to 1.

[0391] The device in this embodiment can be used for FIG. 10B , FIG. 13B , FIG. 16 The technical solution implemented by the decoder in the method embodiment shown has a similar implementation principle and technical effect, and will not be described again here.

[0392] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and determined by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0393] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0394] Instructions can be determined by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0395] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus used to determine the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (containing one or more processors as described above).

[0396] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for encoding feature data, characterized in that, include: Obtain feature data to be encoded, which includes multiple feature elements of image feature maps or audio feature variables; For the first feature element among the plurality of feature elements, execute: Determine whether entropy encoding needs to be performed on the first feature element; When it is determined that entropy encoding needs to be performed on the first feature element, entropy encoding is performed on the first feature element, and the entropy encoding result is written into the bitstream; Alternatively, if it is determined that entropy encoding of the first feature element is not required, the entropy encoding of the first feature element is skipped, and no data related to the first feature element is written to the bitstream.

2. The method according to claim 1, characterized in that, The determination of whether entropy encoding needs to be performed on the first feature element includes: Obtain the probability estimation result of the first feature element; Based on the probability estimation result of the first feature element, determine whether to perform entropy encoding on the first feature element.

3. The method according to claim 2, characterized in that, The step of determining whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element includes: When the probability estimation result of the first feature element meets a preset condition, it is determined that entropy encoding needs to be performed on the first feature element; or When the probability estimation result of the first feature element does not meet the preset conditions, it is determined that entropy encoding is not required for the first feature element.

4. The method according to claim 3, characterized in that, The probability estimation result of the first feature element is the probability value of the first feature element taking the value k. The preset condition is that the probability value of the first feature element taking the value k is less than or equal to a first threshold, where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

5. The method according to claim 3, characterized in that, The probability estimation result of the first feature element includes parameters of the probability distribution of the first feature element, and the preset condition is: The parameter is greater than or equal to the second threshold.

6. The method according to claim 5, characterized in that: When the probability distribution is a Gaussian distribution, the parameter is the mean or variance of the Gaussian distribution; or When the probability distribution is a Laplace distribution, the parameter is the location parameter of the Laplace distribution or the scale parameter of the Laplace distribution.

7. The method according to claim 4, characterized in that, The method further includes: A threshold candidate list is constructed, the first threshold is placed in the threshold candidate list, and the index number corresponding to the first threshold is written into the encoded bitstream, wherein the length of the threshold candidate list is T, and T is an integer greater than or equal to 1.

8. The method according to claim 3, characterized in that, The probability estimation result of the first feature element is obtained through a Gaussian mixture distribution, and the preset condition is: The sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution of the first feature element and the values ​​of the first feature element with the value k, and the sum of any variance of the Gaussian mixture distribution of the first feature element, is greater than or equal to the fifth threshold; or The difference between any mean of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is greater than or equal to the sixth threshold; or The variance of any Gaussian mixture distribution of the first feature element is greater than or equal to the seventh threshold. Where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

9. The method according to claim 3, characterized in that, The probability estimation result of the first feature element is obtained through an asymmetric Gaussian distribution, and the preset condition is: The absolute value of the difference between the mean of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is greater than or equal to the eighth threshold; or The first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the ninth threshold; or The second variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the tenth threshold. Where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

10. The method according to any one of claims 4-9, characterized in that: The probability value of the first feature element being k is the maximum probability value among all the candidate values ​​of the first feature element.

11. The method according to claim 2, characterized in that, The step of determining whether to perform entropy encoding on the first feature element based on the probability estimation result of the first feature element includes: The probability estimation results of the feature data are input into the generator network to obtain the decision information of the first feature element. Based on the decision information of the first feature element, determine whether to perform entropy encoding on the first feature element.

12. The method according to claim 11, characterized in that, The decision information of the feature data is a decision graph. The step of determining whether to perform entropy encoding on the first feature element based on the decision information of the first feature element includes: When the value corresponding to the location of the first feature element in the decision graph is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; or When the value corresponding to the location of the first feature element in the decision graph is not the preset value, it is determined that entropy encoding is not required for the first feature element.

13. The method according to claim 11, characterized in that, The step of determining whether to perform entropy encoding on the first feature element based on the decision information of the first feature element includes: When the decision information of the feature data is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; or... When the decision information is not a preset value, it is determined that entropy encoding is not required for the first feature element.

14. The method according to claim 1, characterized in that, The determination of whether entropy encoding needs to be performed on the first feature element includes: Obtain the side information of the feature data, and input the side information of the feature data into the joint network to obtain the decision information of the first feature element; Based on the decision information of the first feature element, determine whether to perform entropy encoding on the first feature element.

15. The method according to claim 14, characterized in that, The decision information of the feature data is a decision graph. The step of determining whether to perform entropy encoding on the first feature element based on the decision information of the first feature element includes: When the value corresponding to the location of the first feature element in the decision graph is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; or When the value corresponding to the position of the first feature element in the decision graph is not a preset value, it is determined that entropy encoding is not required for the first feature element.

16. The method according to claim 14, characterized in that, The step of determining whether to perform entropy encoding on the first feature element based on the decision information of the first feature element includes: When the decision information is a preset value, it is determined that entropy encoding needs to be performed on the first feature element; or When the decision information is not a preset value, it is determined that entropy encoding is not required for the first feature element.

17. A method for decoding feature data, characterized in that, include: Obtain feature data to be decoded, wherein the feature data to be encoded includes multiple feature elements of an image feature map or an audio feature variable; For the first feature element among the plurality of feature elements, execute: Determine whether entropy decoding needs to be performed on the first feature element; When it is determined that entropy decoding needs to be performed on the first feature element, entropy decoding is performed on the bitstream, and the value of the first feature element is set to the result of the entropy decoding; or, when it is determined that entropy decoding does not need to be performed on the first feature element, entropy decoding is skipped for the first feature element, and the value of the first feature element is set to k, where k is an integer and is one of multiple candidate values ​​among the first feature elements.

18. The method according to claim 17, characterized in that, The determination of whether entropy decoding needs to be performed on the first feature element includes: Obtain the probability estimation result of the first feature element; Based on the probability estimation result of the first feature element, determine whether to perform entropy decoding on the first feature element.

19. The method according to claim 18, characterized in that, The step of determining whether entropy decoding needs to be performed on the first feature element based on the probability estimation result of the first feature element includes: When the probability estimation result of the first feature element meets a preset condition, it is determined that entropy decoding needs to be performed on the first feature element; or When the probability estimation result of the first feature element does not meet the preset conditions, it is determined that entropy decoding is not required for the first feature element.

20. The method according to claim 19, characterized in that, The probability estimation result of the first feature element is the probability value of the first feature element taking the value k. The preset condition is that the probability value of the first feature element taking the value k is less than or equal to a first threshold, where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

21. The method according to claim 19, characterized in that, The probability estimation result of the first feature element includes parameters of the probability distribution of the first feature element, and the preset condition is: The parameter is greater than or equal to the second threshold.

22. The method according to claim 21, characterized in that: When the probability distribution is a Gaussian distribution, the parameter is the mean or variance of the Gaussian distribution; or When the probability distribution is a Laplace distribution, the parameter is the location parameter of the Laplace distribution or the scale parameter of the Laplace distribution.

23. The method according to claim 19, characterized in that, The probability estimation result of the first feature element is obtained through a Gaussian mixture distribution, and the preset condition is: The sum of the absolute values ​​of the differences between all means of the Gaussian mixture distribution of the first feature element and the values ​​of the first feature element with the value k, and the sum of any variance of the Gaussian mixture distribution of the first feature element, is greater than or equal to the fifth threshold; or The difference between any mean of the Gaussian mixture distribution of the first feature element and the value k of the first feature element is greater than or equal to the sixth threshold; or The variance of any Gaussian mixture distribution of the first feature element is greater than or equal to the seventh threshold. Where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

24. The method according to claim 19, characterized in that, The probability estimation result of the first feature element is obtained through an asymmetric Gaussian distribution, and the preset condition is: The absolute value of the difference between the mean of the asymmetric Gaussian distribution of the first feature element and the value k of the first feature element is greater than or equal to the eighth threshold; or The first variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the ninth threshold; or The second variance of the asymmetric Gaussian distribution of the first feature element is greater than or equal to the tenth threshold. Where k is an integer and k is one of the multiple candidate values ​​of the first feature element.

25. The method according to any one of claims 19-24, characterized in that, The probability value of the first feature element being k is the maximum probability value among all the candidate values ​​of the first feature element.

26. The method according to claim 18, characterized in that, The step of determining whether to perform entropy decoding based on the probability estimation result of the first feature element includes: The probability estimation results of the feature data are input into the generator network to obtain the decision information of the first feature element. Based on the decision information of the first feature element, determine whether to perform entropy decoding on the first feature element.

27. The method according to claim 26, characterized in that, The decision information of the first feature element is a decision graph. The step of determining whether to perform entropy decoding on the first feature element based on its decision information includes: When the value corresponding to the location of the first feature element in the decision graph is a preset value, it is determined that entropy decoding needs to be performed on the first feature element; or, When the value corresponding to the location of the first feature element in the decision graph is not the preset value, it is determined that entropy decoding does not need to be performed for the first feature element.

28. The method according to claim 26, characterized in that, The step of determining whether to perform entropy decoding on the first feature element based on the decision information of the first feature element includes: When the decision information is a preset value, it is determined that entropy decoding needs to be performed on the first feature element; or... When the decision information is not a preset value, it is determined that entropy decoding is not required for the first feature element.

29. The method according to claim 17, characterized in that, The determination of whether entropy decoding needs to be performed on the first feature element includes: Obtain the side information of the feature data to be decoded; The side information of the feature data to be decoded is input into the joint network to obtain the decision information of the first feature element; Based on the decision information of the first feature element, determine whether to perform entropy decoding on the first feature element.

30. The method according to claim 29, characterized in that, The decision information of the first feature element is a decision graph. The step of determining whether to perform entropy decoding on the first feature element based on its decision information includes: When the value corresponding to the location of the first feature element in the decision graph is a preset value, it is determined that entropy decoding needs to be performed on the first feature element; or, When the value corresponding to the position of the first feature element in the decision graph is not the preset value, it is determined that entropy decoding is not required for the first feature element, and the value of the first feature element is set to k, where k is an integer.

31. The method according to claim 29, characterized in that, The step of determining whether to perform entropy decoding on the first feature element based on the decision information of the first feature element includes: When the decision information is a preset value, it is determined that entropy decoding needs to be performed on the first feature element; or... When the decision information is not the preset value, it is determined that entropy decoding does not need to be performed on the first feature element.

32. An encoder, characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 16.

33. A decoder, characterized in that, Includes processing circuitry for performing the method according to any one of claims 17 to 31.

34. A computer program product, characterized in that, Includes program code, which, when judged on a computer or processor, is used to judge the method described in any one of claims 1 to 16, 17 to 31.

35. A computer-readable storage medium, characterized in that, This includes the bitstream obtained by the encoding method according to any one of claims 1-16.

36. An encoder, characterized in that, include: One or more processors; A computer-readable storage medium coupled to the processor and storing a program determined by the processor, wherein the program, when determined by the processor, causes the encoder to perform the method according to any one of claims 1 to 16.

37. A decoder, characterized in that, include: One or more processors; A computer-readable storage medium coupled to the processor and storing a program determined by the processor, wherein the program, when determined by the processor, causes the decoder to perform the method according to any one of claims 17 to 31.

Images