Encoding method, decoding method, and electronic device
Patent Information
- Application Number
- MYPI2025007547
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-11
- Filing Date
- 2024-04-15
- Publication Date
- 2025-01-11
AI Technical Summary
In the prior art, variance adjustment requires a large amount of calculation and low efficiency, resulting in low encoding and decoding efficiency.
By converting the multiplication operation between variance and gain into an addition operation between the index of the variance and the gain converted to the logarithmic domain, the calculation amount of variance adjustment is reduced and the efficiency of variance adjustment is improved.
On the premise of ensuring compression performance, the calculation amount of variance adjustment is reduced, the encoding and decoding efficiency is improved, and the code rate overhead is reduced in some cases.
Smart Images

Figure PI2025007547_DRAWING_001 
Figure PI2025007547_DRAWING_002 
Figure PI2025007547_DRAWING_003
Abstract
Description
Coding and decoding method and electronic device
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 11, 2023, with application number 202310848558.4 and application name “Encoding and Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art
[0003] The AI (Artificial Intelligence) image compression algorithm is based on deep learning and has better compression effects than traditional image compression technologies such as JPEG (Joint Photographic Experts Group) and BPG (Better Portable Graphics).
[0004] In an end-to-end AI image compression framework, after extracting features from the original image to generate a feature map, the probability distribution of the feature map, such as its variance, is determined. The variance is then adjusted and quantized. The probability distribution of each feature point in the feature map is determined based on the quantized variance. Finally, entropy encoding is performed on the eigenvalues of each feature point based on the probability distribution to generate a bitstream. However, in existing technologies, variance adjustment requires a considerable amount of computation and is inefficient.
[0005] Summary of the Invention
[0006] In order to solve the above technical problems, the present application provides a coding and decoding method and electronic device. The method can reduce the computational complexity of variance adjustment and improve the efficiency of variance adjustment.
[0007] In a first aspect, an embodiment of the present application provides a coding method, which includes: first, obtaining an image to be encoded; then, performing feature extraction on the image to be encoded to obtain a first feature map, wherein the number of channels of the first feature map is c, the height is h, and the width is w, the first feature map includes N feature points, N is the product of c, h and w, and c, h and w are positive integers; and determining the probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points; then, performing a first adjustment process on the first feature map according to the first gain vector to obtain a second feature map, the second feature map includes N feature points; and performing a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes, wherein the second gain vector is obtained by converting the first gain vector to a logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points; then, entropy coding is performed on the second feature map according to the N second indexes to obtain a code stream.
[0008] That is to say, in the variance adjustment process, the present application converts the multiplication operation between the variance and the gain into an addition operation between the index of the variance and the gain converted to the logarithmic domain; in this way, in some cases, it is possible to reduce the amount of calculation of the variance adjustment and improve the efficiency of the variance adjustment while ensuring the compression performance, thereby improving the coding efficiency. In other cases, while reducing the amount of calculation of the variance adjustment and improving the efficiency of the variance adjustment, the bit rate overhead can be reduced (because the training and testing processes of the entropy estimation network for outputting the variance (i.e., the entropy estimation network of the prior art) are inconsistent, while the training and testing processes of the entropy estimation network for outputting the first index of the variance (i.e., the entropy estimation network of the present application) are consistent, so in some cases, the bit rate overhead can be reduced).
[0009] In addition, the probability distribution table of the present application corresponds one-to-one to the index of the variance. Furthermore, during the entropy coding process, the present application can directly determine the corresponding probability distribution table based on the quantized second index. After that, the eigenvalues of the feature points in the second feature map can be entropy coded according to the probability distribution table. However, after the prior art adjusts and quantizes the variance, it is necessary to find the nearest preset variance from multiple preset variances (the probability distribution table corresponds one-to-one to the preset variance); then, the probability distribution table corresponding to the nearest preset variance is used to entropy code the eigenvalues of the feature points in the second feature map. Among them, the process of finding the nearest preset variance in the prior art requires a certain amount of computation, so the present application can also reduce the computational complexity of entropy coding and improve the efficiency of entropy coding.
[0010] For example, the first feature map ∈ R c*h*w, where c*h*w can also be written as c×h×w. "*" and "×" do not represent multiplication operations, but represent the dimension of R, that is, the dimension of the first feature map. "c" refers to the number of channels of the first feature map, "h" represents the height of the feature map output by each channel, and "w" represents the width of the feature map output by each channel; that is, the first feature map includes feature maps of c channels, and the feature map of each channel may include h*w feature points, each feature point has an eigenvalue, and the first feature map may include eigenvalues of N feature points, where N is the product of c, h, and w, and c, h, and w are all positive integers. For the convenience of subsequent description, the eigenvalues of the N feature points in the first feature map are referred to as first eigenvalues; that is, the first feature map may include the first eigenvalues of N feature points.
[0011] For example, the second feature map ∈ R c*h*w , the second feature map may include second feature values of N feature points.
[0012] Exemplarily, the probability distribution parameter corresponding to the first feature map may include an index (an index is used to uniquely identify a variance; for ease of explanation, hereinafter referred to as the first index) of the variance corresponding to each feature point in the N feature points of the first feature map. It should be understood that the probability distribution parameter may also include other parameters, and this application does not limit this.
[0013] Exemplarily, the first adjustment process may be a multiplication operation. It should be noted that a multiplication operation may include two types of calculations: "multiplication" and "division". This application takes the first adjustment process as "multiplication" as an example for explanation.
[0014] It should be noted that the addition operation can include two calculations: "addition" and "subtraction"; when the first adjustment processing is "multiplication", the second adjustment processing is "addition"; when the first adjustment processing is "division", the second adjustment processing is "subtraction"; this application takes the second adjustment processing as "addition" as an example for explanation.
[0015] For example, the first gain vector can be determined based on the target bitrate. The size of the first gain vector can be c*1, where "c*1" can also be written as "c×1." The "*" and "×" do not represent multiplication operations, but rather represent the dimensions of the first gain vector. The first gain vector can include c gains, with one gain corresponding to one channel in the first feature map; that is, all feature points in the same channel correspond to the same gain in the first gain vector.
[0016] Exemplarily, the first gain vector may be converted to a logarithmic domain according to the first index of the variance to obtain a second gain vector.
[0017] Exemplarily, the size of the second gain vector may be c*1.
[0018] Furthermore, estimated information for determining probability distribution parameters can be written into the bitstream and transmitted to the decoder, so that the entropy estimation network at the decoder can determine the probability distribution parameters corresponding to the first feature map based on the estimated information. The estimated information can be determined based on the first feature map or based on a feature map obtained by processing the first feature map, and this application does not impose any restrictions on this.
[0019] According to the first aspect, before entropy encoding the second feature map according to N second indexes to obtain the code stream, the method also includes: performing a third adjustment process on the second feature map according to the first step length; performing a fourth adjustment process on the N second indexes according to the second step length, wherein the second step length is obtained by converting the first step length to the logarithmic domain, and the fourth adjustment process is an addition operation. In this way, the feature map (i.e., the second feature map) and the index (i.e., the second index) used for entropy encoding can be adjusted again, so that the second eigenvalue of the feature point and the second index of the variance are more concentrated, thereby reducing the code rate overhead to a certain extent. In the variance adjustment process, the present application converts the multiplication operation of the variance and the second step length into the addition operation of the index of the variance and the second step length, reducing the amount of calculation of the variance adjustment, improving the efficiency of the variance adjustment, and thus improving the coding efficiency.
[0020] Exemplarily, the first step length may be determined based on factors such as target bit rate, target image quality, and channel quality.
[0021] Exemplarily, the first step size may be converted to a logarithmic domain according to the first index of the variance to obtain the second step size.
[0022] It should be understood that, before performing the first adjustment process on the first feature map according to the first gain vector to obtain the second feature map, the third adjustment process may be performed on the first feature map according to the first step size; and before performing the second adjustment process on the N first indices according to the second gain vector to obtain the N second indices, the fourth adjustment process may be performed on the N first indices according to the second step size. In other words, this application does not limit the order in which the first adjustment process, the second adjustment process, the third adjustment process, and the fourth adjustment process are performed.
[0023] According to the first aspect, or any implementation of the first aspect above, the method further includes: generating a mask image corresponding to the first feature image, wherein the mask image is used for the third adjustment process and / or the fourth adjustment process.
[0024] The mask map, which can also be called a binary mask map, is used to extract regions of interest and mask regions of no interest. The size of the mask map corresponding to the first feature map is the same as that of the first feature map. The mask map corresponding to the first feature map may include a mask value corresponding to each of the N feature points of the first feature map, where the mask value corresponding to a feature point is either 1 or 0. The first feature map is multiplied by the mask map corresponding to the first feature map, and feature points corresponding to mask values of 1 are extracted, and the third and fourth adjustment processes are performed. Feature points corresponding to mask values of 0 are masked.
[0025] Exemplarily, the mask image is generated according to N second indices.
[0026] In this way, the second feature map and the second index can be adaptively adjusted in step size to reduce the difference between the eigenvalues of each feature point and the difference between the indexes of the corresponding variances of each feature point, thereby reducing the quantization loss and reducing the bit rate overhead to a certain extent.
[0027] According to the first aspect, or any implementation method of the first aspect above, a mask map of the first feature map is generated, including: dividing the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; pooling is performed on the Z second indices corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; based on the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain, a mask map corresponding to the first feature map is generated.
[0028] According to the first aspect, or any implementation of the first aspect above, the probability distribution parameter further includes N means, the N means corresponding one-to-one to the N feature points, and before performing a first adjustment process on the first feature map according to the first gain vector to obtain the second feature map, the method further includes: subtracting the corresponding mean from the eigenvalue of each feature point in the first feature map. In this way, the probability distribution of each feature point can be adjusted to approximate a Gaussian distribution, thereby reducing the number of probability distribution tables used for entropy coding and reducing memory usage.
[0029] According to the first aspect, or any implementation of the first aspect above, the pooling process includes at least one of the following: average pooling process, maximum pooling process, or minimum pooling process.
[0030] According to the first aspect, or any implementation of the first aspect above, before entropy encoding the second feature map according to the N second indices to obtain a bitstream, the method further includes rounding the N second indices. Since a corresponding probability distribution table must be generated in advance for each index, after rounding the second indices, only the probability distribution table corresponding to the integer needs to be pre-set, which can reduce the number of probability distribution tables used for entropy encoding and reduce memory usage.
[0031] For example, rounding may be referred to as quantization / quantization processing.
[0032] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the number of channels of the second feature map is c, the height is h, and the width is w, N is the product of c, h and w, and c, h and w are positive integers; then, determining probability distribution parameters, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points; then, adding the N first indexes according to the second gain vector to obtain N second indexes, and the N second indexes correspond one-to-one to the N feature points; thereafter, entropy decoding is performed on the encoded data of the N feature points according to the N second indexes to obtain a second feature map; then, adjusting the second feature map according to the first gain vector to obtain a first feature map, wherein the second gain vector is obtained by converting the first gain vector to the logarithmic domain; reconstructing the first feature map to obtain a reconstructed image of the image.
[0033] In the variance adjustment process, the present application converts the multiplication operation between the variance and the gain into an addition operation between the index of the variance and the gain converted to the logarithmic domain; in this way, in some cases, the calculation amount of the variance adjustment can be reduced while ensuring the compression performance, the efficiency of the variance adjustment can be improved, and thus the decoding efficiency can be improved.
[0034] In addition, the probability distribution table of the present application corresponds one-to-one to the index of the variance. Furthermore, during the entropy decoding process, the present application can directly determine the corresponding probability distribution table based on the quantized second index. After that, the eigenvalues of the feature points in the second feature map can be entropy decoded according to the probability distribution table. However, after the prior art adjusts and quantizes the variance, it is necessary to find the nearest preset variance from multiple preset variances (the probability distribution table corresponds one-to-one to the preset variance); then, the probability distribution table corresponding to the nearest preset variance is used to entropy decode the eigenvalues of the feature points in the second feature map. Among them, the process of finding the nearest preset variance in the prior art requires a certain amount of computation, so the present application can also reduce the computational complexity of entropy decoding and improve the efficiency of entropy decoding.
[0035] Exemplarily, the second feature map is obtained by performing a first adjustment process on the first feature map, the first feature map is obtained by performing feature extraction on the image, the first feature map includes N feature points, and the first feature map has c channels, h height, and w width. Exemplarily, the probability distribution parameter corresponds to the first feature map, which can also be understood as the probability distribution parameter corresponding to the second feature map.
[0036] Exemplarily, the size of the first gain vector may be c*1.
[0037] Exemplarily, the size of the second gain vector may be c*1.
[0038] For example, the "adjustment process" referred to in the second aspect may refer to the "fifth adjustment process" in the specific implementation. This adjustment process is the inverse of the first adjustment process. That is, when the first adjustment process is "multiplication," this adjustment process is "division," and when the first adjustment process is "division," this adjustment process is "multiplication."
[0039] According to the second aspect, the code stream includes the encoded data of N feature points in the second feature map after the third adjustment processing; before reconstructing the first feature map to obtain the reconstructed image of the image, the method also includes: performing a sixth adjustment processing on the first feature map according to the first step length, and the sixth adjustment processing is the inverse process of the third adjustment processing; before entropy decoding the encoded data of the N feature points according to the N second indexes to obtain the second feature map, the method also includes: performing a fourth adjustment processing on the N second indexes according to the second step length, wherein the second step length is obtained by converting the first step length to the logarithmic domain, and the fourth adjustment processing is an addition operation.
[0040] According to the second aspect, or any implementation of the second aspect above, the method further includes: generating a mask image corresponding to the first feature image, wherein the mask image is used for the sixth adjustment process and / or the fourth adjustment process.
[0041] According to the second aspect, or any implementation method of the second aspect above, a mask map of the first feature map is generated, including: dividing the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; performing pooling processing on the Z second indices corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; determining the mask map corresponding to the first feature map based on the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain.
[0042] According to the second aspect, or any implementation of the second aspect above, the probability distribution parameters also include N means, and the N means correspond one-to-one to the N feature points; before reconstructing the first feature map to obtain a reconstructed image of the image, the method also includes: adding the corresponding mean to the feature value of each feature point in the first feature map.
[0043] According to the second aspect, or any implementation of the above second aspect, the pooling processing includes at least one of the following: average pooling processing, maximum pooling processing, or minimum pooling processing.
[0044] According to the second aspect, or any implementation of the second aspect above, before entropy decoding the encoded data of N feature points according to the N second indexes to obtain the second feature map, the method also includes: rounding the N second indexes.
[0045] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0046] In a third aspect, an embodiment of the present application also provides an encoding method, which includes: first, obtaining an image to be encoded; then, performing feature extraction on the image to be encoded to obtain a first feature map, wherein the number of channels of the first feature map is c, the height is h, and the width is w, c, the first feature map includes N feature points, N is the product of c, h and w, and c, h and w are positive integers; and determining the probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points; then, performing a first adjustment process on the first feature map according to the first step length to obtain a second feature map, wherein the second feature map includes N feature points; and performing a second adjustment process on the first indexes corresponding to the N feature points according to the second step length to obtain N second indexes, wherein the second step length is obtained by converting the first step length to the logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points; then, entropy encoding is performed on the second feature map according to the N second indexes to obtain a code stream.
[0047] In the variance adjustment process, the present application converts the multiplication operation of the variance and the second step length into the addition operation of the index of the variance and the second step length. In this way, in some cases, while ensuring the compression performance, the amount of calculation of the variance adjustment is reduced, the efficiency of the variance adjustment is improved, and thus the coding efficiency is improved. In other cases, while reducing the amount of calculation of the variance adjustment and improving the efficiency of the variance adjustment, the bit rate overhead can be reduced (because the entropy estimation network used to output the variance (i.e., the entropy estimation network of the prior art) has inconsistent processes during training and testing, and the entropy estimation network used to determine the first index of the variance (i.e., the entropy estimation network of the present application) has inconsistent processes during training and testing, so in some cases, the bit rate overhead can be reduced).
[0048] According to the third aspect, the method further includes: generating a mask image corresponding to the first feature image, wherein the mask image is used for the first adjustment process and / or the second adjustment process.
[0049] According to the third aspect, or any implementation method of the third aspect above, a mask map of the first feature map is generated, including: dividing the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; pooling is performed on the Z first indexes corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; determining the mask map corresponding to the first feature map based on the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain.
[0050] According to the third aspect, or any implementation of the third aspect above, the probability distribution parameters also include N means, and the N means correspond one-to-one to the N feature points; before performing a first adjustment process on the first feature map according to the first step length to obtain the second feature map, the method also includes: subtracting the corresponding mean from the feature value of each feature point in the first feature map.
[0051] According to the third aspect, or any implementation of the third aspect above, the pooling processing includes at least one of the following: average pooling processing, maximum pooling processing, or minimum pooling processing.
[0052] According to the third aspect, or any implementation of the third aspect above, before entropy encoding the second feature map according to the N second indexes to obtain the code stream, the method also includes: rounding the N second indexes.
[0053] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0054] In a fourth aspect, an embodiment of the present application also provides a decoding method, which includes: first, receiving a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the number of channels of the second feature map is c, the height is h, and the width is w, N is the product of c, h and w, and c, h and w are positive integers; then, determining probability distribution parameters, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points; then, adding the N first indexes according to the second step size to obtain N second indexes, and the N second indexes correspond one-to-one to the N feature points; then, entropy decoding the encoded data of the N feature points according to the N second indexes to obtain a second feature map; then, adjusting the second feature map according to the first step size to obtain a first feature map, wherein the second step size is obtained by converting the first step size to the logarithmic domain; reconstructing the first feature map to obtain a reconstructed image of the image.
[0055] Exemplarily, the second feature map is obtained by performing a first adjustment process on the first feature map, the first feature map is obtained by performing feature extraction on the image, the first feature map includes N feature points, and the first feature map has c channels, h height, and w width. Exemplarily, the probability distribution parameter corresponds to the first feature map, which can also be understood as the probability distribution parameter corresponding to the second feature map.
[0056] For example, the "adjustment process" referred to in the fourth aspect may refer to the "third adjustment process" in the specific implementation. This adjustment process is the inverse of the first adjustment process. That is, when the first adjustment process is "multiplication," this adjustment process is "division," and when the first adjustment process is "division," this adjustment process is "multiplication."
[0057] According to a fourth aspect, the method further includes: generating a mask image corresponding to the first feature image, wherein the mask image is used for adjustment processing and / or addition operation.
[0058] According to the fourth aspect, or any implementation method of the fourth aspect above, a mask map of the first feature map is generated, including: dividing the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; pooling is performed on the Z first indices corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; determining the mask map corresponding to the first feature map based on the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain.
[0059] According to the fourth aspect, or any implementation of the fourth aspect above, the probability distribution parameters also include N means, and the N means correspond one-to-one to the N feature points; before reconstructing the first feature map to obtain a reconstructed image of the image, the method also includes: adding the corresponding mean to the feature value of each feature point in the first feature map.
[0060] According to the fourth aspect, or any implementation of the fourth aspect above, the pooling processing includes at least one of the following: average pooling processing, maximum pooling processing, or minimum pooling processing.
[0061] According to the fourth aspect, or any implementation of the fourth aspect above, before entropy decoding the encoded data of N feature points according to N second indexes to obtain the second feature map, the method also includes: rounding the N second indexes.
[0062] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0063] In a fifth aspect, an embodiment of the present application further provides an encoding device, the device comprising:
[0064] A first acquisition module, configured to acquire an image to be encoded;
[0065] A first encoding network is used to extract features from the image to be encoded to obtain a first feature map, wherein the first feature map has c channels, h height, and w width, and includes N feature points, where N is the product of c, h, and w, and c, h, and w are positive integers;
[0066] A first entropy estimation network is used to determine probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points;
[0067] a first adjustment module, configured to perform a first adjustment process on the first feature map according to the first gain vector to obtain a second feature map, wherein the second feature map includes N feature points;
[0068] The first adjustment module is further configured to perform a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points;
[0069] The first entropy coding network is further used to perform entropy coding on the second feature map according to the N second indexes to obtain a code stream.
[0070] It should be noted that the encoding device of the fifth aspect may also include a communication module, and may also be used to execute any one of the implementation methods of the first aspect, which will not be elaborated here.
[0071] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0072] In a sixth aspect, an embodiment of the present application further provides a decoding device, the device comprising:
[0073] a first receiving module, configured to receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the second feature map has c channels, h height, and w width, N is the product of c, h, and w, and c, h, and w are positive integers;
[0074] a second entropy estimation network, configured to determine probability distribution parameters, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0075] a second adjustment module, configured to perform an addition operation on the N first indexes according to the second gain vector to obtain N second indexes, wherein the N second indexes correspond one-to-one to the N feature points;
[0076] A first entropy decoding network is used to perform entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map;
[0077] The second adjustment module is further configured to adjust the second feature map according to the first gain vector to obtain the first feature map, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain;
[0078] The first decoding network is used to reconstruct the first feature map to obtain a reconstructed image of the image.
[0079] It should be noted that the decoding device of the sixth aspect may further include a communication module, and may also be used to execute any one of the implementation methods of the second aspect, which will not be elaborated here.
[0080] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0081] In a seventh aspect, an embodiment of the present application further provides an encoding device, comprising:
[0082] A second acquisition module is used to acquire the image to be encoded;
[0083] The second encoding network is used to extract features from the image to be encoded to obtain a first feature map, wherein the number of channels of the first feature map is c, the height is h, and the width is w, c, the first feature map includes N feature points, N is the product of c, h and w, and c, h and w are positive integers;
[0084] a third entropy estimation network, configured to determine probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points;
[0085] a third adjustment module, configured to perform a first adjustment process on the first feature map according to the first step length to obtain a second feature map, wherein the second feature map includes N feature points;
[0086] a third adjustment module, further configured to perform a second adjustment process on the first indexes corresponding to the N feature points according to a second step length to obtain N second indexes, wherein the second step length is obtained by converting the first step length into a logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points;
[0087] The second entropy coding module is further used to perform entropy coding on the second feature map according to the N second indexes to obtain a code stream.
[0088] It should be noted that the encoding device of the seventh aspect may also include a communication module, and may also be used to execute any one of the implementation methods of the first aspect, which will not be elaborated here.
[0089] The seventh aspect and any implementation of the seventh aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0090] In an eighth aspect, an embodiment of the present application further provides a decoding device, the device comprising:
[0091] a second receiving module, configured to receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the second feature map has c channels, h height, and w width, N is the product of c, h, and w, and c, h, and w are positive integers;
[0092] a fourth entropy estimation network, configured to determine probability distribution parameters, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0093] a fourth adjustment module, configured to perform an addition operation on the N first indexes according to the second step length to obtain N second indexes, where the N second indexes correspond one-to-one to the N feature points;
[0094] A second entropy decoding network is used to perform entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map;
[0095] a fourth adjustment module, further configured to adjust the second feature map according to the first step length to obtain the first feature map, wherein the second step length is obtained by converting the first step length into a logarithmic domain;
[0096] The second decoding network is used to reconstruct the first feature map to obtain a reconstructed image of the image.
[0097] It should be noted that the decoding device of the eighth aspect may also include a communication module, and may also be used to execute any one of the implementation methods of the second aspect, which will not be elaborated here.
[0098] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0099] In the ninth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, enables the electronic device to execute the encoding method in the third aspect or any possible implementation of the third aspect.
[0100] The ninth aspect and any implementation of the ninth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and no further details are given here.
[0101] In the tenth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, enables the electronic device to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.
[0102] The tenth aspect and any implementation of the tenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.
[0103] In the eleventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any possible implementation of the first aspect is executed; or, the method in the third aspect or any possible implementation of the third aspect is executed.
[0104] The eleventh aspect and any implementation of the eleventh aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and are not further described here.
[0105] In the twelfth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any possible implementation of the second aspect is executed; or, the method in the fourth aspect or any possible implementation of the fourth aspect is executed.
[0106] The twelfth aspect and any implementation of the twelfth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.
[0107] In the thirteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, it enables the computer or the processor to execute the encoding method in the third aspect or any possible implementation of the third aspect.
[0108] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which are not repeated here.
[0109] In the fourteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, it enables the computer or the processor to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.
[0110] The fourteenth aspect and any implementation of the fourteenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or may refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.
[0111] In a fifteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or the computer or the processor executes the encoding method in the third aspect or any possible implementation of the third aspect.
[0112] The fifteenth aspect and any implementation of the fifteenth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and no further details are given here.
[0113] In the sixteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the second aspect or any possible implementation of the second aspect, or the computer or the processor executes the encoding method in the fourth aspect or any possible implementation of the fourth aspect.
[0114] The sixteenth aspect and any implementation of the sixteenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or may refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0115] FIG1A is a schematic diagram showing an exemplary system framework;
[0116] FIG1B is a schematic diagram of an exemplary system framework;
[0117] FIG2A is a schematic diagram of an exemplary compression frame;
[0118] FIG2B is a schematic diagram illustrating an exemplary encoding process;
[0119] FIG3 is a schematic diagram illustrating an exemplary decoding process;
[0120] FIG4A is a schematic diagram illustrating an exemplary encoding process;
[0121] FIG4B is a schematic diagram of an exemplary mask image;
[0122] FIG5 is a schematic diagram illustrating an exemplary decoding process;
[0123] FIG6 is a schematic diagram illustrating an exemplary encoding process;
[0124] FIG7 is a schematic diagram illustrating an exemplary decoding process;
[0125] FIG8 is a schematic diagram illustrating an exemplary encoding device;
[0126] FIG9 is a schematic diagram illustrating an exemplary decoding device;
[0127] FIG10 is a schematic diagram illustrating an exemplary encoding device;
[0128] FIG11 is a schematic diagram illustrating an exemplary decoding device;
[0129] FIG12 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION
[0130] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0131] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0132] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.
[0133] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0134] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.
[0135] FIG1A is a schematic diagram showing an exemplary system framework.
[0136] Referring to FIG1A , illustratively, the first electronic device may include a first acquisition module, a first encoding module, a first channel encoding module, a first channel decoding module, a first decoding module, and a first playback module. It should be understood that the first electronic device may include more or fewer modules than those shown in FIG1A , and this application is not limited thereto. Furthermore, the first encoding module may also be referred to as a first encoder, and the first decoding module may also be referred to as a first decoder.
[0137] 1A , illustratively, the second electronic device may include a second acquisition module, a second encoding module, a second channel encoding module, a second channel decoding module, a second decoding module, and a second playback module. It should be understood that the second electronic device may include more or fewer modules than those shown in FIG1A , and this application is not limited thereto. Furthermore, the second encoding module may also be referred to as a second encoder, and the second decoding module may also be referred to as a second decoder.
[0138] Exemplarily, the process of encoding and transmitting video data by a first electronic device to a second electronic device, and decoding and playing by the second electronic device can be as follows: the first acquisition module can perform video acquisition and output video data to the first encoding module. Then, the first encoding module can encode the video data and output the code stream to the first channel encoding module. Afterwards, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device via a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can channel decode the received data to obtain the code stream and output the code stream to the second decoding module. Then, the second decoding module can decode the code stream to obtain reconstructed video data; then, the reconstructed video data is output to the second playback module, and played by the second playback module.
[0139] It should be understood that the process of the second electronic device encoding and transmitting video data to the first electronic device, which is decoded and played by the first electronic device, is similar to the process of the first electronic device transmitting video data to the second electronic device, which is played by the second electronic device, and will not be repeated here.
[0140] It should be understood that the first electronic device and the second electronic device can be directly connected without communicating through a wireless or wired network communication device, and this application does not impose any restrictions on this.
[0141] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.
[0142] For example, the system framework of Figure 1A can be applied to any scenario requiring encoding and decoding, such as VR (Virtual Reality) / AR (Augmented Reality) scenarios. In VR and AR scenarios: In one possible approach, the first electronic device is a server and the second electronic device is a VR / AR device; in another possible approach, the second electronic device is a server and the first electronic device is a VR / AR device.
[0143] For example, when a first electronic device encodes video data and a second electronic device decodes and reconstructs the video data, the first electronic device can be referred to as an encoding end and the second electronic device can be referred to as a decoding end. When a second electronic device encodes video data and a first electronic device decodes and reconstructs the video data, the second electronic device can be referred to as an encoding end and the first electronic device can be referred to as a decoding end.
[0144] FIG1B is a schematic diagram showing an exemplary system framework.
[0145] 1B (1), illustratively, a wireless or core network device may include: a channel decoding module, other decoding modules, an encoding module (ie, the encoding module involved in this application), and a channel encoding module. The wireless or core network device may be used for transcoding.
[0146] Exemplarily, the specific application scenario of Figure 1B(1) may be: when the first electronic device is not provided with an encoding module but only with other encoding modules; and the second electronic device is only provided with a decoding module but not with other decoding modules, in order to enable the second electronic device to decode and play the video data encoded by the first electronic device using other encoding modules, wireless or core network equipment may be used for transcoding.
[0147] Specifically, the first electronic device uses another encoding module to encode the video data to obtain code stream 1; it then performs channel encoding on code stream 1 and sends it to a wireless or core network device. The channel decoding module of the wireless or core network device then performs channel decoding and outputs the decoded code stream 1 to another decoding module. The other decoding module then decodes code stream 1 to obtain video data and outputs the video data to the encoding module. The encoding module then encodes the video data to obtain code stream 2 and outputs it to the channel encoding module. The channel encoding module then performs channel encoding on code stream 2 and sends it to the second electronic device. The second electronic device then calls the decoding module to decode code stream 2 obtained by channel decoding to obtain reconstructed video data; the reconstructed video data can then be played.
[0148] 1B (2), illustratively, a wireless or core network device may include: a channel decoding module, a decoding module (ie, an encoding module involved in this application), other encoding modules, and a channel encoding module. The wireless or core network device may be used for transcoding.
[0149] Exemplarily, the specific application scenario of Figure 1B (2) may be: when the first electronic device is only provided with an encoding module and no other encoding modules; and the second electronic device is not provided with a decoding module and only provided with other decoding modules, in order to enable the second electronic device to decode and play the video data encoded by the encoding module of the first electronic device, wireless or core network equipment may be used for transcoding.
[0150] Specifically, the first electronic device uses an encoding module to encode video data to obtain code stream 1. It then performs channel encoding on code stream 1 and sends it to a wireless or core network device. The channel decoding module of the wireless or core network device then performs channel decoding and outputs the decoded code stream 1 to the decoding module. The decoding module then decodes code stream 1 to obtain video data and outputs the video data to another encoding module. The other encoding module then encodes the video data to obtain code stream 2 and outputs it to the channel encoding module. The channel encoding module then performs channel encoding on code stream 2 and sends it to a second electronic device. The second electronic device then calls another decoding module to decode code stream 2 obtained through channel decoding to reconstruct video data, which can then be played.
[0151] Figure 2A is a schematic diagram of an exemplary compression framework. In the embodiment of Figure 2A, an end-to-end AI compression framework is shown; Figure 2A uses encoding and decoding of an image as an example for illustration.
[0152] 2A , an exemplary end-to-end AI compression framework may include an encoding network, a quantization module, an entropy encoding module, an entropy estimation network, an entropy decoding module, a decoding network, a first adjustment module, a second adjustment module, a third adjustment module, and a fourth adjustment module.
[0153] Exemplarily, the encoding network can be used to perform spatial transformation (also called feature extraction) on the image to be encoded, transforming the image to be encoded into another space. Exemplarily, the encoding network can be a convolutional neural network.
[0154] Exemplarily, a quantization module can be used to perform quantization processing.
[0155] Exemplarily, the entropy estimation network can be used to determine the probability distribution parameters corresponding to the feature map, such as the mean, variance, etc., which is not limited in this application.
[0156] Exemplarily, the entropy coding module can be used to perform entropy coding on the feature map according to the probability distribution parameters.
[0157] Exemplarily, the entropy decoding module can be used to perform entropy decoding on the feature map according to the probability distribution parameters.
[0158] Exemplarily, the decoding network can be used to perform inverse spatial transformation (also called reconstruction or feature recovery) on the feature map and output a reconstructed image. Exemplarily, the decoding network can be a convolutional neural network.
[0159] For example, entropy coding refers to coding that does not lose any information during the coding process according to the entropy principle. Entropy coding can include many types, such as Shannon coding, Huffman coding, and arithmetic coding, etc. This application does not limit this.
[0160] Exemplarily, the image to be encoded input to the encoding network can be any one of a RAW (unprocessed) image, an RGB (Red Green Blue) image, and a YUV ("Y" represents brightness (Luminance, Luma), and "U" and "V" represent chrominance, Chroma) image, and this application does not impose any restrictions on this.
[0161] Exemplarily, the first adjustment module and the third adjustment module are used to perform adjustment processing on the feature map, such as multiplication operation.
[0162] Exemplarily, the second adjustment module and the fourth adjustment module are used to perform adjustment processing, such as addition operation, on the index of the variance.
[0163] It should be noted that the present application does not limit the implementation of the entropy estimation network, for example, it can be a super prior network, etc.
[0164] Among them, the feature maps involved in the embodiments of the present application include: a first feature map, a second feature map, a third feature map, a fourth feature map and a fifth feature map, all ∈ R c*h*w , where "c" refers to the number of channels in the feature map, "h" represents the height of the feature map output by each channel, and "w" represents the width of the feature map output by each channel; the feature map of each channel may include eigenvalues of h*w feature points; that is, the feature map includes N feature points, where N is the product of c, w, and h. For ease of distinction, the eigenvalue of a feature point in the first feature map may be referred to as the first eigenvalue, the eigenvalue of a feature point in the second feature map may be referred to as the second eigenvalue, the eigenvalue of a feature point in the third feature map may be referred to as the third eigenvalue, the eigenvalue of a feature point in the fourth feature map may be referred to as the fourth eigenvalue, and the eigenvalue of a feature point in the fifth feature map may be referred to as the fifth eigenvalue.
[0165] The following describes an encoding and decoding process of an image (which may be an image in video data or an independent image) based on FIG. 2A and referring to the embodiments of FIG. 2B and FIG. 3 .
[0166] FIG2B is a schematic diagram illustrating an exemplary encoding process.
[0167] S201: Obtain an image to be encoded.
[0168] Illustratively, the encoding end may obtain an image to be encoded, and then refer to S202 to S206 to encode the image to be encoded to obtain a code stream.
[0169] S202 , extracting features from the image to be encoded to obtain a first feature map, where the first feature map includes N feature points.
[0170] Exemplarily, the encoding network may perform feature extraction on the image to be encoded to transform the image to be encoded into another space, thereby reducing the temporal redundancy and spatial redundancy of the image to be encoded, and obtaining a first feature map.
[0171] For example, the first feature map ∈ R c*h*w , that is, the size of the first feature map is c*h*w; where "c" refers to the number of channels in the first feature map, "h" represents the height of the feature map output by each channel, and "w" represents the width of the feature map output by each channel; that is, the first feature map includes feature maps of c channels, and the feature map of each channel may include h*w feature points. The first feature map may include N feature points, and each feature point has an eigenvalue, where N is the product of c, h, and w, and c, h, and w are all positive integers. The eigenvalues of the N feature points in the first feature map can be called first eigenvalues.
[0172] S203 : Determine probability distribution parameters corresponding to the first feature map, where the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points.
[0173] In one possible approach, referring to FIG. 2A , the first feature map may be input into an entropy estimation network to obtain probability distribution parameters corresponding to the first feature map output by the entropy estimation network.
[0174] In one possible approach, feature extraction can be performed on the first feature map to obtain a third feature map. The third feature map is then input into an entropy estimation network to obtain the probability distribution parameters corresponding to the first feature map output by the entropy estimation network. The size of the third feature map is the same as that of the first feature map.
[0175] In one possible approach, the feature map obtained by adjusting and quantizing the first feature map can be input into an entropy estimation network to obtain probability distribution parameters corresponding to the first feature map output by the entropy estimation network. The size of the feature map obtained by adjusting and quantizing the first feature map is the same as the size of the first feature map.
[0176] That is to say, the present application can directly input the first feature map into the entropy estimation network, or input the feature map obtained after processing the first feature map into the entropy estimation network to obtain the probability distribution parameters corresponding to the first feature map; the present application does not impose any restrictions on this.
[0177] Exemplarily, the entropy estimation network may process the input feature maps (e.g., the first feature map and the third feature map) to obtain estimation information; then, based on the estimation information, probability estimation is performed to determine the probability distribution parameters corresponding to the first feature map. Exemplarily, when the entropy estimation network is a super-prior network, the estimation information may be super-prior information.
[0178] Exemplarily, the probability distribution parameter corresponding to the first feature map may include an index (an index is used to uniquely identify a variance; for ease of explanation, hereinafter referred to as the first index) of the variance corresponding to each feature point in the N feature points of the first feature map. It should be understood that the probability distribution parameter may also include other parameters, and this application does not limit this.
[0179] S204: Perform a first adjustment process on the first feature map according to the first gain vector to obtain a second feature map.
[0180] For example, different gain vectors can be set in advance for different target bit rates to obtain a gain vector set M = [M1, M2, ..., M g ]; wherein, the gain vector set M may include g gain vectors, the size of each gain vector is c*1, and g is a positive integer.
[0181] Exemplarily, the first adjustment process is a multiplication operation. It should be noted that the multiplication operation can include two calculations: "multiplication" and "division". This application takes the first adjustment process as "multiplication" as an example for explanation.
[0182] Exemplarily, a corresponding gain vector can be selected from the gain vector set M as the first gain vector according to the current target bit rate; then, a first adjustment process is performed on the first feature map according to the first gain vector to obtain a second feature map.
[0183] Specifically, for each channel of the first feature map, the first eigenvalue of each feature point among the h*w feature points included in the channel is multiplied by the gain corresponding to the channel in the first gain vector to obtain the second eigenvalue of the feature point; in this way, the second eigenvalues of N feature points can be obtained, and the second eigenvalues of these N feature points can form the second feature map. For example, the second feature map ∈ R c*h*w , the second feature map may include second feature values of N feature points.
[0184] For example, in the embodiment of FIG2B , taking a feature point in a first feature map as an example, a first adjustment process is performed on the first eigenvalue of the feature point according to the first gain vector to obtain the second eigenvalue, which can be obtained by referring to the following formula (1):
[0185] In formula (1), y2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue; is the channel c in the first gain vector i The corresponding gain, y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) first eigenvalue. Among them, c i is a positive integer less than or equal to c, h j is a positive integer less than or equal to h, w k is a positive integer less than or equal to w, and b is a positive integer less than or equal to g.
[0186] According to formula (1), the larger the vector in the first gain vector, the larger the step size; in this way, the first eigenvalues of more feature points will be adjusted to the same eigenvalue, thereby achieving the effect of different bit rates in the same AI compression framework.
[0187] For example, the second eigenvalues of the N feature points may be rounded (ie, quantized); in this case, formula (1) may be converted to formula (2):
[0188] In formula (2), [] indicates rounding.
[0189] It should be understood that the gain vector set M = [M1 -1 ,M2 -1 ,......,M g -1 ], so that in the process of performing the first adjustment process on the first feature map according to the first gain vector, c i The first eigenvalue of the feature point in the channel is divided by c in the first gain vector i The gain corresponding to the channel is not limited in this application. This application is M=[M1,M2,......,M g ] as an example to illustrate.
[0190] In one possible approach, the probability distribution parameters may further include N means, and the N means correspond one-to-one to the N feature points. In this case, the first eigenvalue of each feature point in the N feature points in the first feature map may be subtracted from the corresponding mean to obtain the fourth eigenvalue of each feature point. The fourth feature group of these N feature points may form a fourth feature map. The following formula (3) may be referred to: r4(c i ,h j ,w k )=y1(c i ,h j ,w k ) -μ (c i ,h j ,w k ) (3)
[0191] In formula (3), y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) first eigenvalue, r4(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s fourth eigenvalue; μ(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponding to the mean.
[0192] Afterwards, the fourth feature map may be subjected to a first adjustment process according to the first gain vector to obtain a second feature map and the second eigenvalues of the N feature points in the second feature map may be rounded; this may be referred to formula (4):
[0193] In formula (4), y2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue, r4(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s fourth eigenvalue; is the channel c in the first gain vector i The corresponding gain.
[0194] S204 may be executed by the first adjustment module.
[0195] It should be noted that this application does not limit the execution order of S203 and S204.
[0196] S205, performing a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes, wherein the second gain vector is obtained by converting the first gain vector to the logarithmic domain, the size of the second gain vector is c*1, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points.
[0197] For example, the gain vector set M can be converted to the log domain (logarithmic domain) in advance to obtain the gain vector set M l =[m1 l ,m2 l ,......,m g l ]; wherein, the method of converting the gain in the gain vector of the gain vector set into the log domain can refer to the following formula (5):
[0198] In formula (5), is the c of the d-th gain vector in the gain vector set M i The gain corresponding to the channel, is the gain vector set M l The c of the d-th gain vector in i The gain corresponding to the channel, σ max The maximum value of the possible variance is 0.11, σ min is the minimum value of the possible variance, such as 100; d is a positive integer less than or equal to g; L is the total number of indices of possible variance.
[0199] It should be noted that the addition operation can include two calculations: "addition" and "subtraction"; in the embodiment of Figure 2B, when the first adjustment processing is "multiplication", the second adjustment processing is "addition"; when the first adjustment processing is "division", the second adjustment processing is "subtraction"; this application takes the second adjustment processing as "addition" as an example for explanation.
[0200] Exemplarily, in the embodiment of FIG2B , taking a feature point as an example, a second adjustment process is performed on the first index of the variance corresponding to the feature point according to the second gain vector to obtain the second index of the variance corresponding to the feature point, which can be referred to the following formula (6):
[0201] In formula (6), index2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponds to the second index of the variance; index1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponds to the first index of the variance; c in the second gain vector i The gain corresponding to the channel.
[0202] Afterwards, the second index can be rounded. In this case, formula (6) can be converted to formula (7):
[0203] In formula (7), [] indicates rounding.
[0204] Wherein, S205 can be executed by the second adjustment module. This application does not limit the execution order of S204 and S205.
[0205] S206: Perform entropy coding on the second feature map according to the N second indices to obtain a code stream.
[0206] Afterwards, the entropy coding module can determine the probability distribution table corresponding to the N feature points based on the N second indexes. Subsequently, the entropy coding module can perform entropy coding on the second eigenvalues of the N feature points in the second feature map (i.e., perform entropy coding on the second feature map) based on the probability distribution table corresponding to the N feature points to obtain a bitstream. In this way, the obtained bitstream can include the encoded data of the N feature points in the second feature map.
[0207] In addition, the estimation information can also be written into the code stream and transmitted to the decoding end, so that the entropy estimation network at the decoding end can determine the probability distribution parameters corresponding to the first feature map based on the estimation information.
[0208] Exemplarily, the encoding end may further include a device for transmitting a code stream; the device for transmitting a code stream may include: a transmitter and at least one storage medium, wherein the at least one storage medium is used to store the code stream generated in the embodiment of FIG. 2B above; the transmitter is used to obtain the code stream from the storage medium and send the code stream to the end-side device via the transmission medium.
[0209] Exemplarily, after the encoding end generates a bitstream, it can send the bitstream to a system for distributing the bitstream. The system for distributing the bitstream can include: at least one storage medium and a streaming device; the storage medium is used to store the at least one bitstream generated in the embodiment of FIG. 2B; the streaming device is used to obtain a target bitstream from the at least one storage medium and send the target bitstream to the end-side device, wherein the streaming device includes a content server or a content distribution server.
[0210] Figure 3 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the encoding process of Figure 2B.
[0211] Exemplarily, the decoding end may include a device for storing a code stream. The device for storing a code stream may include: a receiver for receiving the code stream; and at least one storage medium for storing the code stream. The code stream may then be decoded with reference to S302 to S306.
[0212] S301, receiving a code stream.
[0213] Exemplarily, the code stream may include encoded data of N feature points in the second feature map and encoded data of the estimation information.
[0214] S302 : Determine probability distribution parameters corresponding to the first feature map, where the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points.
[0215] Exemplarily, the entropy decoding module may first perform entropy decoding on the encoded data of the estimation information in the bitstream to obtain the estimation information. The entropy estimation network may then perform probability estimation based on the estimation information to determine probability distribution parameters corresponding to the first feature map; wherein the probability distribution parameters include a first index of the variance corresponding to each of the N feature points in the first feature map.
[0216] S303 : Perform a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes, where the N second indexes correspond one-to-one to the N feature points.
[0217] For example, S303 may refer to the description of S205 above, which will not be repeated here. S303 may be executed by the fourth adjustment module.
[0218] S304 , performing entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map.
[0219] Exemplarily, the entropy decoding module can determine the probability distribution table corresponding to N feature points based on N second indexes; then, according to the probability distribution table corresponding to the N feature points, the encoded data of the N feature points in the second feature map in the code stream can be entropy decoded to obtain the second feature map.
[0220] S305 : Perform a third adjustment process on the second feature map according to the first gain vector to obtain a first feature map.
[0221] Exemplarily, S305 may be executed by a third adjustment module.
[0222] It should be noted that the third adjustment process in the embodiment of FIG. 3 is the reverse process of the first adjustment process in the embodiment of FIG. 2A .
[0223] For example, when the first adjustment process in S204 is "multiplication", the third adjustment process in S305 is "division"; when the first adjustment process in S204 is "division", the third adjustment process in S305 is "multiplication". This application takes the third adjustment process as "division" as an example for explanation.
[0224] Exemplarily, a corresponding gain vector can be selected from the gain vector set M as the first gain vector according to the current target bit rate; then, a third adjustment process is performed on the second feature map according to the first gain vector to obtain the first feature map.
[0225] Specifically, for each channel of the second feature map, the second eigenvalue of each feature point among the h*w feature points included in the channel is divided by the gain corresponding to the channel in the first gain vector to obtain the first eigenvalue of the feature point; in this way, the first eigenvalues of N feature points can be obtained, and the first eigenvalues of these N feature points can constitute the first feature map.
[0226] For example, in the embodiment of FIG3 , taking a feature point as an example, a third adjustment process is performed on the second eigenvalue of the feature point according to the first gain vector to obtain the first eigenvalue of the feature point, which can be obtained by referring to the following formula (8):
[0227] In formula (8), y2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue; is the first gain vector, y1(c i ,h j ,w k ) represents the feature point [c i ,hj ,w k ] is the first eigenvalue of .
[0228] When the encoder performs the first adjustment processing on the fourth feature map according to the first gain vector to obtain the second feature map, the decoder performs the third adjustment processing on the second feature map according to the first gain vector to obtain the fourth feature map; then, the fourth eigenvalue of each feature point in the N features of the fourth feature map can be added to the corresponding mean to obtain the first feature map; the following formula (9) can be referred to: y1(c i ,h j ,w k )=r4(c i ,h j ,w k )+μ (c i ,h j ,w k ) (9)
[0229] In formula (9), y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) first eigenvalue, r4(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s fourth eigenvalue; μ(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponding to the mean.
[0230] S306: Reconstruct the first feature map to obtain a reconstructed image.
[0231] Afterwards, the first feature map can be input into the decoding network, which reconstructs the first feature map and outputs a reconstructed image.
[0232] Based on the encoding process of the embodiment of FIG2B and the decoding process of the embodiment of FIG3, it can be seen that in the process of adjusting the variance before quantization, the present application converts the multiplication operation between the variance and the gain into an addition operation between the index of the variance and the gain converted to the logarithmic domain; in this way, in some cases, it is possible to reduce the amount of calculation of the variance adjustment and improve the efficiency of the variance adjustment, thereby improving the encoding and decoding efficiency while ensuring the compression performance. In other cases, the bit rate overhead can be reduced while reducing the amount of calculation of the variance adjustment and improving the efficiency of the variance adjustment (because the training and testing processes of the entropy estimation network for outputting the variance (i.e., the entropy estimation network of the prior art) are inconsistent, and the training and testing processes of the entropy estimation network for determining the first index of the variance (i.e., the entropy estimation network of the present application) are consistent, so in some cases, the bit rate overhead can be reduced).
[0233] In addition, the probability distribution table of the present application corresponds one to one with the index of the variance. Furthermore, in the entropy encoding and decoding process, the present application can directly determine the corresponding probability distribution table according to the second index. After that, the eigenvalues of the feature points in the second feature map can be entropy encoded and decoded according to the probability distribution table. After the prior art adjusts the variance, it is necessary to find the nearest preset variance from multiple preset variances (the probability distribution table corresponds one to one with the preset variance); then, the probability distribution table corresponding to the nearest preset variance is used to entropy encode and decode the eigenvalues of the feature points in the second feature map. Among them, the process of finding the nearest preset variance in the prior art requires a certain amount of calculation, so the present application can also reduce the amount of calculation for entropy encoding and decoding, and improve the efficiency of entropy encoding and decoding.
[0234] 2A , and with reference to the embodiments of FIG. 4A and FIG. 5 , another encoding and decoding process of an image (which may be a frame of image in video data or an independent image) will be described below.
[0235] FIG4A is a schematic diagram illustrating an exemplary encoding process.
[0236] S401: Obtain an image to be encoded.
[0237] S402: Extract features from the image to be encoded to obtain a first feature map.
[0238] S403: Determine probability distribution parameters corresponding to the first feature map, where the probability distribution parameters include a first index of the variance.
[0239] For example, S401 to S403 may refer to the description of S201 to S203 above, which will not be repeated here.
[0240] S404: Perform a first adjustment process on the first feature map according to the first step length to obtain a second feature map.
[0241] Exemplarily, S404 may be executed by the first adjustment module.
[0242] For example, the first step length may be pre-set according to factors such as image quality requirements, bit rate requirements, and channel quality. Thus, after obtaining the first feature map, the first adjustment module may perform a first adjustment process on the first feature map according to the first step length to obtain the second feature map.
[0243] Exemplarily, a mask image corresponding to the first feature map can be generated first, and then the first feature map is subjected to a first adjustment process based on the mask image corresponding to the first feature map and the first step length to obtain a second feature map. The mask image can also be called a binary mask image, which is used to extract areas of interest and shield areas of no interest; the size of the mask image corresponding to the first feature map is the same as the size of the first feature map, and the mask image corresponding to the first feature map can include a mask value corresponding to each of the N feature points of the first feature map, and the mask value corresponding to a feature point is either 1 or 0. The first feature map is multiplied by the mask image corresponding to the first feature map, and the feature points corresponding to the mask value of 1 are extracted, and the first adjustment process and the second adjustment process are performed; the feature points corresponding to the mask value of 0 are shielded.
[0244] Exemplarily, a mask image corresponding to the first feature map can be generated based on the N first indices corresponding to the N feature points in the first feature map. Exemplarily, the first feature map can be divided into S feature blocks, where S is a positive integer; then, the Z first indices corresponding to the Z feature points contained in the S feature blocks are pooled to obtain S pooled values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; then, a mask image corresponding to the first feature map is generated based on the S pooled values corresponding to the S feature blocks and the threshold converted to the logarithmic domain.
[0245] Specifically, for each channel of the first feature map, the feature map can be divided into p feature blocks of size L1*L2. In this way, the first feature map can be divided into S (S=p*c) feature blocks. Among them, L1 and L2 can be equal or unequal, and this application does not impose any restrictions on this. Then, for a feature block, the first index of the variance corresponding to the Y feature points contained in the feature block can be pooled to obtain the pooled value corresponding to the feature block; the specific pooling process can refer to the following formula (10):
[0246] In formula (10), index q Represents the pooling value corresponding to the qth feature block, where q is a positive integer less than or equal to M.
[0247] In formula (10), MaxPool represents the maximum pooling process, pooling_block_size represents the size of the feature block, that is, L1*L2; index represents the first index of the variance, and the meaning of MaxPool(pooling_block_size,index) is: perform maximum pooling on the first index of the variance corresponding to Y feature points (Y equals L1 times L2) contained in the feature block of size L1*L2, that is, select the maximum value of the first index of the variance among the Y feature points as the pooling value corresponding to the feature block.
[0248] In formula (10), AvgPool represents average pooling processing, and the meaning of AvgPool(pooling_block_size, index) is: average pooling processing is performed on the first index of the variance corresponding to the Y feature points contained in the feature block of size L1*L2, that is, the average value of the first index of the variance corresponding to the Y feature points is calculated as the pooling value corresponding to the feature block.
[0249] In formula (10), MinPool represents minimum pooling processing. The meaning of MinPool(pooling_block_size, index) is: perform minimum pooling processing on the first index of the variance corresponding to the Y feature points contained in the feature block of size L1*L2, that is, select the minimum value of the first index of the variance among the Y feature points as the pooling value corresponding to the feature block.
[0250] In formula (10), pooling_mode=0, pooling_mode=1, and pooling_mode=2 correspond to three pooling processing strategies; the specific ones can be determined according to the target bit rate, and this application does not impose any restrictions on this.
[0251] In this way, S pooling values can be obtained; the S pooling values correspond one-to-one to the S feature blocks.
[0252] Next, a mask corresponding to the first feature map is generated based on the S pooled values corresponding to the S feature blocks and the threshold converted to the logarithmic domain. Specifically, the threshold can be pre-set and converted to the logarithmic domain.
[0253] Specifically, the threshold can be converted to the logarithmic domain by referring to formula (11):
[0254] In formula (11), threshold is the threshold value set according to the variance, threshold log is the threshold for conversion to the logarithmic domain, σ max is the maximum possible variance, σ min is the minimum possible variance.
[0255] It should be noted that when the above pooling value is obtained by average pooling, since the average value of the variance after conversion to the log domain is less than the average value of σ, η can be a positive number less than 1, such as 0.85. In this way, the mask image determined according to the first index of the variance can be guaranteed to be the same or close to the mask image determined according to the variance, thereby ensuring that the compression performance remains unchanged. When the above pooling value is obtained by maximum pooling or minimum pooling, η can be 1.
[0256] Afterwards, the S pooling values corresponding to the S feature blocks can be compared with the threshold converted to the logarithmic domain to generate a mask corresponding to the first feature map; specifically, refer to the following formula (12):
[0257] In formula (12), mask q (c i ,h j ,w k ) represents the feature points (c i ,h j ,w k ) The mask value corresponding to the greater_flag can be set as required, such as "0" or "1". index q (c i ,h j ,w k ) represents the feature points (c i ,h j ,w k ) corresponds to the first index pooling value of the variance, that is, the pooling value corresponding to the qth feature block.
[0258] For example, when greater_flag=1, if index q (c i ,h j ,w k ) is greater than thresholdlog, then the feature point (c i ,h j ,w k ) corresponds to a mask value of 1; otherwise, the feature point (c i ,h j ,w k) corresponds to a mask value of 0; thus, the mask map corresponding to the first feature map obtained can be shown in FIG4B(1). FIG4B(1) shows the mask map corresponding to the feature map (size is 8*8) of one channel of the first feature map, and the size of the feature block is 4*4; the mask value corresponding to each feature point contained in feature block 1 is 1, the mask value corresponding to each feature point contained in feature block 2 is 0, the mask value corresponding to each feature point contained in feature block 3 is 0, and the mask value corresponding to each feature point contained in feature block 4 is 1.
[0259] For example, when greater_flag=0, if index q (c i ,h j ,w k ) is less than thresholdlog, then the feature point (c i ,h j ,w k ) corresponds to a mask value of 1; otherwise, the feature point (c i ,h j ,w k ) corresponds to a mask value of 0. In this way, the mask map corresponding to the first feature map obtained can be shown in Figure 4B(2). Figure 4B(2) shows the mask map corresponding to the feature map of one channel of the first feature map (size is 8*8), and the size of the feature block is 4*4; the mask value corresponding to each feature point contained in feature block 1 is 0, the mask value corresponding to each feature point contained in feature block 2 is 1, the mask value corresponding to each feature point contained in feature block 3 is 1, and the mask value corresponding to each feature point contained in feature block 4 is 0.
[0260] Next, a first adjustment process can be performed on the first feature map based on the mask map corresponding to the first feature map and the first step length to obtain a second feature map. Specifically, for each of the N feature points in the first feature map, when the mask value corresponding to the feature point is 1, the first feature value of the feature point can be first adjusted according to the first step length; when the mask value corresponding to the feature point is 0, the first adjustment process can be omitted.
[0261] In the embodiment of FIG4A , the first adjustment process is a multiplication operation. It should be noted that the multiplication operation may include two types of calculations, “multiplication” and “division”. This application takes the case where the first adjustment process is “multiplication” as an example for explanation. Specifically, taking a feature point as an example, the process of performing the first adjustment process on the first eigenvalue of the feature point according to the first step length to obtain the second eigenvalue of the feature point can be referred to the following formula (13): y2(ci ,h j ,w k )=y1(c i ,h j ,w k )*scale (13)
[0262] In formula (13), y2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue; scale is the first step length, y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) is the first eigenvalue of .
[0263] For example, the second eigenvalues of the N feature points can also be rounded; in this case, formula (14) can be converted to formula (14): y2(c i ,h j ,w k )=[y1(c i ,h j ,w k )*scale] (14)
[0264] In formula (14), [] indicates rounding.
[0265] It should be understood that in the embodiment of FIG4A , the probability distribution parameters may also include N means in the first feature map, where the N means correspond one-to-one to the N feature points. In this case, the first eigenvalue of each of the N feature points in the first feature map can be subtracted from the corresponding mean to obtain a fourth feature map. Subsequently, a first adjustment process can be performed on the fourth feature map based on the mask map corresponding to the first feature map and the first step length to obtain a second feature map. For details, please refer to the above description and will not be repeated here.
[0266] S405 , performing a second adjustment process on the N first indexes according to a second step length to obtain N second indexes, wherein the second step length is obtained by converting the first step length into a logarithmic domain, and the N second indexes correspond one-to-one to the N feature points.
[0267] Exemplarily, S405 may be executed by the second adjustment module.
[0268] For example, in the embodiment of FIG4A , the second adjustment process is an addition operation. It should be noted that addition operations can include both "addition" and "subtraction" calculations. In the embodiment of FIG4A , when the first adjustment process is "multiplication," the second adjustment process is "addition"; when the first adjustment process is "division," the second adjustment process is "subtraction." This application uses the second adjustment process of "addition" as an example for explanation.
[0269] For example, the first step length can be converted to the logarithmic domain to obtain the second step length; refer to the following formula (15):
[0270] In formula (15), scale is the first step length, scale log is the second step length, σ max is the maximum possible variance, σ min is the minimum possible variance.
[0271] Then, a second adjustment process can be performed on the N first indices based on the mask image corresponding to the first feature map and the second step size to obtain N second indices. Specifically, for each feature point in the N feature points of the first feature map, when the mask value corresponding to the feature point is 1, a second adjustment process can be performed on the first index of the variance corresponding to the feature point according to the second step size; when the mask value corresponding to the feature point is 0, the second adjustment process can be omitted for the first eigenvalue of the variance corresponding to the feature point.
[0272] Specifically, taking a feature point as an example, the second adjustment process is performed on the first index of the variance corresponding to the feature point according to the second step length to obtain the second index of the variance corresponding to the feature point, which can be referred to the following formula (16): index2(c i ,h j ,w k )=index1(c i ,h j ,w k )+scale log (16)
[0273] In formula (16), index2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponds to the second index of the variance; scalelog is the second step length, index1(c i ,h j ,w k ) represents the feature point (c i ,hj ,w k ) corresponds to the first index of the variance.
[0274] After that, the second index can be rounded. At this time, formula (16) can be converted to formula (17): index2(c i ,h j ,w k )=[index1(c i ,h j ,w k )+scale log ] (17)
[0275] In formula (17), [] indicates rounding.
[0276] S406: Perform entropy coding on the second feature map according to the N second indices to obtain a code stream.
[0277] For example, S406 may refer to the description of S206 above, which will not be repeated here.
[0278] Fig. 5 is a schematic diagram illustrating an exemplary decoding process, wherein the decoding process in Fig. 5 corresponds to the encoding process in Fig. 4A.
[0279] S501, receiving a code stream.
[0280] S502 : Determine probability distribution parameters corresponding to the first feature map, where the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points.
[0281] For example, S501 to S502 may refer to the description of S301 to S302 above, which will not be repeated here.
[0282] S503 , performing a second adjustment process on the N first indexes according to the second step length to obtain N second indexes, where the N second indexes correspond one-to-one to the N feature points.
[0283] For example, S503 may refer to the description of S405 above, which will not be repeated here. S503 may be executed by the fourth adjustment module.
[0284] S504 , performing entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map.
[0285] For example, S504 may refer to the description of 304 above, which will not be repeated here.
[0286] S505 , performing a third adjustment process on the second feature map according to the first step length to obtain a first feature map.
[0287] Exemplarily, S505 may be executed by a third adjustment module.
[0288] Illustratively, the third adjustment process in the embodiment of FIG. 5 is the reverse process of the first adjustment process in the embodiment of FIG. 4A .
[0289] For example, when the first adjustment process in S404 is "multiplication", the third adjustment process in S505 is "division"; when the first adjustment process in S404 is "division", the fifth adjustment process in S505 is "multiplication". This application takes the third adjustment process as "division" as an example for explanation.
[0290] For example, taking a feature point as an example, the third adjustment process for the feature point according to the first step can be performed by referring to the following formula (18):
[0291] In formula (18), y2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )'s second eigenvalue; scale is the first step length, y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) is the first eigenvalue of .
[0292] When the encoding end performs the first adjustment processing on the fourth feature map according to the first step length to obtain the second feature map, the decoding end performs the third adjustment processing on the second feature map according to the first step length to obtain the fourth feature map; then, the fourth eigenvalue of each feature point in the N feature points of the fourth feature map can be added to the corresponding mean to obtain the first feature map; this can be referred to as shown in the above formula (9), which will not be repeated here.
[0293] S506: Reconstruct the first feature map to obtain a reconstructed image.
[0294] For example, S506 may refer to the description of S306 above, which will not be repeated here.
[0295] 2A , and with reference to the embodiments of FIG. 6 and FIG. 7 , another encoding and decoding process of an image (which may be a frame of image in video data or an independent image) will be described below.
[0296] FIG6 is a schematic diagram illustrating an exemplary encoding process.
[0297] S601: Obtain an image to be encoded.
[0298] S602: Extract features from the image to be encoded to obtain a first feature map.
[0299] S603, determine the probability distribution parameters corresponding to the first feature map, the probability distribution parameters include N first indices and N means, the N means correspond one-to-one to the N feature points, the N first indices correspond one-to-one to the N variances, and the N variances correspond one-to-one to the N feature points.
[0300] For example, S601 to S603 may refer to the description of S201 to S203 above, which will not be repeated here.
[0301] S604 : Subtract the corresponding mean from the first eigenvalue of each feature point in the N feature points of the first feature map to obtain a fourth feature map.
[0302] Exemplarily, the fourth feature map includes the fourth eigenvalues of N feature points, and the fourth eigenvalue is as r4[c i ,h j ,w k ] is shown in the figure and will not be repeated here.
[0303] S605 : Perform a first adjustment process on the fourth feature map according to the first gain vector to obtain a second feature map.
[0304] Illustratively, the first adjustment process in S605 is the same as the first adjustment process in S204 described above, and will not be described in detail herein.
[0305] Exemplarily, the fourth feature map includes fourth eigenvalues of N feature points, and the calculation method of the fourth eigenvalue can be shown in the following formula (19):
[0306] In formula (19), r2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue, r5(c i ,h j ,w k ) represents the feature point [c i ,h j ,w k ] the fourth eigenvalue, m b is the first gain vector.
[0307] S606 : Perform a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes.
[0308] Illustratively, the second adjustment process in S606 is the same as the second adjustment process in S205 described above, and will not be described in detail herein.
[0309] S607 , performing a third adjustment process on the second feature map according to the first step length to obtain a fifth feature map.
[0310] Illustratively, the third adjustment process in S607 is the same as the first adjustment process in S404 described above, and will not be described in detail herein.
[0311] For example, the fifth feature map includes the fifth eigenvalues of N feature points, and the calculation method of the fifth eigenvalue can be shown in the following formula (20): r5(c i ,h j ,w k )=r2(c i ,h j ,w k ) *scale (20)
[0312] In formula (20), r2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue, r5(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ), where scale is the first step length.
[0313] For example, the fifth eigenvalue of the feature point may be rounded; in this case, formula (20) may be converted to formula (21):
[0314] In formula (21), [] indicates rounding.
[0315] S608 , performing a fourth adjustment process on the N second indexes according to the second step length to obtain N third indexes.
[0316] Illustratively, the fourth adjustment process in S608 is the same as the second adjustment process in S405 described above, and will not be described in detail here.
[0317] For example, the third index can be calculated as shown in the following formula (22): index3(c i ,h j ,w k )=index2(ci ,h j ,w k )+scale log (twenty two)
[0318] In formula (22), index3(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponds to the third index of the variance; scalelog is the second step length, index2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) corresponds to the second index of the variance.
[0319] After that, the third index can be rounded. At this time, formula (22) can be converted to formula (23): index3(c i ,h j ,w k )=[index2(c i ,h j ,w k )+scale log ] (twenty three)
[0320] In formula (23), [] indicates rounding.
[0321] Among them, before executing S607 and S608, a mask image corresponding to the first feature map can be generated according to the N second indexes; specifically, the first feature map can be divided into S feature blocks, where S is a positive integer; the Z second indexes corresponding to the Z feature points contained in the S feature blocks are pooled to obtain S pooling values corresponding to the S feature blocks; based on the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain, a mask image corresponding to the first feature map is generated.
[0322] It should be understood that the present application does not limit the execution order of S605 , S606 , S607 and S608 .
[0323] S609: Perform entropy coding on the fifth feature map according to the N third indices to obtain a code stream.
[0324] For example, S609 may refer to the description of S306 above, which will not be repeated here.
[0325] FIG7 is a schematic diagram illustrating an exemplary decoding process, wherein the decoding process in FIG7 corresponds to the encoding process in FIG6 .
[0326] S701, receiving a code stream.
[0327] Exemplarily, the code stream may include encoded data of the fifth feature map (obtained by entropy encoding the fourth feature map (i.e., the fifth feature map) that has undergone the first adjustment process, the third adjustment process, and rounding) and encoded data of the estimation information.
[0328] S702, determine the probability distribution parameters corresponding to the first feature map, the probability distribution parameters include N first indices and N means, the N first indices correspond one-to-one to N variances, the N variances correspond one-to-one to N feature points, and the N means correspond one-to-one to N feature points.
[0329] For example, S701 to S702 may refer to the description of S301 to S302 above, which will not be repeated here.
[0330] S703 , performing a second adjustment process on the N first indexes according to the second gain vector to obtain N second indexes, where the N second indexes correspond one-to-one to the N feature points.
[0331] S704 , performing a fourth adjustment process on the N second indexes according to the second step length to obtain N third indexes, where the N third indexes correspond one-to-one to the N feature points.
[0332] For example, S703 to S704 may refer to the description of S604 and S608 above, which will not be repeated here.
[0333] It should be noted that this application does not limit the execution order of S703 and S704.
[0334] S705 , performing entropy decoding on the encoded data of the fifth feature map in the code stream according to the N third indices to obtain the fifth feature map.
[0335] For example, S705 may refer to the description of S304 above, which will not be repeated here.
[0336] S706 , performing a fifth adjustment process on the fifth feature map according to the first gain vector to obtain a second feature map.
[0337] Illustratively, the fifth adjustment process of S706 is the same as the third adjustment process of S305 and will not be described in detail here.
[0338] Exemplarily, the second feature map includes the second eigenvalues of N feature points, and the fourth eigenvalue can be calculated as shown in the following formula (24):
[0339] In formula (24), r2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s second eigenvalue, r5(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ), is the first gain vector.
[0340] S707 , performing a sixth adjustment process on the second feature map according to the first step length to obtain a fourth feature map.
[0341] Illustratively, the sixth adjustment process of S707 is the same as the third adjustment process of S505 and will not be described in detail here.
[0342] Exemplarily, the fourth feature map includes fourth eigenvalues of N feature points, and the fourth eigenvalues can be calculated as shown in the following formula (25):
[0343] In formula (25), r4(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s fourth eigenvalue, r2(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ), and scale is the first step length.
[0344] It should be noted that this application does not limit the execution order of S706 and S707.
[0345] S708 , adding the corresponding mean value to the fourth eigenvalue of each feature point in the N feature points of the fourth feature map to obtain the first feature map.
[0346] For example, the first feature map includes the first eigenvalues of N feature points, and the calculation method of the first eigenvalue can be shown in the following formula (26): y1(c i ,h j ,w k )=r4(c i ,h j ,wk )+μ(c i ,h j ,w k ) (26)
[0347] In formula (26), r4(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k )’s fourth eigenvalue, y1(c i ,h j ,w k ) represents the feature point (c i ,h j ,w k ) first eigenvalue, μ(c i ,h j ,w k ) is the feature point (c i ,h j ,w k ) corresponding to the mean.
[0348] S709: Reconstruct the first feature map to obtain a reconstructed image.
[0349] For example, S709 may refer to the description of S306 above, which will not be repeated here.
[0350] For example, after testing, under the same quality, the bit rate obtained by using the encoding method of Figure 6 is -0.26% compared with the bit rate obtained by using the encoding method of the prior art; that is, under the same quality, the bit rate loss of this application is lower.
[0351] Based on the embodiment of Figure 2B, the first index can be multiplied by 2 to the power of f (f is a positive integer) to obtain a fourth index; and the second gain vector can be multiplied by 2 to the power of f to obtain a third gain; then, the fourth index can be subtracted from the third gain to obtain a fifth index; then, the second feature map is entropy encoded according to the fifth index to obtain a code stream. In other words, the present application performs operations on the order of 2 to the power of f. In the prior art, the variance 1 output by the entropy estimation network is first multiplied by 2 to the power of f1 (f1 is a positive integer) to obtain variance 2, and the first gain vector is multiplied by 2 to the power of f2 (f2 is a positive integer) to obtain a fourth gain; then, the variance 2 is multiplied by the fourth gain to adjust the variance; that is, the prior art performs operations on the order of 2 to the power of (f1+f2). It can be seen that the order of magnitude of the variance adjustment of the present application is small, which can reduce the computing memory; and the quantization loss after the variance adjustment of the present application is small, and the bit rate loss is less.
[0352] Figure 8 is a schematic diagram of an exemplary encoding device. The encoding device in Figure 8 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0353] 8 , the encoding apparatus may include:
[0354] A first acquisition module 801 is used to acquire an image to be encoded;
[0355] A first encoding network 802 is configured to extract features from the image to be encoded to obtain a first feature map, wherein the first feature map has c channels, h height, and w width, and includes N feature points, where N is the product of c, h, and w, and c, h, and w are positive integers;
[0356] A first entropy estimation network 803 is configured to determine probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0357] A first adjustment module 804 is configured to perform a first adjustment process on the first feature map according to the first gain vector to obtain a second feature map, wherein the size of the first gain vector is c*1, and the second feature map includes N feature points;
[0358] The first adjustment module 805 is further configured to perform a second adjustment process on the N first indices according to the second gain vector to obtain N second indices, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain, the size of the second gain vector is c*1, the second adjustment process is an addition operation, and the N second indices correspond one-to-one to the N feature points;
[0359] The first entropy coding network 806 is further configured to perform entropy coding on the second feature map according to the N second indexes to obtain a code stream.
[0360] Figure 9 is a schematic diagram of an exemplary decoding device. The decoding device in Figure 9 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0361] 9 , the decoding apparatus may include:
[0362] A first receiving module 901 is configured to receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, where the second feature map has c channels, h height, and w width, where N is the product of c, h, and w, and c, h, and w are positive integers;
[0363] A second entropy estimation network 902 is configured to determine probability distribution parameters, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0364] A second adjustment module 903 is configured to perform an addition operation on the N first indexes according to the second gain vector to obtain N second indexes, wherein the N second indexes correspond one-to-one to the N feature points;
[0365] A first entropy decoding network 904 is configured to perform entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map;
[0366] The second adjustment module 905 is further configured to adjust the second feature map according to the first gain vector to obtain the first feature map, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain;
[0367] The first decoding network 906 is used to reconstruct the first feature map to obtain a reconstructed image.
[0368] Figure 10 is a schematic diagram of an exemplary encoding device. The encoding device in Figure 10 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0369] 10 , the encoding apparatus may include:
[0370] The second acquisition module 1001 is used to acquire the image to be encoded;
[0371] The second encoding network 1002 is used to extract features from the image to be encoded to obtain a first feature map, where the number of channels of the first feature map is c, the height is h, and the width is w, and the first feature map includes N feature points, where N is the product of c, h, and w, and c, h, and w are positive integers;
[0372] A third entropy estimation network 1003 is configured to determine probability distribution parameters corresponding to the first feature map, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0373] A third adjustment module 1004 is configured to perform a first adjustment process on the first feature map according to the first step length to obtain a second feature map, wherein the second feature map includes N feature points;
[0374] The third adjustment module 1005 is further configured to perform a second adjustment process on the first indices corresponding to the N feature points according to a second step size to obtain N second indices, wherein the second step size is obtained by converting the first indices into a logarithmic domain, the second adjustment process is an addition operation, and the N second indices correspond one-to-one to the N feature points;
[0375] The second entropy coding module 1005 is further configured to perform entropy coding on the second feature map according to the N second indexes to obtain a code stream.
[0376] Figure 11 is a schematic diagram of an exemplary decoding device. The decoding device in Figure 11 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0377] 11 , the decoding apparatus may include:
[0378] A second receiving module 1101 is configured to receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, where the second feature map has c channels, h height, and w width, where N is the product of c, h, and w, and c, h, and w are positive integers;
[0379] A fourth entropy estimation network 1102 is configured to determine probability distribution parameters, wherein the probability distribution parameters include N first indices, the N first indices correspond one-to-one to N variances, and the N variances correspond one-to-one to N feature points;
[0380] A fourth adjustment module 1103 is configured to perform an addition operation on the N first indexes according to the second step length to obtain N second indexes, where the N second indexes correspond one-to-one to the N feature points;
[0381] A second entropy decoding network 1104 is configured to perform entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain a second feature map;
[0382] The fourth adjustment module 1105 is further configured to adjust the second feature map according to the first step size to obtain the first feature map, wherein the second step size is obtained by converting the first step size into the logarithmic domain, and the third adjustment process is the inverse process of the first adjustment process;
[0383] The second decoding network 1106 is used to reconstruct the first feature map to obtain a reconstructed image.
[0384] In an example, FIG12 shows a schematic block diagram of a device 1200 according to an embodiment of the present application. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .
[0385] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1204 in the figure.
[0386] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute the instructions in the memory 1203 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.
[0387] The apparatus 1200 may be the electronic device or a chip of the electronic device in the above method embodiment.
[0388] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0389] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 1202.
[0390] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.
[0391] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.
[0392] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.
[0393] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0394] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0395] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0396] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0397] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0398] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.
[0399] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0400] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
[0401] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0402] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0403] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
Claims
1. A coding method, characterized in that: The method comprises: Obtaining an image to be encoded; Performing feature extraction on the image to be encoded to obtain a first feature map, wherein the number of channels of the first feature map is c, the height is h, and the width is w, the first feature map includes N feature points, N is the product of c, h, and w, and c, h, and w are positive integers; Determine a probability distribution parameter corresponding to the first feature map, wherein the probability distribution parameter includes N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points; Performing a first adjustment process on the first feature map according to the first gain vector to obtain a second feature map, where the second feature map includes the N feature points; Performing a second adjustment process on the N first indexes according to a second gain vector to obtain N second indexes, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond one-to-one to the N feature points; The second feature map is entropy encoded according to the N second indexes to obtain a code stream.
2. The method according to claim 1, characterized in that Before entropy encoding the second feature map according to the N second indexes to obtain a bitstream, the method further includes: Performing a third adjustment process on the second feature map according to the first step length; A fourth adjustment process is performed on the N second indexes according to a second step length, wherein the second step length is obtained by converting the first step length into a logarithmic domain, and the fourth adjustment process is an addition operation.
3. The method according to claim 2, characterized in that The method further comprises: Generate a mask image corresponding to the first feature image, wherein the mask image is used for the third adjustment process and / or the fourth adjustment process.
4. The method according to claim 3, characterized in that The step of generating a mask map of the first feature map includes: Divide the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; Performing pooling processing on the Z second indexes corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; A mask image corresponding to the first feature map is generated according to the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain.
5. The method according to any one of claims 1 to 4, characterized in that: The probability distribution parameters further include N means, and the N means correspond one-to-one to the N feature points; Before performing a first adjustment process on the first characteristic map according to the first gain vector to obtain a second characteristic map, the method further includes: Subtract the corresponding mean from the feature value of each feature point in the first feature map.
6. The method according to claim 4, characterized in that The pooling process includes at least one of the following: average pooling process, maximum pooling process or minimum pooling process.
7. The method according to any one of claims 1 to 6, characterized in that: Before entropy encoding the second feature map according to the N second indexes to obtain a bitstream, the method further includes: The N second indexes are rounded.
8. A decoding method, characterized in that: The method comprises: Receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the second feature map has a channel number of c, a height of h, and a width of w, N is a product of c, h, and w, and c, h, and w are positive integers; Determine probability distribution parameters, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points; Performing an addition operation on the N first indexes according to the second gain vector to obtain N second indexes, wherein the N second indexes correspond one-to-one to the N feature points; Performing entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain the second feature map; Adjusting the second feature map according to the first gain vector to obtain a first feature map, wherein the second gain vector is obtained by converting the first gain vector into a logarithmic domain; The first feature map is reconstructed to obtain a reconstructed image of the image.
9. The method according to claim 8, characterized in that The bitstream includes the encoded data of N feature points in the second feature map after the third adjustment process; Before reconstructing the first feature map to obtain a reconstructed image of the image, the method further includes: Performing a sixth adjustment process on the first feature map according to the first step length, wherein the sixth adjustment process is a reverse process of the third adjustment process; Before entropy decoding the encoded data of the N feature points according to the N second indexes to obtain the second feature map, the method further includes: A fourth adjustment process is performed on the N second indexes according to a second step length, wherein the second step length is obtained by converting the first step length into a logarithmic domain, and the fourth adjustment process is an addition operation.
10. The method according to claim 9, characterized in that The method further comprises: Generate a mask image corresponding to the first feature image, wherein the mask image is used for the sixth adjustment process and / or the fourth adjustment process.
11. The method according to claim 10, characterized in that The step of generating a mask map of the first feature map includes: Divide the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; Performing pooling processing on the Z second indexes corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; According to the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain, a mask image corresponding to the first feature map is determined.
12. The method according to any one of claims 8 to 11, characterized in that The probability distribution parameters further include N means, and the N means correspond one-to-one to the N feature points; Before reconstructing the first feature map to obtain a reconstructed image of the image, the method further includes: The feature value of each feature point in the first feature map is added with the corresponding mean value.
13. The method according to claim 11, characterized in that The pooling process includes at least one of the following: average pooling process, maximum pooling process or minimum pooling process.
14. The method according to any one of claims 8 to 13, characterized in that Before entropy decoding the encoded data of the N feature points according to the N second indexes to obtain the second feature map, the method further includes: The N second indexes are rounded.
15. A coding method, characterized in that: The encoding method comprises: Obtaining an image to be encoded; Performing feature extraction on the image to be encoded to obtain a first feature map, wherein the number of channels of the first feature map is c, the height is h, and the width is w, the first feature map includes N feature points, N is the product of c, h, and w, and c, h, and w are positive integers; Determine a probability distribution parameter corresponding to the first feature map, wherein the probability distribution parameter includes N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points; According to the first step length, the first feature map is subjected to a first adjustment process to obtain a second feature map, wherein the second feature map includes Describe N feature points; Performing a second adjustment process on the first indexes corresponding to the N feature points according to a second step length to obtain N second indexes, wherein the second step length is obtained by converting the first step length into a logarithmic domain, the second adjustment process is an addition operation, and the N second indexes correspond to the N feature points one by one; The second feature map is entropy encoded according to the N second indexes to obtain a code stream.
16. The method according to claim 15, characterized in that The method further comprises: Generate a mask image corresponding to the first feature image, wherein the mask image is used for the first adjustment process and / or the second adjustment process.
17. The method according to claim 16, characterized in that The step of generating a mask map of the first feature map includes: Divide the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; Performing pooling processing on the Z first indexes corresponding to the Z feature points contained in the S feature blocks to obtain S pooling values corresponding to the S feature blocks, where Z is the product of L1, L2 and S; According to the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain, a mask image corresponding to the first feature map is determined.
18. The method according to any one of claims 15 to 17, characterized in that The probability distribution parameters further include N means, and the N means correspond one-to-one to the N feature points; Before performing a first adjustment process on the first feature map according to the first step length to obtain a second feature map, the method further includes: Subtract the corresponding mean from the feature value of each feature point in the first feature map.
19. The method according to claim 17, characterized in that The pooling process includes at least one of the following: average pooling process, maximum pooling process or minimum pooling process.
20. The method according to any one of claims 15 to 19, characterized in that Before entropy encoding the second feature map according to the N second indexes to obtain a bitstream, the method further includes: The N second indexes are rounded.
21. A decoding method, characterized in that: The decoding method comprises: Receive a code stream, wherein the code stream includes encoded data of N feature points in a second feature map, the second feature map has a channel number of c, a height of h, and a width of w, N is a product of c, h, and w, and c, h, and w are positive integers; Determine probability distribution parameters, wherein the probability distribution parameters include N first indexes, the N first indexes correspond one-to-one to N variances, and the N variances correspond one-to-one to the N feature points; Performing an addition operation on the N first indexes according to the second step length to obtain N second indexes, wherein the N second indexes correspond one-to-one to the N feature points; Performing entropy decoding on the encoded data of the N feature points according to the N second indexes to obtain the second feature map; Adjusting the second feature map according to the first step length to obtain a first feature map, wherein the second step length is obtained by converting the first step length into a logarithmic domain; The first feature map is reconstructed to obtain a reconstructed image of the image.
22. The method according to claim 21, characterized in that The method further comprises: Generate a mask image corresponding to the first feature image, wherein the mask image is used for the adjustment processing and / or the addition operation.
23. The method according to claim 22, characterized in that The step of generating a mask map of the first feature map includes: Divide the first feature map into S feature blocks, wherein the height of the feature block is L1 and the width is L2, and L1, L2 and S are positive integers; Pooling is performed on the Z first indexes corresponding to the Z feature points contained in the S feature blocks to obtain the first indexes corresponding to the S feature blocks. There are S pooling values, Z is the product of L1, L2 and S; According to the S pooling values corresponding to the S feature blocks and the threshold converted to the logarithmic domain, a mask image corresponding to the first feature map is determined.
24. The method according to any one of claims 21 to 23, characterized in that The probability distribution parameters further include N means, and the N means correspond one-to-one to the N feature points; Before reconstructing the first feature map to obtain a reconstructed image of the image, the method further includes: The feature value of each feature point in the first feature map is added with the corresponding mean value.
25. The method according to claim 23, characterized in that The pooling process includes at least one of the following: average pooling process, maximum pooling process or minimum pooling process.
26. The method according to any one of claims 21 to 25, characterized in that Before entropy decoding the encoded data of the N feature points according to the N second indexes to obtain the second feature map, the method further includes: The N second indexes are rounded.
27. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method according to any one of claims 1 to 7, or executes the encoding method according to any one of claims 15 to 20.
28. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 8 to 14, or executes the encoding method according to any one of claims 21 to 26.
29. A chip, characterized in that: The method comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method according to any one of claims 1 to 7 are executed, or the steps of the method according to any one of claims 15 to 20 are executed.
30. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 8 to 14 are executed, or the steps of the method as described in any one of claims 21 to 26 are executed.
31. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 26.
32. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 26 are performed.
