Coding method, decoding method and related equipment

Abstract

The invention discloses a decoding method which can be applied to encoding and decoding scenes. The method comprises the following steps: acquiring a first code stream and a code control feature used for constraining the bit number of a first coding feature, and decoding the first code stream according to the code control feature to obtain reconstructed data. Wherein the code control feature is related to the input data, or the code control feature is related to a neural network, the neural network comprises at least one of a coding network and a decoding network, and the first coding feature is obtained according to the second coding feature. According to the method, the first code stream can be decoded based on the preset number of bits indicated by the code control feature to obtain the reconstructed data in an entropy decoding mode that code rate estimation does not need to be carried out through an entropy model. The first code stream is obtained by mapping the first coding feature instead of the occurrence probability of the elements in the first coding feature, so that the conversion between the first coding feature and the first code stream is differentiable, and the joint coding and decoding of the channel and the information source are facilitated.

Description

Technical Field

[0001] The present application relates to the field of computer communications, and in particular to an encoding method, a decoding method and related equipment. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] In recent years, with the increase of Internet traffic, the amount of data that needs to be transmitted has also increased. Data is often encoded / compressed before transmission / storage to reduce the space / transmission volume occupied by the data.

[0004] At present, the existing AI coding will first estimate the probability of each element in the coding feature, and then assign different codewords to each feature element based on the probability. However, the operation of assigning codewords based on probability is not differentiable. Summary of the invention

[0005] The present application provides an encoding method, a decoding method and related devices, which are used to reduce the problem of non-differentiable encoding and decoding due to probability allocation, and to optimize channel-source joint encoding and decoding by indicating that a first encoding feature is mapped to a preset number of bits through a code control feature.

[0006] The first aspect of the embodiment of the present application provides a decoding method that can be applied to a coding and decoding scenario. The method can be performed by a decoding device, or by a component of the decoding device (such as a processor, a chip, or a chip system, etc.). The method includes: obtaining a first code stream; obtaining a code control feature, the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits, the code control feature is related to the input data, or the code control feature is related to a neural network, the neural network includes at least one of a coding network or a decoding network, the coding network is used to obtain a second coding feature of the input data, and the first coding feature is obtained according to the second coding feature; based on the code control feature, the first code stream is decoded to obtain reconstructed data, and the reconstructed data has an associated relationship with the input data.

[0007] In the embodiment of the present application, compared with the prior art in which a symbol (codeword) is assigned to a coding feature by looking up a probability table according to the estimated probability of the coding feature, the present application indicates that the first coding feature is mapped to a preset number of bits through a code control feature, and the code control feature is related to the input data / coding network. That is, the entropy decoding method that does not require code rate estimation through an entropy model can be used to decode the first code stream based on the preset number of bits indicated by the code control feature to obtain reconstructed data. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0008] Optionally, in a possible implementation of the first aspect, the above-mentioned step of: decoding the first code stream based on the code control feature to obtain reconstructed data includes: determining a first quantity based on the code control feature, the bits of the first quantity being used to represent the bits occupied by the first coding feature; decoding the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature; and obtaining the reconstructed data based on the first coding feature.

[0009] In this possible implementation, the number of bits in the first code stream is determined and extracted through the code control feature, and then the data is reconstructed according to the number and the size of the first coding feature.

[0010] Optionally, in a possible implementation manner of the first aspect, the step of determining the first quantity based on the code control feature includes: determining a value range of the first coding feature based on the code control feature as [s,2 b -1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; the value range is [s,2 b -1+s], and the entropy coding of the first bit stream is direct binary coding, the first number is determined to be b; in the value range of [s,2 b -1+s], and the entropy coding of the first bitstream is truncated binary coding, the threshold is determined based on the value range, and the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, floor(n) means rounding down the code control feature; the first number is determined to be floor(n), and the first coding feature obtained based on floor(n) is less than the threshold. If the first coding feature obtained based on floor(n) is greater than the threshold, the first number is adjusted to floor(n)+1.

[0011] In this possible implementation, different quantities can be determined by the value range determined by the code control feature and the encoding method of the first code stream, and the quantity corresponds to the encoding side, so that correct reconstructed data can be obtained.

[0012] Optionally, in a possible implementation of the first aspect, the above-mentioned step of: determining the first quantity based on the code control feature includes: determining a value range of the first coding feature based on the code control feature; determining the first quantity based on the value range and the coding type, and the coding type includes direct binary coding or truncated binary coding.

[0013] In this possible implementation, the value range of the first coding feature is first determined by the code control feature, and then the bits occupied by the first coding feature can be determined according to the value range and the coding type. That is, in the process of determining the first number, the constraint of the code control feature on the first coding feature is considered, so as to realize the conversion between the first code stream and the first coding feature in combination with the subsequent entropy coding.

[0014] Optionally, in a possible implementation manner of the first aspect, the above-mentioned step: determining the first quantity based on the code control feature includes: determining a value range of the first coding feature as [s, b-1+s] based on the code control feature, b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; when the value range is [s, b-1+s] and the entropy coding of the first code stream is direct binary coding, determining the first quantity to be log2b; when the value range is [s, b-1+s] and the entropy coding of the first code stream is truncated binary coding, determining a threshold based on the value range, the threshold is -round(n)+s, floor(log2n) means rounding log2n down; determine that the first quantity is floor(log2n), and the first coding feature obtained based on floor(log2n) is less than the threshold; if the first coding feature obtained based on floor(log2n) is greater than the threshold, adjust the first quantity to floor(log2n)+1.

[0015] In this possible implementation, different quantities can be determined by the value range determined by the code control feature and the encoding method of the first code stream, and the quantity corresponds to the encoding side, so that correct reconstructed data can be obtained.

[0016] Optionally, in a possible implementation manner of the first aspect, the above-mentioned step: decoding the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature includes: determining the bits to be decoded from the first code stream according to the first quantity; decoding the bits to be decoded; and aligning the decoded bits to be decoded with the size to obtain the first coding feature.

[0017] In this possible implementation, for example, the bits to be decoded are extracted in the first bit stream in sequence using a first number, and the decoded bits are aligned with the size, so as to restore the first coding feature as much as possible.

[0018] Optionally, in a possible implementation manner of the first aspect, the above step: acquiring the reconstructed data based on the first coding feature includes: decoding the first coding feature through a decoding network to obtain the reconstructed data.

[0019] In this possible implementation, the reliance on super-prior information during the decoding process is reduced, the decoding process is relatively simple, and the decoding efficiency is improved.

[0020] Optionally, in a possible implementation of the first aspect, the above steps also include: obtaining a predicted mean, the predicted mean is obtained based on a second bitstream, a factor entropy model and a super decoding network; obtaining reconstructed data based on the first coding feature, including: obtaining a second coding feature based on the predicted mean and the first coding feature; decoding the second coding feature through a decoding network to obtain reconstructed data.

[0021] In this possible implementation, when the first coding feature is a residual feature, reconstructed data is obtained by predicting the mean and the first coding feature.

[0022] Optionally, in a possible implementation manner of the first aspect, the above-mentioned code control features are parameters in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0023] In this possible implementation, the code control feature is a parameter in the neural network, that is, it is independent of the code stream and can be applied to the decoding process for different code streams.

[0024] Optionally, in a possible implementation manner of the first aspect, the above-mentioned step: obtaining the code control feature includes: determining the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / the second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained by training based on a loss function of a neural network, the neural network includes an encoding network and a decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained according to the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0025] In this possible implementation, the code control feature is obtained through a mapping relationship, that is, the code control feature is related to the input data / second coding feature in the mapping relationship, so that it can be more closely matched to the input data, thereby making the reconstructed data obtained by decoding closer to the input data, thereby improving the decoding quality.

[0026] Optionally, in a possible implementation manner of the first aspect, the above step: before obtaining the first code stream, the method also includes: obtaining a second code stream; decoding the second code stream to obtain a super a priori feature; obtaining a code control feature, including: obtaining the code control feature based on the super a priori feature and a super a priori decoding network.

[0027] In this possible implementation, super a priori information is obtained through the second code stream, and the code control feature is obtained according to the super a priori information, thereby improving the possibility of obtaining the code control feature.

[0028] Optionally, in a possible implementation manner of the first aspect, the above step: obtaining the code control feature includes: receiving the code control feature sent by the encoding device.

[0029] In this possible implementation, the code control features are obtained by receiving the information sent by the decoding device, thereby reducing the resources occupied by the decoding device in calculating the code control features.

[0030] The second aspect of the embodiment of the present application provides a coding method that can be applied to a coding and decoding scenario. The method can be performed by a coding device, or by a component of the coding device (such as a processor, a chip, or a chip system, etc.). The method includes: obtaining a code control feature for constraining the number of bits of a first coding feature. Or it can be understood that the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits. And the first coding feature is encoded according to the code control feature to obtain a code stream. Among them, the code control feature is related to the input data, or the code control feature is related to a neural network, and the neural network includes at least one of a coding network or a decoding network, and the coding network is used to obtain a second coding feature of the input data, and the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature.

[0031] In the embodiment of the present application, the first coding feature is mapped to a preset number of bits through the code control feature indication. That is, the entropy coding method that does not require code rate estimation through an entropy model can encode the first coding feature based on the preset number of bits indicated by the code control feature to obtain a first code stream. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0032] Optionally, in a possible implementation manner of the second aspect, the step of: encoding the first coding feature based on the coding feature to obtain the first code stream includes: performing a first processing on the first coding feature based on the coding feature, the first processing including a truncation processing and a quantization processing, the truncation processing is used to limit the value range of the first coding feature to [s,2 b -1+s] or [s,b-1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; entropy encoding is performed on the first coding feature after the first processing to obtain a first code stream.

[0033] In this possible implementation, the code control feature is used to limit the value range of the first coding feature to [s,2 b -1+s] or [s,b-1+s], thereby controlling the preset number of bits of the first encoding feature map.

[0034] Optionally, in a possible implementation manner of the second aspect, the truncation process includes at least one of the following: a clamp function, a S-type function, a periodic function, a piecewise linear function, etc.

[0035] In this possible implementation, the truncation process can use a method of constraining values ​​out of range (i.e., clamp function), or a method of constraining all values ​​(i.e., S-type function). In addition, when using the S-type function, the nonlinear transformation module of the S-type function is combined with the subsequent uniform quantization, which can be regarded as a non-uniform quantization process.

[0036] Optionally, in a possible implementation manner of the second aspect, the above-mentioned step of: performing entropy encoding on the first coding feature after the first processing to obtain a first code stream includes: obtaining multiple binary bits corresponding to the first coding feature after the first processing; and selecting a first number of bits from the multiple binary bits to obtain the first code stream.

[0037] In this possible implementation, the decimal coding feature is converted into binary bits, and a first number of bits are selected to generate a first code stream. Since the first code stream is obtained by mapping the first coding feature itself, rather than being obtained based on the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint encoding and decoding of the channel source.

[0038] Optionally, in a possible implementation manner of the second aspect, the step of: selecting a first number of bits from a plurality of binary bits to obtain a first code stream includes: selecting a first number of bits in a value range of [s,2 b-1+s], and when the entropy coding is direct binary coding, b bits are selected from multiple binary bits to obtain a first code stream; when the value range is [s,b-1+s], and the entropy coding is direct binary coding, log2b bits are selected from multiple binary bits to obtain the first code stream.

[0039] In this possible implementation, this direct binary coding is simple to implement and easy to parallelize the coding. In addition, the first coding feature in this case can represent the value range with an integer number of bits.

[0040] Optionally, in a possible implementation manner of the second aspect, when the entropy coding is truncated binary coding, entropy coding is performed on the first coding feature after the first processing to obtain a first code stream, including: determining a second number according to a threshold, the threshold being obtained according to a value range; obtaining multiple binary bits corresponding to the first coding feature after the first processing; and selecting a second number of bits from the multiple binary bits to obtain the first code stream.

[0041] In this possible implementation, the first coding feature in this case can use decimal bits to represent the value range, thereby saving more code words and improving coding efficiency compared to direct binary coding.

[0042] Optionally, in a possible implementation of the second aspect, in the value range of [s,2 b -1+s], and the first coding feature is less than the threshold, the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed. floor(n) means rounding down the code control feature. The second number is floor(n). In the range of [s,2 b -1+s], and the first coding feature is greater than the threshold, the second number is floor(n)+1;

[0043] When the value range is [s, b-1+s] and the first coding feature is less than the threshold, the threshold is -round(n)+s, floor(log2n) means rounding log2n down, and the second quantity is floor(log2n); when the value range is [s,b-1+s] and the first coding feature is greater than the threshold, the second quantity is floor(log2n)+1.

[0044] In this possible implementation, the setting of the threshold is relatively simple, which improves the coding efficiency.

[0045] Optionally, in a possible implementation manner of the second aspect, the above-mentioned code control features are parameters in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0046] In this possible implementation, the code control feature is a parameter in the neural network, that is, it is independent of the input data and can be applied to the encoding process for different input data.

[0047] Optionally, in a possible implementation manner of the second aspect, the above-mentioned step: obtaining the code control feature includes: determining the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / the second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained by training based on a loss function of a neural network, the neural network includes an encoding network and a decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained according to the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0048] In this possible implementation, the code control feature is obtained through a mapping relationship, that is, the code control feature is related to the input data / second coding feature in the mapping relationship, so that it can be more consistent with the input data, which is beneficial to the encoding process of the input data.

[0049] Optionally, in a possible implementation manner of the second aspect, the first coding feature is the second coding feature; and the method further includes: sending a first code stream.

[0050] In this possible implementation, the reliance on super-prior information during the encoding process is reduced, the encoding process is relatively simple, and the encoding efficiency is improved.

[0051] Optionally, in a possible implementation manner of the second aspect, the first coding feature is a super a priori feature, and the super a priori feature is obtained according to the second coding feature and the super a priori coding network.

[0052] In this possible implementation, differentiable operations can be applied to the code stream scenario of transmitting super-prior features, so as to optimize the channel-source joint encoding and decoding.

[0053] Optionally, in a possible implementation of the second aspect, the above step: obtaining the code control feature includes: inputting the second coding feature into a super-prior encoding network to obtain the super-prior feature; and obtaining the code control feature based on the super-prior feature, a factor entropy model and a super-prior decoding network.

[0054] In this possible implementation, the code control features are obtained by using super-prior information, thereby improving the possibility of obtaining the code control features.

[0055] Optionally, in a possible implementation of the second aspect, the first coding feature is the difference between the second coding feature and the predicted mean, and the predicted mean is obtained based on the super prior feature, the factor entropy model and the super prior decoding network; the method also includes: encoding the quantized super prior feature to obtain a second code stream; sending the first code stream and the second code stream.

[0056] In this possible implementation, when the first coding feature is a residual feature, the quality of reconstructing input data on the decoding side can be improved by sending the first code stream and the second code stream.

[0057] The third aspect of the embodiment of the present application provides a decoding device. The decoding device includes: an acquisition unit, which is used to acquire a first code stream; the acquisition unit is also used to acquire a code control feature, the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits, the code control feature is related to the input data, or the code control feature is related to the neural network, the neural network includes at least one of the encoding network or the decoding network, the encoding network is used to acquire the second coding feature of the input data, the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature; the decoding unit is used to decode the first code stream based on the code control feature to obtain reconstructed data, and the reconstructed data has an associated relationship with the input data.

[0058] Optionally, in a possible implementation of the third aspect, the above-mentioned decoding unit is specifically used to determine a first quantity based on the code control feature, and the bits of the first quantity are used to represent the bits occupied by the first coding feature; the decoding unit is specifically used to decode the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature; the decoding unit is specifically used to obtain reconstructed data based on the first coding feature.

[0059] Optionally, in a possible implementation of the third aspect, the above steps: the above decoding unit is specifically used to determine the value range of the first coding feature based on the code control feature; the decoding unit is specifically used to determine the first quantity based on the value range and the coding type, and the coding type includes direct binary coding or truncated binary coding.

[0060] Optionally, in a possible implementation manner of the third aspect, the decoding unit is specifically configured to determine, based on the code control feature, that the value range of the first coding feature is [s,2 b -1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; the value range is [s,2b -1+s], and when the entropy coding of the first bit stream is direct binary coding, the above decoding unit is specifically used to determine that the first number is b; in the value range of [s,2 b -1+s], and the entropy coding of the first bit stream is truncated binary coding, the above decoding unit is specifically used to determine the threshold value based on the value range, and the threshold value is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, floor(n) indicates rounding down the code control feature; the above-mentioned decoding unit is specifically used to determine that the first quantity is floor(n), and the first coding feature obtained based on floor(n) is less than a threshold value. If the first coding feature obtained based on floor(n) is greater than the threshold value, the above-mentioned decoding unit is specifically used to adjust the first quantity to floor(n)+1.

[0061] Optionally, in a possible implementation manner of the third aspect, the above-mentioned decoding unit is specifically used to determine that the value range of the first coding feature is [s, b-1+s] based on the code control feature, b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; when the value range is [s, b-1+s] and the entropy coding of the first code stream is direct binary coding, the above-mentioned decoding unit is specifically used to determine that the first number is log2b; when the value range is [s, b-1+s] and the entropy coding of the first code stream is truncated binary coding, the above-mentioned decoding unit is specifically used to determine a threshold based on the value range, and the threshold is -round(n)+s, floor(log2n) means rounding log2n down; the above-mentioned decoding unit is specifically used to determine that the first quantity is floor(log2n), and the first coding feature obtained based on floor(log2n) is less than a threshold value. If the first coding feature obtained based on floor(log2n) is greater than the threshold value, the above-mentioned decoding unit is specifically used to adjust the first quantity to floor(log2n)+1.

[0062] Optionally, in a possible implementation manner of the third aspect, the above-mentioned decoding unit is specifically used to determine the bits to be decoded from the first code stream according to the first quantity; the decoding unit is specifically used to decode the bits to be decoded; the decoding unit is specifically used to align the decoded bits to be decoded with the size to obtain the first coding feature.

[0063] Optionally, in a possible implementation manner of the third aspect, the above-mentioned decoding unit is specifically used to decode the first coding feature through a decoding network to obtain reconstructed data.

[0064] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is also used to obtain a predicted mean, which is obtained according to a second bit stream, a factor entropy model and a super decoding network; the decoding unit is specifically used to obtain a second coding feature based on the predicted mean and the first coding feature; the decoding unit is specifically used to decode the second coding feature through a decoding network to obtain reconstructed data.

[0065] Optionally, in a possible implementation manner of the third aspect, the above-mentioned code control features are parameters in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0066] Optionally, in a possible implementation manner of the third aspect, the above-mentioned acquisition unit is specifically used to determine the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes an encoding network and a decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0067] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is also used to acquire a second code stream; the decoding unit is also used to decode the second code stream to obtain a super-prior feature; the acquisition unit is specifically used to obtain the code control feature based on the super-prior feature and the super-prior decoding network.

[0068] Optionally, in a possible implementation manner of the third aspect, the above-mentioned acquisition unit is specifically used to receive the code control characteristics sent by the encoding device.

[0069] The fourth aspect of the embodiment of the present application provides a coding device. The coding device includes: an acquisition unit, which is used to acquire input data; the acquisition unit is also used to acquire a code control feature, the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits, the code control feature is related to the input data, or the code control feature is related to a neural network, the neural network includes at least one of a coding network or a decoding network, the coding network is used to acquire a second coding feature of the input data, the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature; the coding unit is used to encode the first coding feature based on the code control feature to obtain a first code stream.

[0070] Optionally, in a possible implementation manner of the fourth aspect, the encoding unit is specifically used to perform a first processing on the first encoding feature based on the code control feature, the first processing includes a truncation processing and a quantization processing, and the truncation processing is used to limit the value range of the first encoding feature to [s,2 b -1+s] or [s,b-1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; a coding unit, specifically used to perform entropy coding on the first coding feature after the first processing to obtain a first code stream.

[0071] Optionally, in a possible implementation manner of the fourth aspect, the truncation processing includes at least one of the following: a clamp function, a S-type function, a periodic function, a piecewise linear function, etc.

[0072] Optionally, in a possible implementation of the fourth aspect, the above-mentioned encoding unit is specifically used to obtain multiple binary bits corresponding to the first encoding feature after the first processing; the encoding unit is specifically used to select a first number of bits from the multiple binary bits to obtain a first code stream.

[0073] Optionally, in a possible implementation of the fourth aspect, the encoding unit is specifically configured to: b -1+s], and when the entropy coding is direct binary coding, the coding unit is specifically used to select b bits from multiple binary bits to obtain a first code stream; when the value range is [s,b-1+s], and the entropy coding is direct binary coding, the coding unit is specifically used to select log2b bits from multiple binary bits to obtain the first code stream.

[0074] Optionally, in a possible implementation of the fourth aspect, when the entropy coding is truncated binary coding, the coding unit is specifically used to determine the second number according to a threshold, and the threshold is obtained according to a value range; the coding unit is specifically used to obtain multiple binary bits corresponding to the first coding feature after the first processing; the coding unit is specifically used to select a second number of bits from the multiple binary bits to obtain a first code stream.

[0075] Optionally, in a possible implementation of the fourth aspect, in the value range [s,2 b -1+s], and the first coding feature is less than the threshold, the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed. floor(n) means rounding down the code control feature. The second number is floor(n). In the range of [s,2 b -1+s], and the first coding feature is greater than the threshold, the second number is floor(n)+1; when the value range is [s,b-1+s], and the first coding feature is less than the threshold, the threshold is -round(n)+s, floor(log2n) means rounding log2n down, and the second quantity is floor(log2n); when the value range is [s,b-1+s] and the first coding feature is greater than the threshold, the second quantity is floor(log2n)+1.

[0076] Optionally, in a possible implementation manner of the fourth aspect, the above-mentioned code control features are parameters in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0077] Optionally, in a possible implementation manner of the fourth aspect, the above-mentioned acquisition unit is specifically used to determine the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes an encoding network and a decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0078] Optionally, in a possible implementation manner of the fourth aspect, the above-mentioned first coding feature is the second coding feature; the encoding device also includes: a sending unit, used to send a first code stream.

[0079] Optionally, in a possible implementation manner of the fourth aspect, the above-mentioned first coding feature is a super priori feature, and the super priori feature is obtained according to the second coding feature and the super priori coding network.

[0080] Optionally, in a possible implementation of the fourth aspect, the above-mentioned acquisition unit is specifically used to input the second coding feature into the super-prior coding network to obtain the super-prior feature; the acquisition unit is specifically used to obtain the code control feature based on the super-prior feature, the factor entropy model and the super-prior decoding network.

[0081] Optionally, in a possible implementation of the fourth aspect, the above-mentioned first coding feature is the difference between the second coding feature and the predicted mean, and the predicted mean is obtained based on the super prior feature, the factor entropy model and the super prior decoding network; the encoding unit is also used to encode the quantized super prior feature to obtain a second code stream; the encoding device also includes: a sending unit, used to send the first code stream and the second code stream.

[0082] The fifth aspect of an embodiment of the present application provides a coding device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the coding device implements the method in the above-mentioned second aspect or any possible implementation method of the second aspect.

[0083] The sixth aspect of an embodiment of the present application provides a decoding device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the decoding device implements the method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0084] A seventh aspect of an embodiment of the present application provides a coding and decoding system, including: the coding device of the fifth aspect above, and / or the decoding device of the sixth aspect above.

[0085] An eighth aspect of an embodiment of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction is run on a computer, the computer executes the method in the aforementioned second aspect or any possible implementation of the second aspect, or the computer executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0086] A ninth aspect of an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, it enables the computer to execute the method in the aforementioned second aspect or any possible implementation of the second aspect, or enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0087] A tenth aspect of an embodiment of the present application provides a readable storage medium, on which is stored a code stream encoded according to the method in the aforementioned second aspect or any possible implementation manner of the second aspect.

[0088] An eleventh aspect of an embodiment of the present application provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0089] A twelfth aspect of an embodiment of the present application provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0090] A thirteenth aspect of an embodiment of the present application provides a system for distributing code streams, the system comprising at least one storage medium and a video stream device; at least one storage medium is used to store the code stream, the code stream is encoded by an encoder according to the method in the aforementioned second aspect or any possible implementation of the second aspect; the video stream device is used to respond to a request from a decoder so that the target code stream in the at least one storage medium can be sent to the decoder.

[0091] A fourteenth aspect of an embodiment of the present application provides a method for distributing a code stream, the method comprising: receiving a first request; selecting a target code stream from at least one storage medium in response to the first request; sending the target code stream to a destination device; at least one storage medium is used to store the code stream, the code stream being encoded by an encoder according to the method in the aforementioned second aspect or any possible implementation manner of the second aspect.

[0092] A fifteenth aspect of an embodiment of the present application provides a system for processing a code stream, the system comprising an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain image data of the image source device through an interface, and encode the image data to obtain one or more code streams, the code streams being encoded by the encoder according to the method in the aforementioned second aspect or any possible implementation of the second aspect; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0093] It can be seen from the above technical solutions that the present application has the following advantages: compared with the operation of allocating codewords according to probability in the prior art, the present application indicates that the first coding feature is mapped to a preset number of bits through a code control feature, and the code control feature is related to the input data / coding network. That is, there is no need for an entropy decoding method that estimates the code rate through an entropy model, and the first code stream can be decoded based on the preset number of bits indicated by the code control feature to obtain reconstructed data. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint encoding and decoding of the channel source. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figures 1 to 3 Schematic diagrams of several application scenarios provided for embodiments of the present application;

[0095] Figure 4 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0096] Figure 5 A schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0097] Figure 6 A schematic diagram of the structure of the coding network provided in the embodiment of the present application;

[0098] Figure 7 A schematic diagram of a process for obtaining code control features provided in an embodiment of the present application;

[0099] Figure 8 An example diagram of coding provided in an embodiment of the present application;

[0100] Fig. 9 A flowchart of a decoding method provided in an embodiment of the present application;

[0101] Fig.10 A schematic diagram of the structure of a decoding network provided in an embodiment of the present application;

[0102] Fig.11 A schematic diagram of a flow chart of a coding and decoding method provided in an embodiment of the present application;

[0103] Fig.12 Another schematic diagram of the process flow of the encoding and decoding method provided in the embodiment of the present application;

[0104] Fig.13 A comparison diagram of the effects of the method provided in the embodiment of the present application and the prior art;

[0105] Fig.14 A schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0106] Fig.15 A schematic diagram of a decoding device provided in an embodiment of the present application;

[0107] Fig.16 Another schematic diagram of the structure of the encoding device provided in the embodiment of the present application;

[0108] Fig.17 Another structural schematic diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0109] The present application provides an encoding method, a decoding method and related devices for reducing the problem of non-differentiable encoding and decoding due to probability allocation, and optimizing channel-source joint encoding and decoding by indicating that a first encoding feature is mapped to a preset number of bits through a code control feature.

[0110] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0111] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.

[0112] 1. Neural Networks

[0113] A neural network can be composed of neural units. A neural unit can be a s is the weight, b is the bias operation unit, and the output of the operation unit can be:

[0114]

[0115] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0116] 2. Loss Function

[0117] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0118] 3. Transformer

[0119] The transformer structure is a feature extraction network that includes an encoding part and a decoding part (similar to a convolutional neural network). The encoding part: features are learned in the global receptive field through self-attention. The decoding part: features of the required module are learned through self-attention and cross-attention.

[0120] 4. Encoder and Decoder

[0121] Encoder is mainly used to encode signals or digital streams (usually for transmission, storage or encryption) or to obtain an encoded stream.

[0122] The decoder is mainly used to restore the coded stream into a signal or digital stream.

[0123] In some possible cases, the encoder or decoder may include the above-mentioned neural network (such as a convolutional neural network, a fully connected network, etc.), transformer, etc.

[0124] 5. Entropy Coding Engine

[0125] An entropy coding engine refers to lossless coding that does not lose any information according to the principle of information entropy. Generally speaking, if you want the compression efficiency of entropy coding to be higher, you need to provide the probability of each character as accurately as possible. In image and video AI coding tasks, it is generally assumed that the features conform to the Gaussian distribution, and then the probability of each feature element is estimated based on the Gaussian distribution. Common entropy coding engines for AI image compression include ANS and arithmetic coding.

[0126] 6. Bit rate

[0127] In image and video compression tasks, bit rate refers to the average encoding length required to encode a unit pixel.

[0128] 7. Rate-distortion performance

[0129] Rate-distortion performance is an indicator used to measure the performance of compression algorithms, which comprehensively considers two data: bit rate and decoded image distortion.

[0130] 8. Direct binary encoding

[0131] Direct binary encoding means converting a decimal integer directly into a binary integer, with each element transmitted at a specific bit length.

[0132] 9. Truncated binary encoding

[0133] Truncated binary encoding is an entropy encoding method suitable for symbols with uniform distribution characteristics. When the number of symbols is not an integer power of 2, truncated binary encoding can shorten the average code length compared to using ordinary fixed-length direct binary encoding.

[0134] In recent years, with the increase in Internet traffic, the amount of data that needs to be transmitted has increased accordingly. Data is often encoded / compressed before being transmitted / stored, thereby reducing the space / transmission volume occupied by the data. Currently, existing AI coding first estimates the probability of each element in the coding feature, and then assigns different codewords to each feature element based on the probability. However, the operation of assigning codewords based on probability is not differentiable.

[0135] In order to solve the above technical problems, an embodiment of the present application provides a coding method. Compared with the operation of allocating codewords according to probability in the prior art, the present application indicates that the first coding feature is mapped to a preset number of bits through a code control feature, and the code control feature is related to the input data / coding network. That is, the entropy decoding method that does not require code rate estimation through an entropy model can be used to decode the first code stream based on the preset number of bits indicated by the code control feature to obtain reconstructed data. Since the first code stream is obtained by mapping the first coding feature itself, rather than obtaining it based on the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0136] The following describes the application scenarios to which the method provided in the embodiments of the present application is applicable.

[0137] The method provided in the embodiments of the present application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live broadcast technology, storage, and transmission.

[0138] Figure 1 An example of an application scenario provided in an embodiment of the present application can also be understood as a coding and decoding application scenario. The application scenario includes: an AI coding unit, a storage / transmission unit, and an AI decoding unit. The method provided in an embodiment of the present application can be applied in an AI coding unit / AI decoding unit.

[0139] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application.

[0140] The storage / transmission unit is used to store (terminal, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live broadcast technology, etc.) the encoded input data.

[0141] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0142] There are many application scenarios in the above examples, which are described in detail below:

[0143] The first one is the storage application scenario.

[0144] The storage application scenarios include: terminal photo album, video surveillance, etc. The schematic diagram of this scenario is as follows Figure 2 As shown, the scene includes: an AI encoding unit, a file saving unit, a file loading unit, and an AI decoding unit.

[0145] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application to obtain a binary file.

[0146] The file saving unit is used to save the binary file output by the AI ​​encoding unit.

[0147] The loader file unit is used to load binary files.

[0148] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0149] The second is the terminal-cloud interaction scenario.

[0150] The terminal-cloud interaction scenarios include: live broadcast scenarios, public clouds, etc. The schematic diagram of this scenario is as follows Figure 3 As shown, the scenario includes: an encoding unit, a decoding unit, an AI encoding unit, an AI decoding unit, an encoding unit, and a decoding unit.

[0151] Among them, the terminal side includes an encoding unit and a decoding unit, and the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0152] The terminal obtains input data, and encodes the input data through the encoding unit to obtain encoding features. The terminal sends the encoding features to the cloud. After the cloud receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data. Then the AI ​​encoding unit encodes the reconstructed data according to the encoding method provided in the embodiment of the present application to obtain a code stream / file. The cloud decodes the code stream / file through the AI ​​decoding unit according to the decoding method provided in the embodiment of the present application to obtain reconstructed data. The cloud encodes the reconstructed data through the encoding unit to obtain the encoding features. And send the encoding features to the terminal. After the terminal receives the encoding features, it decodes the encoding features through the decoding unit to obtain the reconstructed data.

[0153] Understandably, Figure 3 The encoding methods used by the encoding unit and the AI ​​encoding unit may be inconsistent (or it can be understood that the operations of the decoding unit and the AI ​​encoding unit in the cloud can be called transcoding operations, and the operations of the AI ​​decoding unit and the encoding unit in the cloud can be called transcoding operations). For example, the cloud usually has more computing power resources than the terminal, so an algorithm with high computing power requirements and higher compression efficiency (AI encoding) can be used to re-compress the input data (transcoding) to obtain a higher compression multiple and save cloud storage space. The process of transcoding: decode first, then encode in another way.

[0154] It is understandable that the above-mentioned scenarios are just examples, and there may be other scenarios in actual applications, which are not specifically limited here.

[0155] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0156] Figure 4 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 40. The chip can be arranged as follows: Figure 3 In the execution device 310 shown in FIG. 1 , the calculation module 311 is used to complete the calculation work. The chip can also be set in Figure 3 The training device 320 shown is used to complete the training work of the training device 320 and output the target model / rule 301.

[0157] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU) and any other processor suitable for large-scale XOR processing. Take NPU as an example: the neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0158] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.

[0159] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 408.

[0160] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0161] In some implementations, the vector calculation unit 407 stores the vector of processed outputs to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.

[0162] The unified memory 406 is used to store input data and output data.

[0163] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0164] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0165] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0166] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0167] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0168] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0169] Figure 4 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 40. The chip can be arranged as follows: Figures 1 to 3 The AI ​​encoding unit / AI decoding unit shown is used to complete the encoding / decoding process.

[0170] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU) and any other processor suitable for large-scale XOR processing. Take NPU as an example: the neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0171] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.

[0172] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 408.

[0173] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0174] In some implementations, the vector calculation unit 407 stores the vector of processed outputs to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.

[0175] The unified memory 406 is used to store input data and output data.

[0176] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0177] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0178] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0179] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0180] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0181] The method provided by the embodiment of the present application is described below. The method can be performed by a coding device, or by a component of the coding device (such as a processor, a chip, or a chip system, etc.). The coding device can be a cloud device or a terminal device. Of course, the method can also be performed by a system consisting of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, GPU in the coding device, and this application does not limit it.

[0182] See also Figure 5 , a flow chart of the encoding method provided by the embodiment of the present application, the application scenario applicable to the method is mainly the encoding and decoding scenario, etc. It can include steps 501 to 503. Steps 501 to 503 are described in detail below.

[0183] Step 501, obtaining input data.

[0184] In the embodiment of the present application, there are multiple ways for the encoding device to obtain input data, which may be through collection / shooting, by receiving data sent by other devices, or by selecting from a database, etc., which are not limited here.

[0185] The input data in the embodiments of the present application may include at least one of the following: image, video, audio, text, etc., which is not specifically limited here.

[0186] Step 502, obtaining code control features.

[0187] The coding device obtains a code control feature, the code control feature is used to constrain the number of bits used for transmission of the first coding feature, or the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits, and the code control feature is related to the input data, or the code control feature is related to the neural network. The neural network includes at least one of a coding network or a decoding network, the coding network is used to obtain a second coding feature of the input data, the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature.

[0188] Optionally, during the encoding process, the first encoding feature mentioned above may also be referred to as a feature to be encoded.

[0189] For example, the coding feature is used to indicate that the first coding feature is mapped to a symbol of 2 to the power of n, and one symbol corresponds to n bits. Alternatively, the coding feature is understood to indicate that the number of values ​​of the first coding feature is 2 to the power of n.

[0190] Compared with the operation of allocating codewords according to probability in the prior art, the first coding feature can be directly converted into a code stream by mapping, and it is beneficial to optimize the channel source joint coding and decoding and parallel coding and decoding.

[0191] In the embodiment of the present application, there are multiple situations in which the first coding feature is obtained according to the second coding feature, which are described below respectively:

[0192] In the first type, the first coding feature is the second coding feature.

[0193] This situation can be understood as that the first coding feature is irrelevant to the super prior information, and the second coding feature is directly used as the feature to be coded. The second coding feature is obtained according to the coding network and the input data. Specifically, the input data is input into the coding network to obtain the second coding feature.

[0194] The encoding network in the embodiment of the present application can be a feedforward neural network (Feedforward Neural Networks, FNN), a convolutional neural network (Convolutional Neural Network, CNN), a Transformer, a generative adversarial network (Generative Adversarial Networks, GAN), etc., which is not limited here.

[0195] For example, taking the encoding network as CNN as an example, the structure of the encoding network can be as follows: Figure 6 As shown, the encoding network includes a first convolutional layer, a plurality of residual layers, and a plurality of second convolutional layers. The residual layers are alternately connected with the convolutional layers. The input of the encoding network is the input data, and the output of the encoding network is the second encoding feature. Figure 6In the example, the first convolutional layer can be specifically expressed as: conv 5×5 3×192 2↓, that is, the convolution kernel size of the first convolutional layer is 5×5, the number of input channels is 3, the number of output channels is 192, and the stride is 2. The residual layer can be specifically expressed as the Residual Nonlinear Transform Unit with Attention Mechanism (ResAU), the output of the previous convolutional layer is point-wise multiplied with the output of the previous convolutional layer after passing through the activation layer and the convolutional layer, and the multiplied value is point-wise added to the output of the previous convolutional layer to obtain the output of the residual layer. The second convolutional layer can be specifically expressed as: conv 5×5192×192 2↓, that is, the convolution kernel size of the second convolutional layer is 5×5, the number of input channels is 192, the number of output channels is 192, and the stride is 2.

[0196] Understandably, Figure 6 This is just an example of a coding network. In actual applications, the coding network may also include more or fewer convolutional layers and residual layers. The parameters in the convolutional layers (such as convolution kernel size, number of channels, number of input channels, number of output channels, step size, etc.) may also be other cases. It may also include other network layers (such as pooling layers, activation layers), etc., which are not limited here.

[0197] In this case, there are many ways to obtain the code control feature (the following description of the various situations related to the code control feature and the input data / second coding feature is not expanded here). For example, the encoding device first obtains the super prior feature based on the second coding feature and the super prior encoding network. Then, the code control feature is obtained based on the super prior feature, the factor entropy model and the super prior decoding network.

[0198] The second type is that the first coding feature is the difference between the second coding feature and the predicted mean (also called residual feature).

[0199] In this case, the encoding device first obtains a super-prior feature based on the second encoding feature and the super-prior encoding network, and then obtains a predicted mean based on the super-prior feature.

[0200] Optionally, the encoding device first inputs the second encoding feature (y) into the super-prior encoding network to obtain the super-prior feature (z). The super-prior feature is then quantized, and the quantized super-prior feature (z_hat) is entropy estimated through a factor entropy model, and entropy encoded through an asymmetric numeric system (ANS) algorithm to generate a second bit stream (bit_z). The second bit stream is entropy decoded through ANS, and the entropy decoded feature (which can also be understood as the reconstructed z_hat) is input into the super-prior decoding network to obtain a predicted mean (mean).

[0201] In addition, the method of obtaining the code control features may be the same as or different from the method of obtaining the above-mentioned predicted mean value, which will be described later and will not be expanded here.

[0202] For example, the process of obtaining the code control feature in this case can be as follows: Figure 7 As shown. The super prior encoding network includes a third convolutional layer, multiple activation layers and multiple second convolutional layers. The activation layers are alternately connected to the convolutional layers. The super prior decoding network includes a first deconvolutional layer, a fourth convolutional layer and multiple activation layers. The activation layers are alternately connected to the deconvolutional layers. Figure 7 In the example, the third convolution layer can be specifically expressed as: conv 3×3 192×192, that is, the convolution kernel size of the third convolution layer is 3×3, the number of input channels is 192, and the number of output channels is 192. The second convolution layer can be specifically expressed as: conv 5×5 192×192 2↓, that is, the convolution kernel size of the second convolution layer is 5×5, the number of input channels is 192, the number of output channels is 192, and the stride is 2. The first deconvolution layer can be specifically expressed as: conv 5×5 192×192 2↑, that is, the convolution kernel size of the first deconvolution layer is 5×5, the number of input channels is 192, the number of output channels is 192, and the stride is 2. The fourth convolutional layer can be specifically expressed as: conv 3×3 192×384, that is, the convolution kernel size of the fourth convolutional layer is 3×3, the number of input channels is 192, and the number of output channels is 576. Among them, 384 in the fourth convolutional layer represents double channels, which are code control features and predicted mean values. Of course, if the fourth convolutional layer also has a scaling factor, that is, triple channels, the fourth convolutional layer can be specifically expressed as: conv 3×3 192×576.

[0203] Understandably, Figure 7It is just an example of a super-prior encoding network, factor entropy model and super-prior decoding network. In actual applications, each network can also include more or fewer convolutional layers and activation layers. The parameters in the convolutional layer (such as convolution kernel size, number of channels, number of input channels, number of output channels, step size, etc.) can also be other cases. It can also include other network layers (such as pooling layer, residual layer), etc., which are not limited here.

[0204] The third type is that the first coding feature is a super prior feature.

[0205] In this case, the encoding device obtains the super a priori feature based on the second encoding feature and the super a priori encoding network. The method of obtaining the super a priori feature can refer to the description in the second case above, which will not be repeated here.

[0206] It can be understood that the aforementioned several situations of the first coding feature are just examples. In practical applications, there may be other situations, which are not limited here.

[0207] In the embodiment of the present application, there are multiple ways to obtain the code control features, which are described below respectively:

[0208] The first one is that the coded features are parameters in the neural network.

[0209] The above-mentioned neural network can be understood as a codec system or a coding network. In this case, it can be understood that the code control feature is independent of the input data, but is related to the codec system or the coding network.

[0210] Exemplarily, the loss function of the neural network in this case can be described as the following formula 1, which is not expanded here.

[0211] Second, the code control feature is related to the input data / second coding feature.

[0212] The code control features are related to the input data / second coding features and can be divided into the following situations:

[0213] 1. The encoding device obtains the code control features based on the mapping relationship.

[0214] In this case, the mapping relationship is used to represent the association relationship between the input data and the code control feature, or the mapping relationship is used to represent the association relationship between the second coding feature and the code control feature. After the coding device obtains the input data / second coding feature, it can determine the code control feature corresponding to the input data / second coding feature according to the above mapping relationship.

[0215] In addition, the above-mentioned mapping relationship can be a mapping relationship preset according to actual needs, or can be obtained by training based on the first loss function of the neural network, etc., which is not limited here.

[0216] Optionally, when the mapping relationship is obtained by training the first loss function based on a neural network, the neural network includes an encoding network and a decoding network. The encoding network is used to obtain encoding features of training data, and the decoding network is used to restore the training data (or be understood as reconstructed training data).

[0217] In addition, the above-mentioned first loss function includes a first loss term and a second loss term (also known as the code rate loss of the coding feature to be transmitted, or understood as the sum of the number of bits allocated to the first coding feature). The first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data. The second loss term is used to represent the total number of bits used for the transmission of the coding feature to be transmitted or the number of possible values ​​of the coding feature. The total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0218] Exemplarily, the first loss function can be shown as Formula 1:

[0219] Formula 1: L1 = a(x-x') + b·sum(n);

[0220] Among them, L1 represents the first loss function; a is the coefficient of the first loss term; x represents training data; x' represents output data; b is the coefficient of the second loss term; n is the number of bits of the coding feature or the number of values ​​of the first coding feature (that is, the second loss term = b·sum(log2n)); a and b can be set according to actual needs and are not limited here.

[0221] In addition, if the first loss function is described from the perspective of the number of values ​​of the first coding feature, the first loss function can also be shown as Formula 2:

[0222] Formula 2: L1 = a(x-x') + b·sum(log2m);

[0223] Among them, L1 represents the first loss function; a is the coefficient of the first loss term; x represents the training data; x' represents the output data; b is the coefficient of the second loss term; m is the number of values ​​of the first coding feature; a and b can be set according to actual needs and are not limited here.

[0224] It can be understood that Formula 1 and Formula 2 are just several examples of the first loss function in this case. In practical applications, the first loss function can also have other forms, which are not specifically limited here.

[0225] 2. The encoding device obtains the code control features based on the super-prior features.

[0226] In this case, the encoding device obtains the second encoding feature, and inputs the second encoding feature into the super-prior encoding network to obtain the super-prior feature, and then obtains the code control feature based on the super-prior feature, the factor entropy model and the super-prior decoding network.

[0227] For example, in this case, the process based on the super-prior feature or the code control feature can be as described above. Figure 7 As shown, Figure 7 The output of the super prior decoding network can include not only the predicted mean but also the code control features.

[0228] Optionally, the neural network in this case includes an encoding network, a decoding network, a super priori encoding network, a factor entropy model and a super priori decoding network. The neural network is trained with the training data as input and the value of the second loss function being less than a threshold as the goal.

[0229] Among them, the second loss function includes a first loss term, a second loss term (also called the code rate loss of the coding feature to be transmitted) and a third loss term (also called the super-prior code rate loss). The first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data. The second loss term is used to represent the total number of bits used for the transmission of the coding feature to be transmitted or the number of possible values ​​of the coding feature. The total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data. The third loss term is the entropy of the super-prior feature estimated by the factor entropy model. For example, the probability of the super-prior feature is first obtained through the factor entropy model, and then the entropy of the super-prior feature is calculated based on the probability.

[0230] Exemplarily, the second loss function can be shown as Formula 3:

[0231] Formula 3: L2 = a(x-x') + second loss term + third loss term;

[0232] Wherein, L2 represents the second loss function; a is the coefficient of the first loss term; x represents training data; x' represents output data; the second loss term can be the second loss term in Formula 1 or Formula 2; a and b can be set according to actual needs and are not limited here.

[0233] It can be understood that Formula 3 is only an example of the second loss function in this case. In practical applications, the second loss function may also have other forms, which are not specifically limited here.

[0234] It can be understood that the above-mentioned methods of obtaining code control features are just examples. In actual applications, there may be other ways to obtain code control features (such as receiving code control features sent by other devices, etc.), which are not specifically limited here.

[0235] Step 503: Encode the first coding feature based on the code control feature to obtain a first code stream.

[0236] After acquiring the code control feature, the encoding device encodes the first coding feature based on the code control feature to obtain a first code stream.

[0237] The encoding in the embodiment of the present application at least includes: first processing and entropy coding. The first processing includes truncation processing and quantization processing. That is, this step includes: performing a first processing on the first coding feature based on the code control feature, and performing entropy coding on the first coding feature after the first processing to obtain a first code stream.

[0238] Optionally, the above-mentioned first processing may also include a scaling process, which is used to adjust the value range of the first coding feature, that is, the scaling process can be coordinated with the truncation process to realize the constraint of the code control feature on the value range of the first coding feature. For example, the scaling process can be understood as performing a dot multiplication process on the scaling factor and the first coding feature. The scaling factor can be obtained by training in conjunction with a neural network, or by using super-prior information, or by using a truncation factor used in the truncation process (for example, the scaling factor is the same as the truncation factor), etc., which is not specifically limited here. In addition, in practical applications, the order of the first processing is often the order of scaling, truncation, and quantization. Of course, it is also possible to adjust the order according to actual needs, which is not specifically limited here.

[0239] The truncation process is used to limit the value range of the first coding feature to [s,2 b -1+s] or [s,b-1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature. Integer processing includes at least one of the following: rounding, rounding up, rounding down, etc.

[0240] Optionally, the truncation process can use a clamp function or an S-type function (such as a periodic function, a trigonometric function, a piecewise linear function, etc.) to limit the value range of the first coding feature. Among them, the clamp function can be understood as constraining values ​​that are out of range. The S-type function can be understood as constraining all values. For example, the sigmoid function, the error function (ERror Function, ERFC), etc. A periodic function means that for the function y=f(x), if there is a non-zero constant T, so that when x takes every value in the domain of definition, f(x+T)=f(x) holds, then the function y=f(x) is called a periodic function.

[0241] In addition, the truncation process can often be understood as a process of multiplying by a truncation coefficient c, which is determined according to the above value range and function. For example, when the truncation process uses the clamp function, taking the clamp function and s as 0.25 as an example, c=(2 n -1)*0.5. For another example, when the truncation process uses the S-type function, the truncation process can be expressed as: the feature after truncation = (2*sigmoid (feature before truncation)-1)*c. This example can map the feature to the value range of the first coding feature described by the code control feature through the sigmoid function. And without the help of the clamp operation. That is, a nonlinear operation is introduced in the transformation process. Considering that the distribution to be compressed is closer to the Gaussian distribution, this nonlinearity should be beneficial to the compression of the feature. From another perspective, if uniform quantization is used, the nonlinear transformation module of the S-type function combined with the subsequent uniform quantization can be regarded as a non-uniform quantization process.

[0242] The quantization processing in the embodiment of the present application can be uniform quantization (such as rounding, rounding up, rounding down, etc.) or non-uniform quantization, which is not limited here.

[0243] The entropy coding in the embodiment of the present application includes direct binary coding or truncated binary coding, which are described below respectively:

[0244] In one possible implementation, entropy coding is direct binary coding. For example, in the range [s,2 b -1+s], the above b=x(n). And the above entropy encoding of the first coding feature after the first processing to obtain the first code stream specifically includes: obtaining multiple bits corresponding to the first coding feature after the first processing (which can also be understood as the process of converting decimal to binary), and then selecting b bits from the multiple bits for binary encoding to obtain the first code stream. For example, in the case where the value range is [s,b-1+s], the above b=x(n). And the above entropy encoding of the first coding feature after the first processing to obtain the first code stream specifically includes: obtaining multiple bits corresponding to the first coding feature after the first processing (which can also be understood as the process of converting decimal to binary), and then selecting log2b bits from the multiple bits for binary encoding to obtain the first code stream.

[0245] For example, s=0, in the case of direct binary coding, the first coding feature after the first processing is recorded as y_hat. As can be seen from the above, the value range of y_hat is limited to [s,2 x(n) -1+s]. Step 1: Shift y_hat to [0,2 x(n)-1]: y_hat1 = y_hat-s (i.e., y_hat1 is the translated y_hat), and the size of y_hat1 is [C×H1×W1]. One way of step 2: Taking an element y_hat1 in the first encoding feature y_hat as an example, the specific steps of converting the decimal y_hat1 tensor into the binary symbol bit_y include: taking out y_hat1 x,y,z x(n) in the storage location x,y,z Bits (e.g. 0th to x(nth) x,y,z -1 bit) as the converted y_hat1 x,y,z The binary symbol bit_y x,y,z (i.e., the sub-bitstream in the first bitstream). x,y,z =0, skip and do not encode. Another way of step 2: scale y_hat1 to get y_hat2 = y_hat1 × 2 i , then the specific steps of converting the decimal y_hat2 tensor into the binary symbol bit_y include: taking out y_hat2 x,y,z x(n) in the storage location x,y,z Bits (e.g., bit i to bit x(n) x,y,z -1+i bits) as the converted y_hat2 x,y,z The binary symbol bit_y x,y,z (i.e., the sub-bitstream in the first bitstream). When x(n) x,y,z =0, skip and do not encode. Figure 8 It shows that taking x(n) for an element in y_hat1 or y_hat2 x,y,z Bit generation bit_y x,y,z process (assuming that the data type of y_hat1 or y_hat2 is int16). In addition, take out x(n) x,y,z The bit method can be shifted or the CPU / GPU memory can be directly operated, etc. The specific method is not limited here.

[0246] In another possible implementation, entropy coding is truncated binary coding. The above b=n. And the above entropy coding of the first coding feature after the first processing to obtain the first code stream specifically includes: determining the second number according to a threshold, and the threshold is obtained according to a value range; obtaining multiple binary bits corresponding to the first coding feature after the first processing (which can also be understood as a process of converting decimal to binary), and then selecting a second number of bits from the multiple binary bits for binary coding to obtain the first code stream.

[0247] Optionally, in the range [s,2 b -1+s], the above threshold is 2 floor(n)+1-round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, floor(n) means rounding down the code control feature; when the first coding feature at the corresponding position is less than the threshold, the second number is k=floor(n) (such as n1 in the following example); when the first coding feature at the corresponding position is greater than the threshold, the second number is k+1 (such as n2 in the following example).

[0248] Among them, when the value range is [s, b-1+s], the above threshold is -round(n)+s, round(n) means rounding n, floor(log2n) means rounding log2n down; when the first coding feature at the corresponding position is less than the threshold, the second number is k=floor(log2n); when the first coding feature at the corresponding position is greater than the threshold, the second number is k+1. It can be understood that the above thresholds are only examples. In practical applications, the above thresholds can be set according to actual needs, and are not specifically limited here.

[0249] For example, s = 0, in the case of truncated binary coding (or understood as n ≥ 0.5, the 0.5 is set according to actual needs and used to determine whether to encode), the first coding feature after the first processing is recorded as y_hat. As can be seen from the above, the value range of y_hat is limited to [s, 2 n First, shift y_hat to the interval [0,2 n -1]: y_hat1 = y_hat-s, the size of y_hat1 is [C×H1×W1]. Taking an element y_hat1 in the first encoding feature y_hat as an example, the specific steps of converting the decimal y_hat1 tensor into the binary symbol bit_y include: let n1 = floor(n), n2 = n1+1, y_hat1 x,y,z The upper limit of the value m = integer (2 n )-1. The above threshold (denoted as thr) is greater than or equal to 0 and less than or equal to m. For 0≤y_hat1 x,y,z ≤thr, assign it an n1-bit codeword, and convert it into a binary symbol bit_y according to step 2 in direct binary encoding. x,y,z (i.e., the sub-bitstream in the first bitstream). x,y,z ≤m assigns it an n2-bit codeword and converts it into a binary symbol bit_y according to step 2 in direct binary encoding. x,y,z(i.e., the sub-bit stream in the first bit stream). And bit_y x,y,z Adjustment is performed (this adjustment can also be done before conversion to binary notation) to ensure that bit_y x,y,z The first n1 codewords and 0≤y_hat1 x,y,z The value range of n1-bit codewords in the case of ≤thr is not interleaved. When n<0.5, it is skipped and not encoded.

[0250] In general, the specific process of the above entropy coding can be understood as that the encoding device performs entropy coding on the first coding feature after the first processing to obtain the first code stream specifically including: obtaining multiple binary bits corresponding to the first coding feature after the first processing; selecting a first number of bits from the multiple binary bits to obtain the first code stream.

[0251] Among them, the entropy coding is direct binary, and the value range is [s,2 b In the case where the entropy coding is direct binary and the value range is [s,b-1+s], a first number of bits are selected from a plurality of binary bits to obtain a first code stream, including: selecting b bits from a plurality of binary bits to obtain the first code stream. In the case where the entropy coding is direct binary and the value range is [s,b-1+s], log2b bits are selected from a plurality of binary bits to obtain the first code stream.

[0252] In the case where the entropy coding is truncated binary, the coding device performs entropy coding on the first coding feature after the first processing to obtain the first code stream, specifically including: determining the second number according to a threshold, the threshold is obtained according to a value range; obtaining multiple binary bits corresponding to the first coding feature after the first processing; and selecting a second number of bits from the multiple binary bits to obtain the first code stream.

[0253] Specifically, in the range of [s,2 b -1+s], and the first coding feature is less than the threshold, the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed. floor(n) means rounding down the code control feature. The second number is floor(n). In the range of [s,2 b -1+s], and the first coding feature is greater than the threshold, the second number is floor(n)+1; when the value range is [s,b-1+s], and the first coding feature is less than the threshold, the threshold is -round(n)+s, floor(log2n) means rounding log2n down, and the second quantity is floor(log2n); when the value range is [s,b-1+s] and the first coding feature is greater than the threshold, the second quantity is floor(log2n)+1.

[0254] It can be understood that the above direct binary encoding and truncated binary encoding are just examples to describe the encoding process. In practical applications, there may be other expressions or other methods, which are not limited here.

[0255] It should be noted that the various situations of the above-mentioned first coding features, the various methods of obtaining code control features, the various methods of truncation processing and the various methods of entropy coding can be combined arbitrarily. Examples will be given later to describe the combination situations, which will not be expanded here.

[0256] In an embodiment of the present application, compared with the operation of allocating codewords according to probability in the prior art, the present application indicates that the first coding feature is mapped to a preset number of bits through a code control feature, and the code control feature is related to the input data / coding network. The first coding feature is mapped to a preset number of bits through a code control feature indication. That is, an entropy coding method that does not require code rate estimation through an entropy model can be used to encode the first coding feature based on the preset number of bits indicated by the code control feature to obtain a first code stream. Since the first code stream is obtained by mapping the first coding feature itself, rather than obtaining it based on the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint encoding and decoding of the channel source.

[0257] Optionally, after acquiring the first code stream, the encoding device sends the first code stream to the decoding device.

[0258] In addition, when the first coding feature is a residual feature, the coding device can also encode the quantized super-prior feature to obtain a second code stream. Figure 7 The second code stream shown is not described here any more. In this case, the encoding device may send the first code stream and the second code stream to the decoding device.

[0259] See also Fig. 9 , a flowchart of a decoding method provided by an embodiment of the present application, the application scenario to which the method is applicable is mainly a coding and decoding scenario, etc. It may include steps 901 to 903. Steps 901 to 903 are described in detail below.

[0260] Step 901: Obtain a first bit stream.

[0261] In the embodiment of the present application, there are multiple ways for the decoding device to obtain input data, which may be through collection / shooting, or by receiving data sent by other devices, or by selecting from a database, etc., which are not limited here.

[0262] Optionally, the decoding device receives a first code stream sent by the encoding device.

[0263] Step 902, obtaining code control features.

[0264] The decoding device obtains the code control feature. The description of the code control feature and the description of obtaining the code control feature can refer to the aforementioned Figure 5 The description of step 502 in the illustrated embodiment is similar and will not be repeated here.

[0265] It can be understood that, in addition to the manner in which the decoding device acquires the code control feature in a manner similar to the manner in which the encoding device acquires the code control feature in the aforementioned step 502, the decoding device can also receive the code control feature sent by the encoding device.

[0266] In which, when the decoding device receives the code control feature sent by the encoding device, the code control feature can be in the first code stream, in the second code stream, or in other code streams (other code streams and the first code stream and the second code stream can be called the total code stream).

[0267] Step 903: decode the first code stream based on the code control feature to obtain reconstructed data.

[0268] After the decoding device obtains the first code stream and the code control feature, it can decode the first code stream based on the code control feature to obtain reconstructed data. The decoding in this step is the same as the above Figure 5 The encoding in the illustrated embodiment corresponds to the encoding in the illustrated embodiment. That is, the decoding includes entropy decoding. Entropy decoding includes direct binary decoding and truncated binary decoding. Of course, if Figure 5 The encoding in the illustrated embodiment 1 includes a scaling process, and the decoding in this step also includes an inverse scaling process (ie, a process of dividing by a scaling factor).

[0269] Optionally, the above-mentioned reconstructed data has an association relationship with the input data. For example, the similarity between the reconstructed data and the input data is greater than a threshold. For another example, the difference between the corresponding positions of the reconstructed data and the input data (such as the pixel position of the image, etc.) is less than a threshold, etc., which is not specifically limited here.

[0270] Specifically, since the code control feature is used to indicate that the first coding feature is mapped to a preset number of bits. The decoding device can determine the first quantity corresponding to the first coding feature according to the code control feature, and the first quantity is used to reflect the number of bits used for transmission of the first coding feature. Furthermore, the decoding device can decode the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature. The decoding device then obtains the reconstructed data based on the first coding feature. The decoding device obtains the first quantity in the same manner as described above. Figure 5 In the illustrated embodiment, the encoding device obtains the first quantity based on the code control feature in the same manner, which will not be described in detail here.

[0271] It can be understood that the first number can be understood as the aforementioned Figure 5 b, log2b or the second quantity in the embodiment shown.

[0272] Optionally, the determining the first quantity based on the code control feature includes: determining a value range of the first coding feature based on the code control feature; determining the first quantity based on the value range and a coding type, and the coding type includes direct binary coding or truncated binary coding.

[0273] The following is an illustrative description of the specific circumstances of the above steps according to the different value ranges and encoding types. It can be understood that the following value ranges and encoding types are only examples. In actual applications, there may be other forms of value ranges and encoding types, which are not specifically limited here.

[0274] When entropy decoding is direct binary decoding, the value range is [s,2 b For example, when the entropy decoding is direct binary decoding and the value range is [s, b-1+s], the first number is log2b, b=x(n).

[0275] In the case where the entropy decoding is truncated binary decoding, b=n, and the first number is the second number. b -1+s]: when the first coding feature of the corresponding position is less than the threshold, the second number is k=floor(n) (e.g. n1 in the following example); when the first coding feature of the corresponding position is greater than the threshold, the second number is k+1 (e.g. n2 in the following example). For another example, in the case where the value range is [s,b-1+s]: when the first coding feature of the corresponding position is less than the threshold, the second number is k=floor(log2n); when the first coding feature of the corresponding position is greater than the threshold, the second number is k+1.

[0276] Correspondingly, the decoding device determines the first quantity according to the above description.

[0277] The entropy decoding in the embodiment of the present application is different from the aforementioned Figure 5 Corresponding to the entropy coding in the illustrated embodiment, the entropy decoding includes direct binary decoding or truncated binary decoding, which are described below respectively:

[0278] For example, in the case of direct binary decoding, the first bit stream is recorded as bit_y, the code control feature is recorded as n_hat(C×H1×W1), and steps 1 and 2 are performed one by one according to C×H1×W to obtain the tensor y_hat1. That is, steps 1 and 2 can be understood as operations performed on the elements of y_hat1. Step 1: Extract n_hat according to the number of bits described by the code control feature n_hat(C×H1×W1) x,y,z bits, and get the sub-bitstream bit_y x,y,z Step 2: Sub-bitstream bit_y x,y,z Convert from binary to decimal. Specifically, the bit_y x,y,z Directly use the last n1 or n2 bits of an integer of type int (the rest of the bits are 0). Corresponding to the example in the previous direct binary encoding, one way of step 2 is to shift in an integer of type int. Another way of step 2 is to use bit_y x,y,z The i-th digit (i≤n) of i , and then sum the results of all the bits to get y_hat1 x,y,z Step 3: Add s to y_hat1 to restore the original range, thus obtaining the reconstructed y_hat.

[0279] For example, in the case of truncated binary decoding, the first bit stream is recorded as bit_y, the code control feature is recorded as n(C×H1×W1), and steps 1 and 2 are performed one by one according to C×H1×W to obtain the tensor y_hat1. That is, steps 1 and 2 can be understood as operations performed on the elements of y_hat1. Step 1: Extract n1 bits or n2 bits according to the number of bits described by the code control feature n(C×H1×W1) (due to the non-interleaved processing of the encoding process, the specific number of bits can be based on the first n1 bits and n2 bits of the current position). x,y,z If it is n1 bits, then extract n1 bits from the bitstream to obtain the sub-bitstream bit_y x,y,z If it is n2 bits, it is corrected by shifting or other methods as the final sub-bit stream bit_y x,y,z . Get sub-bitstream bit_y x,y,z Afterwards, the sub-bitstream bit_y x,y,z Convert from binary to decimal. Specifically, the bit_y x,y,zDirectly use the last n1 or n2 bits of an integer of type int (the rest of the bits are 0). For example, by shifting in an integer of type int. Or by using bit_y x,y,z The i-th digit (i≤n) of i , and then sum the results of all the bits to get y_hat1 x,y,z Step 3: Add s to y_hat1 to restore the original range, thus obtaining the reconstructed y_hat.

[0280] Optionally, the decoding device decodes the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature, which may specifically include: the decoding device determines the bits to be decoded from the first code stream according to the first quantity and a preset rule, decodes the bits to be decoded, and aligns the decoded bits to be decoded with the size of the first coding feature to obtain the first coding feature.

[0281] It can be understood that size alignment is equivalent to the process of converting a one-dimensional feature into C×H×W (i.e., the size of the first encoded feature is C×H×W), which specifically corresponds to the encoding side. For example, if the encoding side uses a straightening function (such as a flatten function), the size alignment on the decoding side uses the inverse function of the straightening function.

[0282] Exemplarily, the decoding device first fills the bits to be decoded with W, and then fills with H and C.

[0283] Among them, the preset rule can be understood as a preset order of extracting the bits to be decoded from the first code stream (for example, extracting in sequence, extracting at a certain interval, etc.), etc., which can be set according to actual needs and is not specifically limited here.

[0284] In addition, since the first coding feature may be the second coding feature or the residual feature, there are multiple ways for the decoding device to obtain the reconstructed data based on the first coding feature.

[0285] In one possible implementation, that is, when the first coding feature is the second coding feature, the decoding device decodes the second coding feature through a decoding network to obtain reconstructed data.

[0286] In another possible implementation, that is, when the first coding feature is a residual feature, the decoding device first obtains the predicted mean. Then, based on the predicted mean and the first coding feature, the second coding feature is obtained (for example, the predicted mean and the first coding feature are added), and the second coding feature is decoded by the decoding network to obtain the reconstructed data. The way in which the decoding device obtains the second coding feature based on the predicted mean and the first coding feature corresponds to the way in which the encoding device obtains the first coding feature based on the second coding feature and the predicted mean.

[0287] Optionally, corresponding to the encoding side, if the encoding side includes a scaling process / scaling module, the decoding side includes an inverse scaling process / inverse scaling module. The scaling coefficient corresponding to the scaling process / scaling module and the inverse scaling coefficient corresponding to the inverse scaling process / inverse scaling module are reciprocal to each other. For example, if the scaling coefficient of the encoding side is C, the inverse scaling coefficient of the decoding side is 1 / C.

[0288] In addition, the method of obtaining the inverse scaling coefficient is similar to that on the encoding side and will not be repeated here.

[0289] Exemplarily, the structure of the decoding network can be as follows Fig.10 As shown, the decoding network includes a third deconvolution layer, a fourth deconvolution layer, a fifth deconvolution layer, a sixth deconvolution layer and a plurality of residual layers. The residual layers are alternately connected with the deconvolution layers. The input of the decoding network is the second encoding feature, and the output of the decoding network is the input data. Fig.10 In the example of , the third deconvolution layer can be specifically expressed as: deconv 5×5 192×96 2↑, that is, the convolution kernel size of the third deconvolution layer is 5×5, the number of input channels is 192, the number of output channels is 96, and the stride is 2. The residual layer can be specifically ResAU. The fourth deconvolution layer can be specifically expressed as: deconv 5×5 96×48 2↑, that is, the convolution kernel size of the fourth deconvolution layer is 5×5, the number of input channels is 96, the number of output channels is 48, and the stride is 2. The fifth deconvolution layer can be specifically expressed as: deconv 5×5 48×24 2↑, that is, the convolution kernel size of the fifth deconvolution layer is 5×5, the number of input channels is 48, the number of output channels is 24, and the stride is 2. The sixth deconvolution layer can be specifically expressed as: deconv 5×5 24×3 2↑, that is, the convolution kernel size of the sixth deconvolution layer is 5×5, the number of input channels is 24, the number of output channels is 3, and the stride is 2.

[0290] Understandably, Fig.10 This is just an example of a decoding network. In actual applications, the decoding network may also include more or fewer deconvolution layers and residual layers. The parameters in the deconvolution layer (such as convolution kernel size, number of channels, number of input channels, number of output channels, step size, etc.) may also be other cases. It may also include other network layers (such as pooling layers, activation layers), etc., which are not limited here.

[0291] In the embodiment of the present application, compared with the operation of allocating codewords according to probability in the prior art, the present application indicates that the first coding feature is mapped to a preset number of bits through a code control feature, and the code control feature is related to the input data / coding network. That is, the entropy decoding method that does not require code rate estimation through an entropy model can be used to decode the first code stream based on the preset number of bits indicated by the code control feature to obtain reconstructed data. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0292] Optionally, before acquiring the first code stream, the decoding device may also receive a second code stream from the encoding device, and decode the second code stream to obtain the super-a priori feature.

[0293] When the decoding device also obtains the second bitstream, it can be combined with the above-mentioned method of obtaining the code control feature based on the super prior feature. That is, after the decoding device decodes the second bitstream to obtain the super prior feature, it obtains the code control feature based on the super prior feature and the super prior decoding network.

[0294] It should be noted that the above Figures 5 to 10 In the illustrated embodiment, various situations of the first coding feature, various methods of obtaining the code control feature, various methods of truncation processing, and various methods of entropy coding can be arbitrarily combined. The following will provide an illustrative description of different combinations in conjunction with multiple embodiments.

[0295] See also Fig.11 , a coding and decoding method provided in an embodiment of the present application. The first coding feature in the method is a residual feature, and the method for obtaining the coding control feature is based on a method of super prior information. The method can be uniformly executed by one device, or can be jointly executed by a coding device and a decoding device, which is not specifically limited here.

[0296] Fig.11 The encoding process shown is as follows:

[0297] Step 1: Input the input data x into the encoding network to obtain the encoding feature y (ie, the second encoding feature in the aforementioned embodiment).

[0298] Step 2: Input the coding feature y into the super-prior coding network to obtain the super-prior feature z, and the super-prior feature is quantized to obtain the quantized super-prior feature z_hat. The quantized super-prior feature z_hat is entropy estimated by the factor entropy model, and then entropy encoded by the ANS algorithm to obtain the bit stream bit_z (i.e., the second bit stream in the aforementioned embodiment). The bit stream bit_z is reconstructed by the ANS algorithm and the factor entropy model, and z_hat is input into the super-prior decoding network to obtain the code control feature n, the predicted mean mean, and the scaling factor k.

[0299] Step 3: Subtract the encoded feature y from the predicted mean mean to obtain the residual feature res, that is, res = y-mean.

[0300] Step 4: The residual feature res is scaled by the scaling factor k to obtain res1. Specifically, the residual feature res is multiplied point by point by the scaling factor k to obtain res1.

[0301] Step 5: truncate res1 to obtain res2. For example, taking the clamp function as an example, the truncation process is to clamp the value range of res1 to the range of (-c+0.25, c+0.25) (that is, the floating point number c in the above embodiment is 0.25), and c is a truncation coefficient calculated using the code control feature n (for example, c=(2 n -1)*0.5).

[0302] Step 6: quantize res2 (for example, round it off to the nearest integer) to obtain res_hat. According to the restrictions in step 5 above, the number of values ​​of res_hat does not exceed the restrictions given by the code control feature, that is, 2 n indivual.

[0303] Step 7: Entropy encode res_hat to obtain bit_res (i.e., a case of the first bit stream in the above embodiment). The entropy encoding can be direct binary encoding or truncated binary encoding. For specific operations, please refer to the above Figure 5 The description in the illustrated embodiment will not be repeated here.

[0304] Step 8: Transmit / store the second code stream bit_z and the first code stream bit_res.

[0305] Fig.11 The decoding process shown is as follows:

[0306] Step 1: Get the second bit stream bit_z and the first bit stream bit_res.

[0307] Step 2: Perform ANS decoding on the second bit stream bit_z through the factor entropy model to obtain the reconstructed z_hat.

[0308] Step 3: Input the reconstructed z_hat into the super-prior decoding network to obtain the code control feature n, the predicted mean mean and the scaling factor k.

[0309] Step 4: Under the constraint of the code control feature n, entropy decoding is performed on the first bit stream bit_res to obtain res_hat. The entropy decoding can be direct binary decoding or truncated binary decoding. For specific operations, please refer to the above Figure 5 The description in the illustrated embodiment will not be repeated here.

[0310] Step 5: res_hat is inversely scaled by the scaling factor k to obtain res3. Specifically, res_hat is divided by the scaling factor k to obtain res3.

[0311] Step 6: Add res3 and the predicted mean mean to obtain the reconstructed coding feature y1.

[0312] Step 7: Input the encoded feature y1 into the decoding network to obtain the reconstructed data.

[0313] In the embodiment of the present application, compared with the operation of allocating codewords according to probability in the prior art, the present application uses the code control feature to constrain the number of bits used for transmission of the residual feature, which is conducive to the joint encoding and decoding of the channel source and parallel encoding and decoding. In addition, in the case of scaling processing, the scaling coefficient and the truncation coefficient are learned separately, which can make the encoding and decoding effect better.

[0314] See also Fig.12 , another encoding and decoding method provided by an embodiment of the present application. The first encoding feature in the method is the second encoding feature. The method can be uniformly executed by one device, or can be jointly executed by an encoding device and a decoding device, which is not specifically limited here.

[0315] Fig.12 The encoding process shown is as follows:

[0316] Step 1: Input the input data x into the encoding network to obtain the encoding feature y (ie, the second encoding feature in the aforementioned embodiment).

[0317] Step 2: The coding feature y is scaled by the scaling factor k to obtain y1. Specifically, the coding feature y is multiplied point by point by the scaling factor k to obtain y1.

[0318] Step 3: truncate y1 to obtain y2. For example, taking the clamp function as an example, the truncation process is to clamp the value range of y1 to the range of (-c+0.25, c+0.25) (i.e., the floating point number c in the above embodiment is 0.25), where c is a truncation coefficient calculated using the code control feature n (e.g., c=(2 n -1)*0.5).

[0319] Step 4: quantize y2 (for example, round it off to the nearest integer) to obtain y_hat. According to the restrictions of step 5 above, the number of values ​​of y_hat does not exceed the limit given by the code control feature, that is, 2 n indivual.

[0320] Step 5: Entropy coding is performed on y_hat to obtain bit_y (i.e., another case of the first bit stream in the above embodiment). The entropy coding can be direct binary coding or truncated binary coding. For specific operations, please refer to the above Figure 5 The description in the illustrated embodiment will not be repeated here.

[0321] Step 6: Transmit / store the first bit stream bit_y.

[0322] Fig.12 The decoding process shown is as follows:

[0323] Step 1: Get the first bit stream bit_y.

[0324] Step 2: Under the constraint of the code control feature n, entropy decoding is performed on the first bit stream bit_y to obtain res_y. The entropy decoding can be direct binary decoding or truncated binary decoding. For specific operations, please refer to the above Figure 5 The description in the illustrated embodiment will not be repeated here.

[0325] Step 3: y_hat is inversely scaled by the scaling factor k to obtain y3. Specifically, y_hat is divided by the scaling factor k to obtain y3.

[0326] Step 4: Input y3 into the decoding network to obtain reconstructed data.

[0327] In the embodiment of the present application, on the one hand, compared with the operation of allocating codewords according to probability in the prior art, the present application uses code control features to constrain the number of bits used for transmission of residual features, which is conducive to channel source joint encoding and decoding and parallel encoding and decoding. On the other hand, the input image is directly converted into a code stream through differentiable operations. And the conventional AI encoding and Fig.11 The illustrated embodiment scheme relies on super-prior information, and the technical framework is very simple.

[0328] Understandably, Fig.11 and Fig.12 The encoding and decoding process shown is only an example of a combination, and the scaling process and the inverse scaling process may not be required, and there may be other ways to obtain the code control feature (such as the aforementioned Figure 5 The description in the illustrated embodiment) etc. are not specifically limited here.

[0329] In addition, the scaling factor k may be equal to or different from the truncation factor c. In the case of being equal, c = k = (2 n -1)*0.5. In this case, the encoding and decoding efficiency is better. In the case of inequality, c=(2 n -1)*0.5, k is the output of the super priori decoding network. Of course, k can also be obtained by other means (such as joint encoding and decoding network training), which is not limited here.

[0330] In addition, in order to more intuitively see the beneficial effects of the methods provided in the embodiments of the present application in different combinations, the performance results of different methods on the Kodak dataset are compared and described below.

[0331] Fig.13 The rate-distortion curves of the method provided in the embodiment of the present application under different combinations, wherein the horizontal axis is the average number of bits per pixel (BPP) and the vertical axis is the peak signal-to-noise ratio (PSNR). Fig.13 Curve 1 in the figure represents the prior art, curve 2 represents the case where c is not equal to k in the method provided in the embodiment of the present application, curves 3 and 4 represent the case where c is equal to k in the method provided in the embodiment of the present application, and curve 3 adopts the truncated binary method, and curve 4 adopts the direct binary method.

[0332] It can be seen that although the rate-distortion performance of this embodiment is lower than that of the conventional solution, the image can be mapped into a code stream in a differentiable manner, which is beneficial to joint optimization and parallelization.

[0333] The encoding and decoding method in the embodiment of the present application is described above. The device in the embodiment of the present application is described below. Fig.14 , an embodiment of the encoding device in the embodiment of the present application includes:

[0334] An acquisition unit 1401 is used to acquire input data;

[0335] The acquisition unit 1401 is further used to acquire a code control feature, where the code control feature is used to indicate that the first code feature is mapped to a preset number of bits, the code control feature is related to the input data, or the code control feature is related to a neural network, the neural network includes at least one of a coding network or a decoding network, the coding network is used to acquire a second code feature of the input data, the decoding network is used to reconstruct the input data, and the first code feature is obtained according to the second code feature;

[0336] The encoding unit 1402 is configured to encode the first encoding feature based on the code control feature to obtain a first code stream.

[0337] Optionally, the encoding unit 1402 is specifically configured to perform a first processing on the first coding feature based on the coding feature, wherein the first processing includes a truncation processing and a quantization processing, and the truncation processing is used to limit the value range of the first coding feature to [s,2 b -1+s] or [s,b-1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; the encoding unit 1402 is specifically used to perform entropy encoding on the first coding feature after the first processing to obtain a first code stream.

[0338] Optionally, the truncation process includes at least one of the following: a clamp function, a S-type function, a periodic function, a piecewise linear function, etc.

[0339] Optionally, the encoding unit 1402 is specifically used to obtain multiple binary bits corresponding to the first encoding feature after the first processing; the encoding unit 1402 is specifically used to select a first number of bits from the multiple binary bits to obtain a first code stream.

[0340] Optionally, the encoding unit 1402 is specifically configured to: b -1+s], and when the entropy coding is direct binary coding, the coding unit 1402 is specifically used to select b bits from multiple binary bits to obtain a first code stream; when the value range is [s,b-1+s], and the entropy coding is direct binary coding, the coding unit 1402 is specifically used to select log2b bits from multiple binary bits to obtain a first code stream.

[0341] Optionally, in the case where the entropy coding is truncated binary coding, the coding unit 1402 is specifically used to determine the second number according to a threshold, and the threshold is obtained according to a value range; the coding unit 1402 is specifically used to obtain multiple binary bits corresponding to the first coding feature after the first processing; the coding unit 1402 is specifically used to select a second number of bits from multiple binary bits to obtain a first code stream.

[0342] Optionally, in the range [s,2 b -1+s], and the first coding feature is less than the threshold, the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed. floor(n) means rounding down the code control feature. The second number is floor(n). In the range of [s,2 b -1+s], and the first coding feature is greater than the threshold, the second number is floor(n)+1; when the value range is [s,b-1+s], and the first coding feature is less than the threshold, the threshold is -round(n)+s, floor(log2n) means rounding log2n down, and the second quantity is floor(log2n); when the value range is [s,b-1+s] and the first coding feature is greater than the threshold, the second quantity is floor(log2n)+1.

[0343] Optionally, the code control feature is a parameter in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0344] Optionally, the acquisition unit 1401 is specifically used to determine the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes an encoding network and a decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0345] Optionally, the first coding feature is the second coding feature; the encoding device also includes: a sending unit 1403, used to send a first code stream.

[0346] Optionally, the first coding feature is a super-prior feature, and the super-prior feature is obtained according to the second coding feature and the super-prior coding network.

[0347] Optionally, the acquisition unit 1401 is specifically used to input the second coding feature into the super-prior coding network to obtain the super-prior feature; the acquisition unit 1401 is specifically used to obtain the code control feature based on the super-prior feature, the factor entropy model and the super-prior decoding network.

[0348] Optionally, the first coding feature is the difference between the second coding feature and the predicted mean, and the predicted mean is obtained based on the super prior feature, the factor entropy model and the super prior decoding network; the encoding unit 1402 is also used to encode the quantized super prior feature to obtain the second code stream; the sending unit 1403 is also used to send the first code stream and the second code stream.

[0349] In this embodiment, the operations performed by each unit in the encoding device are the same as those described above. Figures 1 to 13 The description in the illustrated embodiment is similar and will not be repeated here.

[0350] In this embodiment, the encoding unit 1402 in this embodiment maps the first coding feature to a preset number of bits through the code control feature indication. That is, in an entropy coding method that does not require code rate estimation through an entropy model, the first coding feature can be encoded based on the preset number of bits indicated by the code control feature to obtain a first code stream. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0351] See also Fig.15 , an embodiment of the decoding device in the embodiment of the present application includes:

[0352] The acquisition unit 1501 is used to acquire a first bit stream;

[0353] The acquisition unit 1501 is further used to acquire a code control feature, where the code control feature is used to indicate that the first code feature is mapped to a preset number of bits, the code control feature is related to the input data, or the code control feature is related to a neural network, the neural network includes at least one of a coding network or a decoding network, the coding network is used to acquire a second code feature of the input data, the decoding network is used to reconstruct the input data, and the first code feature is obtained according to the second code feature;

[0354] The decoding unit 1502 is configured to decode the first code stream based on the code control feature to obtain reconstructed data.

[0355] Optionally, the decoding unit 1502 is specifically used to determine a first quantity based on the code control feature, and the bits of the first quantity are used to represent the bits occupied by the first coding feature; the decoding unit 1502 is specifically used to decode the first code stream based on the first quantity and the size of the first coding feature to obtain the first coding feature; the decoding unit 1502 is specifically used to obtain reconstructed data based on the first coding feature.

[0356] Optionally, the decoding unit 1502 is specifically used to determine the bits to be decoded from the first code stream according to the first quantity; the decoding unit 1502 is specifically used to decode the bits to be decoded; the decoding unit 1502 is specifically used to align the decoded bits to be decoded with the size to obtain the first coding feature.

[0357] Optionally, the decoding unit 1502 is specifically used to determine a value range of the first coding feature based on the code control feature; the decoding unit 1502 is specifically used to determine the first quantity based on the value range and the coding type, and the coding type includes direct binary coding or truncated binary coding.

[0358] Optionally, the decoding unit 1502 is specifically configured to determine, based on the code control feature, that the value range of the first coding feature is [s,2 b -1+s], b=x(n) or b=n, s represents a floating point number, n represents a code control feature, and x(n) represents integer processing of the code control feature; the value range is [s,2 b -1+s], and when the entropy coding of the first bit stream is direct binary coding, the above decoding unit 1502 is specifically used to determine that the first number is b; in the value range of [s,2 b -1+s], and the entropy coding of the first bit stream is truncated binary coding, the above decoding unit 1502 is specifically used to determine the threshold value based on the value range, and the threshold value is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, floor(n) indicates rounding down the code control feature; the above-mentioned decoding unit 1502 is specifically used to determine that the first number is floor(n), and the first coding feature obtained based on floor(n) is less than the threshold value. If the first coding feature obtained based on floor(n) is greater than the threshold value, the above-mentioned decoding unit 1502 is specifically used to adjust the first number to floor(n)+1.

[0359] Optionally, the decoding unit 1502 is specifically used to determine that the value range of the first coding feature is [s, b-1+s] based on the code control feature, b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; when the value range is [s, b-1+s] and the entropy coding of the first code stream is direct binary coding, the above-mentioned decoding unit 1502 is specifically used to determine the first number as log2b; when the value range is [s, b-1+s] and the entropy coding of the first code stream is truncated binary coding, the above-mentioned decoding unit 1502 is specifically used to determine the threshold based on the value range, and the threshold is -round(n)+s, floor(log2n) means rounding log2n down; the above-mentioned decoding unit 1502 is specifically used to determine that the first quantity is floor(log2n), and the first coding feature obtained based on floor(log2n) is less than the threshold value. If the first coding feature obtained based on floor(log2n) is greater than the threshold value, the above-mentioned decoding unit 1502 is specifically used to adjust the first quantity to floor(log2n)+1.

[0360] Optionally, the decoding unit 1502 is specifically configured to decode the first coding feature through a decoding network to obtain reconstructed data.

[0361] Optionally, the acquisition unit 1501 is also used to obtain a predicted mean, which is obtained based on the second bit stream, the factor entropy model and the super decoding network; the decoding unit 1502 is specifically used to obtain the second coding feature based on the predicted mean and the first coding feature; the decoding unit 1502 is specifically used to decode the second coding feature through the decoding network to obtain reconstructed data.

[0362] Optionally, the code control feature is a parameter in a neural network, the neural network includes at least one of an encoding network or a decoding network, and the decoding network is used to restore input data.

[0363] Optionally, the acquisition unit 1501 is specifically used to determine the code control feature based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes a coding network and a decoding network, the coding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

[0364] Optionally, the acquisition unit 1501 is further used to acquire a second code stream; the decoding unit 1502 is further used to decode the second code stream to obtain a super-prior feature; the acquisition unit 1501 is specifically used to obtain the code control feature based on the super-prior feature and the super-prior decoding network.

[0365] Optionally, the acquisition unit 1501 is specifically configured to receive a code control feature sent by a coding device.

[0366] In this embodiment, the operations performed by each unit in the decoding device are the same as those described above. Figures 1 to 13 The description in the illustrated embodiment is similar and will not be repeated here.

[0367] In this embodiment, compared with the operation of allocating codewords according to probability in the prior art, the decoding unit 1502 in this embodiment maps the first coding feature to a preset number of bits through the code control feature indication, and the code control feature is related to the input data / coding network. That is, the entropy decoding method that does not require code rate estimation through an entropy model can decode the first code stream based on the preset number of bits indicated by the code control feature to obtain reconstructed data. Since the first code stream is obtained by mapping the first coding feature itself, rather than by the probability of occurrence of elements in the first coding feature, the conversion between the first coding feature and the first code stream is differentiable, which is beneficial to the joint coding and decoding of the channel source.

[0368] See also Fig.16 , a schematic diagram of the structure of another encoding device provided by the present application. The encoding device may include a processor 1601, a memory 1602 and a communication port 1603. The processor 1601, the memory 1602 and the communication port 1603 are interconnected via a line. The memory 1602 stores program instructions and data.

[0369] The memory 1602 stores the aforementioned Figures 1 to 13 In the corresponding implementation manner shown, the steps executed by the encoding device correspond to program instructions and data.

[0370] Processor 1601, used to execute the above Figures 1 to 13 The steps performed by the encoding device shown in any of the embodiments shown.

[0371] Communication port 1603 can be used to receive and send data, and to perform the above Figures 1 to 13 The steps related to obtaining, sending and receiving in any of the embodiments shown.

[0372] In one implementation, the encoding device may include Fig.16 More or fewer components are merely illustrative in this application and are not limiting.

[0373] See also Fig.17 , a schematic diagram of the structure of another decoding device provided by the present application. The decoding device may include a processor 1701, a memory 1702 and a communication port 1703. The processor 1701, the memory 1702 and the communication port 1703 are interconnected via a line. The memory 1702 stores program instructions and data.

[0374] The memory 1702 stores the aforementioned Figures 1 to 13 In the corresponding implementation manner shown, the steps executed by the decoding device correspond to program instructions and data.

[0375] Processor 1701, used to execute the above Figures 1 to 13 The steps performed by the decoding device in any of the illustrated embodiments.

[0376] Communication port 1703 can be used to receive and send data, and to perform the above Figures 1 to 13 The steps related to obtaining, sending and receiving in any of the embodiments shown.

[0377] In one implementation, the decoding device may include a Fig.17 More or fewer components are merely illustrative in this application and are not limiting.

[0378] An embodiment of the present application also provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the possible implementation method of the image processing device (such as an encoding device or a decoding device) in the aforementioned embodiment.

[0379] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation mode of the above-mentioned image processing device.

[0380] The embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned image processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data for the at least one processor. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the terminal device. The chip system may be composed of chips, or may include chips and other discrete devices.

[0381] The present application also provides a readable storage medium, which stores the Figures 1 to 13 The code stream is obtained by encoding using the method in the illustrated embodiment.

[0382] The embodiment of the present application also provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; at least one storage medium is used to store a code stream; the code stream is a code stream generated by an encoder according to the aforementioned Figures 1 to 13 The method in the illustrated embodiment is used for encoding.

[0383] The embodiment of the present application also provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, the code stream being the code stream of the encoder according to the above Figures 1 to 13 The method in the illustrated embodiment is used for encoding.

[0384] The embodiment of the present application also provides a system for distributing code streams, the system comprising at least one storage medium and a video stream device; at least one storage medium is used to store the code stream, the code stream is the code stream of the encoder according to the above Figures 1 to 13 The method in the illustrated embodiment is used to encode the video stream obtained by the method; the video stream device is used to respond to the request of the decoder so that the target code stream in at least one storage medium can be sent to the decoder.

[0385] The embodiment of the present application also provides a method for distributing a bitstream, the method comprising: receiving a first request; selecting a target bitstream from at least one storage medium in response to the first request; sending the target bitstream to a destination device; at least one storage medium is used to store the bitstream, the bitstream being the codestream generated by the encoder according to the aforementioned Figures 1 to 13 The method in the illustrated embodiment is used for encoding.

[0386] The embodiment of the present application also provides a system for processing a code stream, the system comprising an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, the code stream is the encoder according to the above Figures 1 to 13 The method in the illustrated embodiment is used to encode; the encoder device is used to store one or more code streams in one or more storage media; or the encoder device is used to encapsulate one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to a destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0387] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0388] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0389] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0390] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0391] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A decoding method, characterized in that: The method comprises: Get the first code stream; Acquire a code control feature, where the code control feature is used to indicate that a first code feature is mapped to a preset number of bits, where the code control feature is related to input data, or where the code control feature is related to a neural network, where the neural network includes at least one of an encoding network or a decoding network, where the encoding network is used to acquire a second code feature of the input data, where the decoding network is used to reconstruct the input data, and where the first code feature is obtained based on the second code feature; The first code stream is decoded based on the code control feature to obtain reconstructed data.

2. The method according to claim 1, characterized in that The decoding the first code stream based on the code control feature to obtain reconstructed data includes: Determine a first quantity based on the code control feature, where bits of the first quantity are used to represent bits occupied by the first coding feature; decoding the first code stream based on the first number and the size of the first coding feature to obtain the first coding feature; The reconstruction data is obtained based on the first encoding feature.

3. The method according to claim 2, characterized in that The determining the first quantity based on the code control feature comprises: Determining a value range of the first coding feature based on the code control feature; The first number is determined based on the value range and the encoding type, where the encoding type includes direct binary encoding or truncated binary encoding.

4. The method according to claim 2 or 3, characterized in that: The decoding of the first code stream based on the first number and the size of the first coding feature to obtain the first coding feature includes: Determine bits to be decoded from the first bit stream according to the first number; Decoding the to-be-decoded bits; The decoded bits to be decoded are aligned with the size to obtain the first encoding feature.

5. The method according to any one of claims 2 to 4, characterized in that The method further comprises: Obtaining a predicted mean, where the predicted mean is obtained according to the second bitstream, the factor entropy model, and the super decoding network; The acquiring the reconstructed data based on the first coding feature comprises: Obtaining the second coding feature based on the predicted mean and the first coding feature; The second encoded feature is decoded by the decoding network to obtain the reconstructed data.

6. The method according to any one of claims 1 to 5, characterized in that The code control feature is a parameter in a neural network, and the neural network includes the encoding network and / or the decoding network, and the decoding network is used to restore the input data.

7. The method according to any one of claims 1 to 5, characterized in that The obtaining of the code control feature comprises: The code control feature is determined based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / the second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes the encoding network and the decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

8. The method according to any one of claims 1 to 5, characterized in that Before obtaining the first code stream, the method further includes: Get the second code stream; Decoding the second code stream to obtain a super-prior feature; The obtaining of code control features includes: The code control feature is obtained based on the super a priori feature and the super a priori decoding network.

9. The method according to any one of claims 1 to 5, characterized in that The obtaining of code control features includes: The coding feature sent by the encoding device is received.

10. The method according to claim 3, characterized in that: The determining a value range of the first coding feature based on the code control feature includes: Based on the code control feature, the value range of the first coding feature is determined to be [s,2 b -1+s], b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; The determining the first quantity based on the value range and the encoding type includes: In the range of [s,2 b -1+s], and when the encoding type is the direct binary encoding, determining the first number to be b; In the value range [s,2 b -1+s], and the encoding type is the truncated binary encoding, a threshold is determined based on the value range, and the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, where floor(n) means rounding down the code control feature; Determine that the first number is floor(n), and the first coding feature obtained based on floor(n) is less than the threshold value. If the first coding feature obtained based on floor(n) is greater than the threshold value, adjust the first number to floor(n)+1.

11. The method according to claim 3, characterized in that The determining a value range of the first coding feature based on the code control feature includes: Determine based on the code control feature that the value range of the first coding feature is [s, b-1+s], b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; The determining the first quantity based on the value range and the encoding type includes: When the value range is [s, b-1+s] and the encoding type is the direct binary encoding, determining that the first number is log2b; When the value range is [s, b-1+s] and the encoding type is the truncated binary encoding, a threshold is determined based on the value range, and the threshold is floor(log2n) means rounding log2n down; Determine that the first number is floor(log2n), and the first coding feature obtained based on floor(log2n) is less than the threshold; if the first coding feature obtained based on floor(log2n) is greater than the threshold, adjust the first number to floor(log2n)+1.

12. A coding method, characterized in that: The method comprises: Get input data; Acquire a code control feature, where the code control feature is used to indicate that a first code feature is mapped to a preset number of bits, where the code control feature is related to input data, or where the code control feature is related to a neural network, where the neural network includes at least one of an encoding network or a decoding network, where the encoding network is used to acquire a second code feature of the input data, where the decoding network is used to reconstruct the input data, and where the first code feature is obtained based on the second code feature; The first coding feature is encoded based on the coding feature to obtain a first code stream.

13. The method according to claim 12, characterized in that The encoding the first coding feature based on the code control feature to obtain a first code stream includes: The first coding feature is subjected to a first processing based on the code control feature, wherein the first processing includes a truncation processing and a quantization processing, and the truncation processing is used to limit the value range of the first coding feature to [s,2 b -1+s] or [s,b-1+s], b=x(n) or b=n, s represents a floating point number, n represents the code control feature, and x(n) represents integer processing of the code control feature; The first coding feature after the first processing is entropy encoded to obtain the first code stream.

14. The method according to claim 13, characterized in that The truncation process includes at least one of the following: a clamp function, a S-type function, a periodic function, and a piecewise linear function.

15. The method according to claim 13 or 14, characterized in that The entropy encoding the first coding feature after the first processing to obtain the first code stream includes: Obtaining a plurality of binary bits corresponding to the first coding feature after the first processing; A first number of bits are selected from the plurality of binary bits to obtain the first code stream.

16. The method according to claim 15, characterized in that The step of selecting a first number of bits from the plurality of binary bits to obtain the first code stream includes: In the value range [s,2 b -1+s], and when the entropy coding is direct binary coding, selecting b bits from the plurality of binary bits to obtain the first code stream; When the value range is [s, b-1+s] and the entropy coding is direct binary coding, log2b bits are selected from the multiple binary bits to obtain the first code stream.

17. The method according to claim 13 or 14, characterized in that In a case where the entropy coding is truncated binary coding, performing entropy coding on the first coding feature after the first processing to obtain the first code stream includes: Determine a second quantity according to a threshold value, wherein the threshold value is obtained according to the value range; Obtaining a plurality of binary bits corresponding to the first coding feature after the first processing; The first code stream is obtained by selecting the second number of bits from the plurality of binary bits.

18. The method according to claim 17, characterized in that In the range of [s,2 b -1+s], and the first coding feature is less than the threshold, the threshold is 2 floor(n)+1 -round(2 n )+s,round(2 n ) indicates that 2 n Rounding is performed, floor(n) means that the code control feature is rounded down, and the second number is floor(n); in the value range [s,2 b -1+s], and the first coding feature is greater than the threshold, the second number is floor(n)+1; When the value range is [s, b-1+s] and the first coding feature is less than the threshold, the threshold is floor(log2n) means rounding log2n down, and the second number is floor(log2n); when the value range is [s,b-1+s] and the first coding feature is greater than the threshold, the second number is floor(log2n)+1.

19. The method according to any one of claims 12 to 18, characterized in that The code control feature is a parameter in a neural network, and the neural network includes the encoding network and / or the decoding network, and the decoding network is used to restore the input data.

20. The method according to any one of claims 12 to 18, characterized in that The obtaining of the code control feature comprises: The code control feature is determined based on a mapping relationship, the mapping relationship is used to represent the association relationship between the input data / the second coding feature and the code control feature, the mapping relationship is a preset mapping relationship or is obtained through training based on a loss function of a neural network, the neural network includes the encoding network and the decoding network, the encoding network is used to obtain the coding features of the training data, the decoding network is used to restore the training data, the loss function includes a first loss term and a second loss term, the first loss term is used to represent the difference between the output data of the neural network obtained with the training data as input and the training data, the second loss term is used to represent the total number of bits used for transmitting the coding feature to be transmitted, the total number of bits is obtained based on the code control feature and the coding feature to be transmitted, and the coding feature to be transmitted is related to the coding feature of the training data.

21. The method according to any one of claims 12 to 20, characterized in that The first coding feature is the second coding feature; The method further comprises: The first code stream is sent.

22. The method according to any one of claims 12 to 20, characterized in that The first coding feature is a super priori feature, and the super priori feature is obtained according to the second coding feature and a super priori coding network.

23. The method according to any one of claims 12 to 18, characterized in that The obtaining of the code control feature comprises: Inputting the second coding feature into a super priori coding network to obtain a super priori feature; The code control feature is obtained based on the super a priori feature, the factor entropy model and the super a priori decoding network.

24. The method according to claim 23, characterized in that The first coding feature is a difference between the second coding feature and a predicted mean, wherein the predicted mean is obtained according to the super prior feature, the factor entropy model and the super prior decoding network; The method further comprises: Encoding the quantized super-prior features to obtain a second bit stream; The first code stream and the second code stream are sent.

25. A decoding device, characterized in that: The decoding device comprises: An acquisition unit, used for acquiring a first code stream; an acquisition unit, further used to acquire a code control feature, the code control feature is used to indicate that a first coding feature is mapped to a preset number of bits, the code control feature is related to input data, or the code control feature is related to a neural network, the neural network includes at least one of an encoding network or a decoding network, the encoding network is used to acquire a second coding feature of the input data, the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature; A decoding unit is used to decode the first code stream based on the code control feature to obtain reconstructed data.

26. A coding device, characterized in that The encoding device comprises: An acquisition unit, used for acquiring input data; an acquisition unit, further used to acquire a code control feature, the code control feature is used to indicate that a first coding feature is mapped to a preset number of bits, the code control feature is related to input data, or the code control feature is related to a neural network, the neural network includes at least one of an encoding network or a decoding network, the encoding network is used to acquire a second coding feature of the input data, the decoding network is used to reconstruct the input data, and the first coding feature is obtained according to the second coding feature; An encoding unit is used to encode the first encoding feature based on the code control feature to obtain a first code stream.

27. A decoding device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store a program or an instruction, when the program or the instruction is executed by the processor, the decoding device executes the method according to any one of claims 12 to 24.

28. A coding device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding device executes the method as described in any one of claims 1 to 11.

29. A coding and decoding system, characterized in that: The encoding and decoding system comprises the decoding device according to claim 27 and / or the encoding device according to claim 28.

30. A computer-readable storage medium, characterized in that: The medium stores instructions, and when the instructions are executed by a computer, the method according to any one of claims 1 to 24 is implemented.

31. A computer program product, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 24.

32. A readable storage medium, characterized in that: The readable medium stores a code stream encoded according to any one of the encoding methods of claims 12 to 24.

33. A device for storing a code stream, characterized in that: comprising at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the code stream; The code stream is obtained by encoding by an encoder according to any encoding method according to claims 12 to 24.

34. A method for storing a code stream, characterized in that: include: Receiving a code stream via a communication interface; The code stream is stored in one or more storage media, wherein the code stream is encoded by an encoder according to any encoding method according to claims 12 to 24.

35. A system for distributing code streams, characterized in that: including at least one storage medium and a video streaming device; The at least one storage medium is used to store a code stream, wherein the code stream is obtained by encoding by an encoder according to any one of the encoding methods of claims 12 to 24; The video stream device is used to respond to a request of a decoder so that the target code stream in the at least one storage medium can be sent to the decoder.

36. A method for distributing code streams, characterized in that: include: receiving a first request; In response to the first request, selecting a target bitstream from at least one storage medium; Sending the target code stream to a destination device; The at least one storage medium is used to store a code stream, where the code stream is encoded by an encoder according to any one of the encoding methods according to claims 12 to 24.

37. A system for processing a code stream, characterized in that: It includes an image source device, an encoder device, one or more storage media and a destination device; The image source device is used to provide image data; The encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to any one of the encoding methods according to claims 12 to 24; The encoder device is used to store the one or more code streams in one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; The destination device is used to decapsulate the transport code stream to obtain the one or more code streams; The destination device is used to decode the one or more code streams to obtain decoded data.

Images