Entropy coding method and apparatus

Abstract

This application provides an entropy encoding / decoding method and apparatus. The entropy encoding method of this application includes: acquiring data to be encoded, wherein the data to be encoded is non-first-order encoded data among multiple data contained in a current data stream; acquiring reference information, wherein the reference information includes at least one of first context information and first side information, wherein the first context information is obtained by inputting at least one encoded data into an attention decoding network, and the first side information is obtained by inputting the multiple data into an attention encoding network; estimating a first estimated probability distribution based on the reference information; and performing entropy encoding on the data to be encoded based on the first estimated probability distribution to obtain a first bitstream. This application can improve the transmission efficiency of multiple data contained in a current data stream.

Description

Technical Field

[0001] This application relates to the field of data compression technology based on artificial intelligence (AI), and in particular to an entropy encoding and decoding method and apparatus. Background Technology

[0002] Video encoding (video encoding and decoding) is widely used in digital video applications, such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, digital versatile discs (DVD) and Blu-ray discs, video content capture and editing systems, and security applications for portable cameras.

[0003] Even in shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be sent over bandwidth-constrained networks or transmitted otherwise. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware at the source side to encode video data before transmission or storage, thereby reducing the amount of data needed to represent the digital video image. The compressed data is then received at the destination by video decompression equipment. Given limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression ratios with minimal impact on image quality.

[0004] In recent years, applying deep learning to image or video encoding and decoding has gradually become a trend. Related technologies employ a pre-set fixed probability distribution or determine the probability distribution using a trained learnable model, and then encode / decode the data based on this probability distribution. However, the accuracy of the probability distribution obtained by these methods is relatively low, leading to high entropy coding overhead and consequently low data transmission efficiency. Summary of the Invention

[0005] This application provides an entropy encoding / decoding method and apparatus to improve the accuracy of the estimated probability distribution of the data to be encoded, reduce the bit rate in the entropy encoding / decoding process, and thus reduce the entropy encoding / decoding overhead.

[0006] In a first aspect, this application provides an entropy coding method, the method comprising: acquiring data to be encoded, the data to be encoded being non-first-order encoded data among multiple data contained in a current data stream; acquiring reference information, the reference information including at least one of first context information and first side information, the first context information being obtained by inputting at least one encoded data into an attention decoding network, and the first side information being obtained by inputting the multiple data into an attention coding network; estimating a first estimated probability distribution based on the reference information; and performing entropy coding on the data to be encoded based on the first estimated probability distribution to obtain a first bitstream.

[0007] Here, the encoded data refers to the data that the encoder has already entropy encoded among multiple data. Since there is no encoded data yet when entropy encoding is performed on the first data of the current data stream, the data to be encoded needs to be non-first data of the current data stream so that the first context information can be extracted.

[0008] The first estimated probability distribution obtained based on reference information may include at least one estimated probability parameter. For example, this at least one estimated probability parameter may include a mean and a variance, which form a Gaussian distribution. The encoder can calculate the probability value of the data to be encoded based on the first estimated probability distribution, and then perform entropy encoding on the data to be encoded based on this probability value. The first bitstream obtained after entropy encoding may be in binary format.

[0009] Multiple data points can also be referred to as multiple data units. These data points can include video data, image data, audio data, integer data, and other data requiring compression / decompression. This application does not limit the data type. Each data point corresponds to a location information point, and the data to be encoded is not located at the first position among the multiple data points.

[0010] In this entropy coding method, the self-attention decoding network is a neural network with a self-attention mechanism (i.e., including a self-attention structure). The self-attention mechanism is a variant of the attention mechanism, possessing a global receptive field and capable of effectively capturing the internal correlations of data or features. The self-attention decoding network obtains the weights of all encoded input data and the data to be encoded, and then weights all or part of the encoded input data using the corresponding weights to obtain the first context information. This improves the utilization rate of encoded data during the acquisition of the first context information. When estimating the first probability distribution using the first context information, it improves the accuracy of the first estimated probability distribution, further reducing the code rate in the entropy coding process, thereby further reducing the entropy coding overhead.

[0011] Self-attention encoding networks possess a global receptive field, enabling them to determine the correlation between all input data and the data to be encoded. This correlation can be represented as the weights of all input data relative to the data to be encoded. After obtaining the weights of all input data relative to the data to be encoded, the self-attention encoding network weights the corresponding data according to these weights to obtain the first-side information.

[0012] Optionally, the self-attention coding network can weight all or part of the input data using appropriate weights to obtain the first-side information. This improves the data utilization rate during the acquisition of the first-side information. When subsequently using the first-side information to estimate the first estimated probability distribution, the accuracy of the obtained first estimated probability distribution can be further improved, further reducing the code rate in the entropy coding process, and thus further reducing the entropy coding overhead.

[0013] In addition to including at least one of the first context information and the first side information, the reference information may also include at least one of the second context information and the second side information, and therefore may include the following cases:

[0014] (1) Reference information includes first context information

[0015] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network. This probability distribution estimation network may be a single neural network or a structure within a self-attention decoding network; this embodiment of the application does not limit this.

[0016] (2) The reference information includes the first context information and the first side information.

[0017] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the first side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0018] (3) The reference information includes first context information and second context information.

[0019] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the second context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0020] (4) The reference information includes the first context information, the first side information, and the second context information.

[0021] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the first edge information, and the second context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0022] (5) Reference information includes first context information and second side information.

[0023] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0024] (6) Reference information includes first context information, first side information and second side information.

[0025] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the first side information, and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0026] (7) Reference information includes first context information, second context information, and second side information.

[0027] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the second context information, and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0028] (8) The reference information includes the first context information, the first side information, the second context information, and the second side information.

[0029] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the first side information, the second context information, and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0030] (9) Reference information includes first-side information

[0031] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0032] (10) Reference information includes first-side information and context information.

[0033] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first side information and context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0034] (11) Reference information includes first-side information and second-side information.

[0035] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first side information and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0036] (12) Reference information includes first-side information, context information, and second-side information.

[0037] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first side information, the context information, and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0038] When the reference information includes first side information, the method further includes: estimating a second estimated probability distribution; and entropy encoding the first side information according to the second estimated probability distribution to obtain a second bitstream.

[0039] Optionally, a second estimated probability distribution can be estimated based on pre-set information. Alternatively, a learnable model obtained through training can be used to estimate the second estimated probability distribution. Then, the probability value of the first side information is calculated based on the second estimated probability distribution, and entropy encoding is performed on the first side information based on this probability value.

[0040] Optionally, the second bitstream can be sent to the decoding side separately, or the second bitstream can be added to the first bitstream and sent to the decoding side. This application embodiment does not limit the sending method of the second bitstream.

[0041] In one possible implementation, the reference information further includes second context information, which is obtained by inputting at least one piece of data from the at least one encoded data that meets preset conditions into a masked convolutional network. The masked convolutional network includes masked convolutional layers or regular convolutional layers.

[0042] For example, at least one data point that meets the preset conditions can be at least one data point that is the nearest neighbor of the data to be encoded from at least one encoded data point. For one-dimensional data, the nearest neighbor can be the first m encoded data points of the data to be encoded, where m > 0. For two-dimensional data, the nearest neighbor can be adjacent data of the data to be encoded, or encoded data in the outer n rings of data of the data to be encoded, etc., where n > 0. In this embodiment of the application, the nearest neighbor is not limited.

[0043] The use of encoded data in the process of obtaining the second context information can improve the accuracy of the first estimated probability distribution, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0044] Masked convolutional networks possess local receptive fields, which include masked convolutional layers or regular convolutional layers. The input of a masked convolutional network is at least one data point that is the nearest neighbor of the data to be encoded from at least one encoded data point, and the output is the activation features of the convolution output, i.e., second contextual information.

[0045] In one possible implementation, the reference information further includes second side information, which is obtained by inputting at least one data that meets preset conditions from the plurality of data into a hyper-encoder network; the method further includes: estimating a third estimated probability distribution; and entropy encoding the second side information according to the third estimated probability distribution to obtain a third bitstream.

[0046] For example, at least one data point that meets the preset conditions can be at least one data point that is the nearest neighbor to the data to be encoded among multiple data points. For one-dimensional data, the nearest neighbor can be the first m1 bits and / or the last m2 bits of the data to be encoded, where m1, m2 > 0. For two-dimensional data, the nearest neighbor can be adjacent data of the data to be encoded, or data within n concentric rings of the data to be encoded, etc., where n > 0.

[0047] Masked convolutional networks have local receptive fields and include regular convolutional layers. The input of a masked convolutional network is at least one data point that is the nearest neighbor of the data to be encoded from multiple data points, and the output is the activation features of the convolution output, i.e., the second-side information.

[0048] Optionally, the third bitstream can be sent to the decoding side separately, or the third bitstream can be added to the first bitstream and sent to the decoding side. This application embodiment does not limit the method of sending the third bitstream.

[0049] In one possible implementation, the method further includes: acquiring the first-order encoded data from the plurality of data; estimating a fourth estimated probability distribution based on pre-set information; and performing entropy encoding on the first-order encoded data according to the fourth estimated probability distribution to obtain a fourth bitstream.

[0050] For the first-order encoded data, a fourth estimated probability distribution can be estimated based on pre-set information. Alternatively, a learnable model obtained through training can be used to estimate the fourth estimated probability distribution. This application does not limit the method of obtaining the fourth estimated probability distribution in its embodiments.

[0051] In this application, the self-attention encoding network can be, for example, a transform encoder, and the self-attention decoding network can be, for example, a transform decoder. The first bitstream can refer to the first encoded bitstream, the second bitstream can refer to the second encoded bitstream, the third bitstream can refer to the third encoded bitstream, and the fourth bitstream can refer to the fourth encoded bitstream.

[0052] Secondly, this application provides an entropy decoding method, the method comprising: acquiring a first bitstream; acquiring reference information, the reference information including at least one of first context information and decoded first side information, wherein the first context information is obtained by inputting at least one decoded data from an attention decoding network, and the decoded first side information is obtained by entropy decoding a second bitstream; estimating a first estimated probability distribution based on the reference information; and entropy decoding the first bitstream based on the first estimated probability distribution to obtain decoded data, wherein the decoded data is non-first-order decoded data among multiple data contained in the current data stream.

[0053] In this entropy decoding method, the received first bitstream is obtained by entropy encoding the data to be encoded according to a first estimated probability distribution. This first estimated probability distribution is based on reference information, which may include at least one of first context information and decoded first-side information. The self-attention decoding network can weight all the input encoded data using corresponding weights to obtain the first context information. This improves the utilization rate of the encoded data during the acquisition of the first context information. When using the first context information to estimate the first estimated probability distribution, the accuracy of the obtained first estimated probability distribution can be improved, the bit rate in the entropy encoding process can be reduced, thereby reducing the bandwidth occupancy rate when the first bitstream is transmitted to the decoding side and improving the transmission efficiency of the first bitstream to the decoding side.

[0054] In one possible implementation, obtaining the reference information further includes: obtaining a second bitstream; estimating a second estimated probability distribution; and performing entropy decoding on the second bitstream according to the second estimated probability distribution to obtain decoded first-side information. Accordingly, the reference information also includes the decoded first-side information.

[0055] It should be noted that the second estimated probability distribution obtained from the decoding side needs to be consistent with the second estimated probability distribution obtained from the encoding side.

[0056] In addition to including at least one of first context information and decoded first-side information, the reference information may also include at least one of second context information and decoded second-side information, and therefore may include the following cases:

[0057] (1) Reference information includes first context information

[0058] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0059] (2) The reference information includes the first context information and the decoded first side information.

[0060] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the decoded first side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0061] (3) The reference information includes first context information and second context information.

[0062] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the second context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0063] (4) The reference information includes the first context information, the decoded first side information, and the second context information.

[0064] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the decoded first side information, and the second context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0065] (5) The reference information includes the first context information and the decoded second side information.

[0066] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information and the decoded second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0067] (6) The reference information includes the first context information, the decoded first-side information, and the decoded second-side information.

[0068] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the decoded first side information, and the decoded second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0069] (7) The reference information includes the first context information, the second context information, and the decoded second-side information.

[0070] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the second context information, and the decoded second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0071] (8) The reference information includes the first context information, the decoded first side information, the second context information, and the decoded second side information.

[0072] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the first context information, the decoded first side information, the second context information, and the decoded second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0073] (9) The reference information includes the decoded first-side information.

[0074] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the decoded first side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0075] (10) Reference information includes decoded first-side information and context information.

[0076] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the decoded first side information and context information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0077] (11) The reference information includes the decoded first-side information and the decoded second-side information.

[0078] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the decoded first-side information and the decoded second-side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0079] (12) The reference information includes the decoded first-side information, context information, and decoded second-side information.

[0080] Accordingly, estimating the first estimated probability distribution based on the reference information may include: inputting the decoded first-side information, context information, and decoded second-side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0081] In one possible implementation, the reference information further includes second context information, which is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network.

[0082] In one possible implementation, obtaining the reference information further includes: obtaining a third bitstream; estimating a third estimated probability distribution; performing entropy decoding on the third bitstream based on the third estimated probability distribution to obtain decoded second-side information, and correspondingly, the reference information also includes the decoded second-side information.

[0083] It should be noted that the third estimated probability distribution obtained from the decoding side needs to be consistent with the third estimated probability distribution obtained from the encoding side.

[0084] In one possible implementation, the method further includes: acquiring a fourth bitstream; estimating a fourth estimated probability distribution based on pre-set information; and performing entropy decoding on the fourth bitstream based on the fourth estimated probability distribution to obtain decoded first-order data, wherein the decoded first-order data is the first decoded data among the plurality of data.

[0085] It should be noted that the fourth estimated probability distribution obtained from the decoding side needs to be consistent with the fourth estimated probability distribution obtained from the encoding side.

[0086] Thirdly, this application provides an entropy coding apparatus, the apparatus comprising: an acquisition module for acquiring data to be encoded, the data to be encoded being non-first-order encoded data among a plurality of data contained in a current data stream; acquiring reference information, the reference information including at least one of first context information and first side information, the first context information being obtained by inputting at least one encoded data into an attention decoding network, and the first side information being obtained by inputting the plurality of data into an attention coding network; an estimation module for estimating a first estimated probability distribution based on the reference information; and an encoding module for entropy coding the data to be encoded based on the first estimated probability distribution to obtain a first bitstream.

[0087] In one possible implementation, the reference information specifically includes the first context information and the first edge information; the estimation module is specifically used to input the first context information and the first edge information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0088] In one possible implementation, the reference information specifically includes the first context information and the second context information, wherein the second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network; the estimation module is specifically used to input the first context information and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0089] In one possible implementation, the reference information specifically includes the first context information, the first edge information, and the second context information. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the first edge information, and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0090] In one possible implementation, the reference information specifically includes the first context information and the second side information, wherein the second side information is obtained by inputting at least one data that meets a preset condition from the plurality of data into a supercoding network; the estimation module is specifically used to input the first context information and the second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0091] In one possible implementation, the reference information specifically includes the first context information, the first edge information, and the second edge information, wherein the second edge information is obtained by inputting at least one data that meets a preset condition from the plurality of data into a supercoding network; the estimation module is specifically used to input the first context information, the first edge information, and the second edge information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0092] In one possible implementation, the reference information specifically includes first context information, second context information, and second side information. The second side information is obtained by inputting at least one data that meets a preset condition from the plurality of data into a supercoding network. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the second context information, and the second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0093] In one possible implementation, the reference information specifically includes the first context information, the first edge information, the second context information, and the second edge information. The second edge information is obtained by inputting at least one data that meets a preset condition from the plurality of data into a supercoding network. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the first edge information, the second context information, and the second edge information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0094] In one possible implementation, the reference information specifically includes the first edge information and the second context information, wherein the second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network; the estimation module is specifically used to input the first edge information and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0095] In one possible implementation, the reference information specifically includes the first side information and the second side information, wherein the second side information is obtained by inputting at least one data that meets a preset condition from the plurality of data into the supercoding network; the estimation module is specifically used to input the first side information and the second side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0096] In one possible implementation, the reference information specifically includes the first side information, the second context information, and the second side information. The second side information is obtained by inputting at least one data that meets a preset condition from the plurality of data into a supercoding network. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one encoded data into a masking convolutional network. The estimation module is specifically used to input the first side information, the second context information, and the second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0097] In one possible implementation, the estimation module is further configured to estimate a second estimated probability distribution; the encoding module is further configured to entropy encode the first side information according to the second estimated probability distribution to obtain a second bitstream.

[0098] In one possible implementation, the estimation module is further configured to estimate the third estimated probability distribution; the encoding module is further configured to entropy encode the second side information according to the third estimated probability distribution to obtain a third bitstream.

[0099] In one possible implementation, the acquisition module is further configured to acquire the first-order encoded data among the plurality of data; the estimation module is further configured to estimate a fourth estimated probability distribution based on pre-set information; and the encoding module is further configured to perform entropy encoding on the first-order encoded data according to the fourth estimated probability distribution to obtain a fourth bitstream.

[0100] Fourthly, this application provides an entropy decoding apparatus, the apparatus comprising: an acquisition module for acquiring a first bitstream; acquiring reference information, the reference information including at least one of first context information and decoded first side information, wherein the first context information is obtained by inputting at least one decoded data from an attention decoding network, and the decoded first side information is obtained by entropy decoding a second bitstream; an estimation module for estimating a first estimated probability distribution based on the reference information; and a decoding module for entropy decoding the first bitstream based on the first estimated probability distribution to obtain decoded data, wherein the decoded data is non-first-order decoded data among multiple data contained in the current data stream.

[0101] In one possible implementation, the reference information specifically includes the first context information and the decoded first side information; the estimation module is specifically used to input the first context information and the decoded first side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0102] In one possible implementation, the reference information specifically includes the first context information and the second context information, wherein the second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network; the estimation module is specifically used to input the first context information and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0103] In one possible implementation, the reference information specifically includes the first context information, the decoded first side information, and the second context information. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the decoded first side information, and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0104] In one possible implementation, the reference information specifically includes the first context information and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream; the estimation module is specifically used to input the first context information and the decoded second-side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0105] In one possible implementation, the reference information specifically includes the first context information, the decoded first-side information, and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream; the estimation module is specifically used to input the first context information, the decoded first-side information, and the decoded second-side information into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0106] In one possible implementation, the reference information specifically includes first context information, second context information, and decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the second context information, and the decoded second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0107] In one possible implementation, the reference information specifically includes the first context information, the decoded first side information, the second context information, and the decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The estimation module is specifically used to input the first context information, the decoded first side information, the second context information, and the decoded second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0108] In one possible implementation, the reference information specifically includes the decoded first side information and the second context information, wherein the second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network; the estimation module is specifically used to input the decoded first side information and the second context information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0109] In one possible implementation, the reference information specifically includes the decoded first-side information and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream; the estimation module is specifically used to input the decoded first-side information and the decoded second-side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0110] In one possible implementation, the reference information specifically includes the decoded first side information, the second context information, and the decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The estimation module is specifically used to input the decoded first side information, the second context information, and the decoded second side information into a probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

[0111] In one possible implementation, when the reference information includes the decoded first side information, the acquisition module is further configured to acquire the second bitstream; the estimation module is further configured to estimate a second estimated probability distribution; and the decryption module is further configured to perform entropy decoding on the second bitstream according to the second estimated probability distribution to obtain the decoded first side information.

[0112] In one possible implementation, when the reference information includes the decoded second-side information, the acquisition module is further configured to acquire the third bitstream; the estimation module is further configured to estimate a third estimated probability distribution; and the decoding module is further configured to perform entropy decoding on the third bitstream based on the third estimated probability distribution to obtain the decoded second-side information.

[0113] In one possible implementation, the acquisition module is further configured to acquire a fourth bitstream; the estimation module is further configured to estimate a fourth estimated probability distribution based on pre-set information; and the decoding module is further configured to perform entropy decoding on the fourth bitstream based on the fourth estimated probability distribution to obtain decoded first-order data, wherein the decoded first-order data is the first decoded data among the plurality of data.

[0114] Fifthly, this application provides an entropy coding device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any one of the first aspects above.

[0115] In a sixth aspect, this application provides an entropy decoding device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any one of the second aspects above.

[0116] In a seventh aspect, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects above.

[0117] Eighthly, this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects. Attached Figure Description

[0118] Figure 1 An exemplary block diagram of the decoding system 10 provided in this application embodiment;

[0119] Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of this application;

[0120] Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of this application;

[0121] Figure 4An exemplary schematic diagram of a candidate image block provided in an embodiment of this application;

[0122] Figure 5 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0123] Figure 6 This application provides an illustration of another application scenario;

[0124] Figure 7 This is a schematic diagram of the encoder structure in an end-to-end codec architecture provided in an embodiment of this application;

[0125] Figure 8 This is a schematic diagram of the decoder structure in an end-to-end codec architecture provided in an embodiment of this application;

[0126] Figure 9 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application;

[0127] Figure 10 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0128] Figure 11 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application;

[0129] Figure 12 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0130] Figure 13 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application;

[0131] Figure 14 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0132] Figure 15 A schematic diagram of a self-attention structure provided in an embodiment of this application;

[0133] Figure 16 This is a schematic diagram of the structure of a self-attention coding network provided in an embodiment of this application;

[0134] Figure 17 This is a schematic diagram of the structure of a self-attention decoding network provided in an embodiment of this application;

[0135] Figure 18 A flowchart of the entropy encoding method process 100 provided in the embodiments of this application;

[0136] Figure 19 A flowchart of process 200 of the entropy decoding method provided in the embodiments of this application;

[0137] Figure 20 A flowchart of process 300 of the entropy encoding / decoding method provided in the embodiments of this application;

[0138] Figure 21 A schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application;

[0139] Figure 22 A schematic diagram illustrating the entropy coding performance provided in an embodiment of this application;

[0140] Figure 23 A flowchart of process 400 of the entropy encoding / decoding method provided in the embodiments of this application;

[0141] Figure 24 A schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application;

[0142] Figure 25 A flowchart of process 500 of the entropy encoding / decoding method provided in the embodiments of this application;

[0143] Figure 26 A flowchart of process 600 of the entropy encoding / decoding method provided in the embodiments of this application;

[0144] Figure 27 This is a schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application. Detailed Implementation

[0145] This application provides an AI-based data compression / decompression technology, particularly a neural network-based data compression / decompression technology, specifically an entropy encoding / decoding technology to improve traditional hybrid data encoding / decoding systems.

[0146] Data encoding and decoding comprises two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw data to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Data decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original data. The “encoding and decoding” of data in this embodiment should be understood as either “encoding” or “decoding” the data. The encoding and decoding parts are also collectively referred to as encoding and decoding (CODEC).

[0147] In lossless data encoding, the original data can be reconstructed, meaning the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In lossy data encoding, further compression is performed through quantization to reduce the amount of data required to represent the original data, but the decoder cannot completely reconstruct the original data, meaning the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0148] This application's embodiments can be applied to video data, image data, audio data, integer data, and other data requiring compression / decompression. The following description uses video data encoding (hereinafter referred to as video encoding) as an example to illustrate this application's embodiments. Other types of data (such as image data, audio data, integer data, and other data requiring compression / decompression) can be referred to the following description, and will not be repeated in this application's embodiments. It should be noted that, compared to video encoding, the encoding of audio data and integer data does not require dividing the data into blocks; instead, the data can be directly encoded.

[0149] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously.

[0150] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.

[0151] In the following embodiment of the decoding system 10, the encoder 20 and decoder 30 are based on Figures 1 to 3 Describe it.

[0152] Figure 1This is an exemplary block diagram of a decoding system 10 provided for embodiments of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in this application.

[0153] like Figure 1 As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0154] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0155] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.

[0156] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0157] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0158] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).

[0159] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0160] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0161] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0162] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0163] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0164] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0165] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1 The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0166] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0167] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0168] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0169] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to process the input image or image region or image block to obtain reference information of the image block to be encoded, or to process the input reference information to estimate the estimated probability distribution of the image block to be encoded, so as to entropy encode the image block to be encoded according to the estimated probability distribution. For a detailed description of the training engine 25, please refer to the following method and implementation examples.

[0170] although Figure 1 The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0171] According to the description, Figure 1 The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0172] Please refer to Figure 2 and Figure 3 , Figure 2 This is an exemplary block diagram of a video encoder provided in an embodiment of this application. Figure 3 This is an exemplary block diagram of a video decoder provided in an embodiment of this application. The encoder 20 can be implemented using processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 comprises various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 can be used to perform various operations discussed below. If some of the technology is implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of this application. One of the video encoder 20 and video decoder 30 may be integrated into a single device as part of a combined codec (encoder / decoder, CODEC).

[0173] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablets, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use any type of operating system. Source device 12 and destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0174] Source device 12 and destination device 14 can install virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, and can run VR, AR, or MR applications based on user actions (e.g., clicks, touches, swipes, shaking, voice control, etc.). Source device 12 and destination device 14 can acquire images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on a display device based on the acquired images / videos. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in a virtual environment).

[0175] It should be noted that, in this embodiment of the application, the virtual scene application in the source device 12 and the destination device 14 can be an application built into the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider that is installed by the user. No specific limitation is made in this regard.

[0176] Furthermore, source device 12 and destination device 14 can be equipped with real-time video transmission applications, such as live streaming applications. Source device 12 and destination device 14 can capture images / videos via cameras and then display the captured images / videos on a display device.

[0177] In some cases, Figure 1 The video decoding system 10 shown is merely exemplary, and the technology provided in this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve and decode data from memory.

[0178] A video decoding system may include an imaging device, a video encoder, a video decoder (and / or a video encoder / decoder implemented by processing circuitry), an antenna, one or more processors, one or more memory storage devices, and / or a display device.

[0179] Imaging devices, antennas, processing circuitry, video encoders, video decoders, processors, memory storage, and / or display devices are capable of communicating with each other. In different instances, a video decoding system may contain only a video encoder or only a video decoder.

[0180] In some instances, the antenna can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, a display device can be used to present the video data. The processing circuitry can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. The video decoding system may also include an optional processor, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, the memory can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, the memory can be implemented using high-speed cache memory. In other instances, the processing circuitry can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0181] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry or memory storage) an image buffer and (e.g., implemented via processing circuitry) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry to implement reference... Figure 2 And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.

[0182] In some instances, the video decoder 30 can be implemented in a similar manner via processing circuitry to implement the reference. Figure 3 The video decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the video decoder 30, implemented as logic circuitry, may include an image buffer (implemented via processing circuitry or memory storage) and a graphics processing unit (e.g., implemented via processing circuitry). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry to implement the reference... Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.

[0183] In some instances, the antenna can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining code segmentation). The video decoding system may also include a video decoder 30 coupled to the antenna and used to decode the encoded bitstream. A display device is used to present the video frames.

[0184] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0185] For ease of description, embodiments of this application are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0186] Encoders and Encoding Methods

[0187] like Figure 2 As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0188] See Figure 2 Inter-frame prediction units are trained target models (also known as neural networks) that process input images, image regions, or image patches to generate predicted values ​​for those patches. For example, a neural network for inter-frame prediction receives an input image, image region, or image patch and generates predicted values ​​for that image, image region, or image patch.

[0189] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the backward signal path of encoder 20. The backward signal path of encoder 20 corresponds to the signal path of decoder (see [link to decoder]). Figure 3 The decoder 30 in the video encoder 20 consists of an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254.

[0190] Image and image segmentation (images and patches)

[0191] Encoder 20 can be used to receive images (or image data) 17 via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received images or image data can also be pre-processed images (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video encoding when distinguishing the current image from other images, such as those in the same video sequence, i.e., the video sequence that also includes the current image, previously encoded images, and / or decoded images).

[0192] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, including the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color transformation or conversion. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0193] In one embodiment, the video encoder 20 may include an image segmentation unit ( Figure 2 (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. Segmentation units can be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0194] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

[0195] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a single pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0196] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0197] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode images using slices (also called video slices), where images can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0198] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode an image using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0199] Residual calculation

[0200] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 in the following manner (the prediction block 265 is described in detail later): for example, the residual block 205 in the pixel domain is obtained by subtracting the pixel value of the prediction block 265 from the pixel value of the image block 203 pixel by pixel.

[0201] Transformation

[0202] The transformation processing unit 206 performs discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain the transformation coefficients 207 in the transform domain. The transformation coefficients 207 can also be called transformation residual coefficients, representing the residual block 205 in the transform domain.

[0203] Transform processing unit 206 can be used to apply an integer approximation of DCT / DST, such as the transform specified for H.265 / HEVC. This integer approximation is typically scaled by a certain factor compared to orthogonal DCT transforms. To maintain the norm of the residual block after both forward and inverse transforms, other scaling factors are used as part of the transform process. These scaling factors are typically selected based on certain constraints, such as the power of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor can be specified on the encoder 20 side via inverse transform processing unit 212 (and on the decoder 30 side via, for example, inverse transform processing unit 312) for the inverse transform, and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0204] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to output transform parameters such as the type of one or more transforms, for example, directly outputting them or outputting them after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0205] Quantification

[0206] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.

[0207] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated from the quantization parameter using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0208] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use the quantization parameter for decoding.

[0209] Inverse Quantization

[0210] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0211] Inverse Transformation

[0212] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0213] reconstruction

[0214] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 213 and the pixel values ​​of the prediction block 265.

[0215] Filtering

[0216] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 in... Figure 2 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.

[0217] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly outputting or outputting after entropy encoding by the entropy encoding unit 270, for example, enabling the decoder 30 to receive and decode using the same or different loop filter parameters.

[0218] Decoding image buffer

[0219] The decoded picture buffer (DPB) 230 can be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previous filter blocks of the same current image or different images, such as previously reconstructed images, such as previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed i.e., decoded images (and corresponding reference blocks and pixels) and / or partially reconstructed current images (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or generally store unfiltered reconstruction pixels, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction pixels that have not undergone any other processing.

[0220] Pattern selection (segmentation and prediction)

[0221] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for selecting data from the decoded image buffer 230 or other buffers (e.g., column buffers). Figure 2 (Not shown) receives or obtains raw image data such as raw block 203 (current block 203 of the current image 17) and reconstructed image data, for example, filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data for prediction such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0222] The mode selection unit 260 can be used to determine or select a segmentation for the current block (including no segmentation) and the prediction mode (e.g., intra-frame or inter-frame prediction mode) to generate the corresponding prediction block 265 for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0223] In one embodiment, the mode selection unit 260 can be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), which provides the best match or minimum residual (minimum residual refers to better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead refers to better compression in transmission or storage), or considers or balances both. The mode selection unit 260 can be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization. The terms "best," "lowest," and "optimal" in this document do not necessarily refer to "best," "lowest," or "optimal" overall, but can also refer to situations that meet termination or selection criteria. For example, values ​​exceeding or falling below a threshold or other limitations may lead to a "suboptimal choice," but reduce complexity and processing time.

[0224] In other words, segmentation unit 262 can be used to segment images in a video sequence into a sequence of coding tree units (CTUs), CTUs 203 can be further segmented into smaller block portions or sub-blocks (forming blocks again), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT) or any combination thereof, and is used to perform prediction, for example, on each of the block portions or sub-blocks, wherein mode selection includes selecting the tree structure of the segmented block 203 and selecting the prediction mode applied to each of the block portions or sub-blocks.

[0225] The segmentation (e.g., performed by segmentation unit 262) and prediction processing (e.g., performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by video encoder 20 will be described in detail below.

[0226] segmentation

[0227] Segmentation unit 262 can divide (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks of square or rectangular shape. For an image with a three-pixel array, a CTU consists of N×N luma pixel blocks and two corresponding chroma pixel blocks. The maximum allowed size of the luma blocks in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. Image CTUs can be grouped / set into slices / coded block groups, coded blocks, or bricks. A coded block covers a rectangular area of ​​an image, and a coded block can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a coded block. A coded block that is not divided into multiple bricks can be called a brick. However, a brick is a true subset of a coded block and therefore is not called a coded block. VVC supports two coded block group modes: raster scan slice / coded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the raster scan order of the slice's bricks. These smaller blocks (also called sub-blocks) can be further subdivided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root tree level 0 (level 0, depth 0), etc., can be recursively segmented into two or more blocks at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into two or more blocks at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends (because the termination criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad-tree (QT).

[0228] For example, a coding unit (CTU) can be or include a CTB of luma pixels, two corresponding CTBs of chroma pixels in an image with a three-pixel array, a CTB of pixels in a monochrome image, or a CTB of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CTB) can be an N×N pixel block, where N can be set to a value such that the components are divided into CTBs; this is called segmentation. Similarly, a coding unit (CU) can be or include a coding block of luma pixels, two corresponding coding blocks of chroma pixels in an image with a three-pixel array, a coding block of pixels in a monochrome image, or a coding block of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CB) can be an M×N pixel block, where M and N can be set to a value such that the CTB is divided into coding blocks; this is called segmentation.

[0229] For example, in one embodiment, HEVC can divide a Coding Tree Unit (CTU) into multiple Coding Units (CUs) using a quadtree structure represented as a coding tree. At the leaf CU level, a decision is made as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode image regions. Each leaf CU can be further divided into one, two, or four PUs based on the PU partitioning type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying the prediction process according to the PU partitioning type, the leaf CU can be divided into Transform Units (TUs) according to other quadtree structures similar to those used for the coding tree of the CU.

[0230] For example, in one embodiment, according to the latest video coding standard currently under development (called Universal Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition the segmented structure for dividing the coding tree unit. Within the coding tree structure of a coding tree unit, the CU can be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by multi-type tree structures. There are four partitioning types for the multi-type tree structure: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning (SPLIT_BT_HOR), and vertical ternary tree partitioning (SPLIT_BT_HOR). LIT_TT_VER) and horizontal ternary tree partitioning (SPLIT_TT_HOR). Multi-type leaf nodes are called coding units (CUs). Unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other partitioning. In most cases, this means that the CU, PU, ​​and TU have the same block size in the quadtree-nested multi-type tree coding block structure. This anomaly occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC specifies a unique signaling mechanism for partitioning information in a quadtree-nested multi-type tree coding structure. In this signaling mechanism, the coding tree unit (CTU) The root of the quadtree is first divided by the quadtree structure. Then, each quadtree leaf node (if large enough) is further divided into a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) indicates whether a node is further divided. When a node is further divided, a second flag (mtt_split_cu_vertical_flag) indicates the direction of the division, and a third flag (mtt_split_cu_binary_flag) indicates whether the division is a binary or ternary tree. This is determined by mtt_split_cu_ve The decoder can derive the multi-type tree partitioning mode (MttSplitMode) of the CU based on predefined rules or tables using the values ​​of `rtical_flag` and `mtt_split_cu_binary_flag`. It's important to note that for certain designs, such as a 64×64 luma block and a 32×32 chroma pipeline design in a VVC hardware decoder, TT partitioning is not allowed when the width or height of the luma encoding block exceeds 64. Similarly, TT partitioning is not allowed when the width or height of the chroma encoding block exceeds 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages.In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it's necessary to keep the VPDU small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the partitioning of ternary trees (TT) and binary trees (BT) can increase the VPDU size.

[0231] Additionally, it should be noted that when a portion of a tree node block extends beyond the bottom or right edge of the image, the tree node block is forcibly divided until all pixels of each encoded CU are within the image boundary.

[0232] For example, the intra-sub-partitions (ISP) tool can divide the luminance intra-prediction block vertically or horizontally into two or four sub-parts based on the block size.

[0233] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0234] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0235] Intra-frame prediction

[0236] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several conventional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. As another example, to avoid division operations in DC prediction, only the longer side is used to calculate the average value of non-square blocks. Furthermore, the intra-prediction results of planar mode can be modified using the position-dependent intra-prediction combination (PDPC) method.

[0237] Intra-prediction unit 254 is used to generate intra-prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to the intra-prediction mode in the intra-prediction mode set.

[0238] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information indicating the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266 to be included in encoded image data 21, so that video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0239] HEVC's intra-frame prediction modes include DC prediction mode, planar prediction mode, and 33 angle prediction modes, totaling 35 candidate prediction modes. The current block can use pixels from the reconstructed image blocks to its left and above as references for intra-frame prediction. The image blocks in the surrounding region used for intra-frame prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, DC prediction mode is suitable for regions with flat textures in the current block, where all pixels use the average value of the reference pixels in the reference block as the prediction; planar prediction mode is suitable for image blocks with smoothly varying textures, where the current block uses bilinear interpolation of the reference pixels in the reference block to predict all pixels in the current block; angle prediction mode utilizes the high correlation between the texture of the current block and the textures of adjacent reconstructed image blocks, copying the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction for all pixels in the current block.

[0240] The HEVC encoder selects an optimal intra-prediction mode from 35 candidate prediction modes for the current block and writes this optimal intra-prediction mode into the video stream. To improve the coding efficiency of intra-prediction, the encoder / decoder derives three most probable modes from the optimal intra-prediction modes of the reconstructed image blocks in the surrounding regions that use intra-prediction. If the optimal intra-prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-prediction mode is one of these three most probable modes; if the selected optimal intra-prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-prediction mode is one of the other 32 modes (other modes from the 35 candidate prediction modes besides the aforementioned three most probable modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0241] The HEVC encoder derives the three most likely modes by: selecting the best intra-prediction modes of the left and top adjacent image blocks of the current block and adding them to the set; if these two best intra-prediction modes are the same, only one needs to be kept in the set. If these two best intra-prediction modes are the same and both are angle prediction modes, then two angle prediction modes adjacent to that angle direction are selected and added to the set; otherwise, the planar prediction mode, DC mode, and vertical prediction mode are selected and added to the set in sequence, until the number of modes in the set reaches three.

[0242] After performing entropy decoding on the bitstream, the HEVC decoder obtains the mode information of the current block. This mode information includes an indicator that indicates whether the optimal intra-prediction mode of the current block is among the three most likely modes, as well as the index of the optimal intra-prediction mode of the current block among the three most likely modes or the index of the optimal intra-prediction mode of the current block among the other 32 modes.

[0243] Inter-frame prediction

[0244] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least a portion of the previously decoded image stored in the DBP 230) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a portion of the reference image, such as a search window region near the current block, to search for the best matching reference block, and / or, for example, whether to perform pixel interpolation of half-pixel, quarter-pixel, and / or 1 / 16th interpolation.

[0245] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0246] For example, extended merge prediction, this mode's merge candidate list consists of five candidate types in sequence: spatial MVP from spatially adjacent CUs, temporal MVP from co-located CUs, history-based MVP from a FIFO table, pairwise average MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the merge mode's MV. Mergemode with MVD (MMVD) comes from merge modes with motion vector differences. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses MMVD mode. A CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding CU MVD with different precisions. The MVD of the current CU is adaptively selected based on the current CU's prediction mode. When the CU is encoding in merge mode, combined inter / intra prediction (CIIP) mode can be applied to the current CU. CIIP prediction is obtained by weighted averaging of inter-frame and intra-frame prediction signals. For affine motion compensation prediction, the affine motion field of the block is described by motion information from motion vectors of 2 control points (4 parameters) or 3 control points (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vectors of sub-CUs within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computation, particularly in terms of the number of multiplications and the size of the multipliers. In the triangular partitioning mode, the CU is uniformly divided into two triangular parts using both diagonal and anti-diagonal partitioning. Furthermore, the bidirectional prediction mode extends the simple averaging to support weighted averaging of the two prediction signals.

[0247] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in... Figure 2(Not shown in the image). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as one or more other / different previously decoded image blocks 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form the image sequence that forms the video sequence.

[0248] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the offset (spatial offset) between the position (x, y coordinates) of the reference image (or reference image index) and / or the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0249] The motion compensation unit is used to acquire, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction blocks 246. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include performing interpolation with sub-pixel precision. Interpolation filtering can generate pixels of other pixels from pixels of known pixels, thereby potentially increasing the number of candidate prediction blocks available for encoding image blocks. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0250] The motion compensation unit can also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slices. Alternatively, or as an alternative to slices and corresponding syntax elements, coded block groups and / or coded blocks and their corresponding syntax elements can be generated or used.

[0251] In obtaining the candidate motion vector list in the advanced motion vector prediction (AMVP) mode, the motion vectors (MVs) that can be added to the candidate motion vector list as candidates include the MVs of spatially and temporally adjacent image blocks of the current block. The MVs of spatially adjacent image blocks can further include the MVs of candidate image blocks to the left of the current block and the MVs of candidate image blocks above the current block. For example, please refer to... Figure 4 , Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application, such as... Figure 4As shown, the set of candidate image blocks to the left includes {A0, A1}, the set of candidate image blocks above includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as candidates. However, according to existing coding standards, the maximum length of the candidate motion vector list for AMVP is 2. Therefore, it is necessary to determine the MV (Motion Vector Model) to be added to the candidate motion vector list from the three sets according to a specified order. This order can be: first, consider the set of candidate image blocks to the left of the current block {A0, A1} (consider A0 first, then A1 if A0 is unavailable); second, consider the set of candidate image blocks above the current block {B0, B1, B2} (consider B0 first, then B1 if B0 is unavailable, then B2 if B1 is unavailable); finally, consider the set of temporally adjacent candidate image blocks {C, T} (consider T first, then C if T is unavailable).

[0252] After obtaining the candidate motion vector list, the optimal motion vector prediction (MV) is determined from the list using the rate distortion cost (RDcost). The candidate motion vector with the lowest RDcost is then used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula:

[0253] J = SAD + λR

[0254] Where J represents the RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using candidate motion vectors and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0255] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Further, a motion search can be performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0256] During the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list includes the motion information of spatially adjacent or temporally adjacent image blocks of the current block. The spatially adjacent and temporally adjacent image blocks can be referenced... Figure 4 The candidate motion information list corresponds to spatially adjacent blocks (A0, A1, B0, B1, and B2) in the spatial domain. If a spatially adjacent block is unavailable or is intra-frame prediction, its motion information is not added to the candidate motion information list. The temporal candidate motion information for the current block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available; if not, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list using the RD cost as the motion information for the current block. The encoder transmits the index value (denoted as mergeindex) of the optimal motion information in the candidate motion information list to the decoder.

[0257] Entropy coding

[0258] See Figure 2 The entropy coding unit 270 includes a trained self-attention decoding network 2071 and a self-attention coding network 2072. The self-attention decoding network 2071 is used to process the input image or image region or image block to obtain first context information; the self-attention coding network 2072 is used to process the input image or image region or image block to obtain first side information.

[0259] Entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output as an encoded bitstream 21 through output terminal 272, so that video decoder 30 and the like can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0260] Other architectural variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0261] Decoder and Decoding Method

[0262] like Figure 3 As shown, the video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video segments) and associated syntax elements.

[0263] exist Figure 3In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the referenced unit. Figure 2 The video encoder 100 describes the encoding process as the opposite of the decoding process.

[0264] See Figure 3 The entropy decoding unit 304 includes a trained self-attention decoding network 3041, which processes the input image or image region or image patch to obtain first contextual information.

[0265] As described in encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 210, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 212, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0266] Entropy Decoding

[0267] Entropy decoding unit 304 is used to parse bitstream 21 (or generally encoded image data 21) and perform entropy decoding on encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image) Examples of parameters include inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply the decoding algorithm or scheme corresponding to the encoding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice and / or video block level. Furthermore, or as an alternative to slices and corresponding syntax elements, it can receive or use coded block groups and / or coded blocks and corresponding syntax elements.

[0268] Inverse Quantization

[0269] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.

[0270] Inverse Transformation

[0271] The inverse transform processing unit 312 can be used to receive the inverse quantization coefficients 311, also known as transform coefficients 311, and apply a transform to the inverse quantization coefficients 311 to obtain the reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 can also be called transform block 313. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantization coefficients 311.

[0272] reconstruction

[0273] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 313 and the pixel values ​​of the prediction block 365.

[0274] Filtering

[0275] Loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby facilitating pixel transformation or improving video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 320 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 in... Figure 3 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0276] Decoding image buffer

[0277] The decoded video block 321 in one image is then stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. The reference image is used for subsequent motion compensation for other images and / or output displays respectively.

[0278] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0279] predict

[0280] Inter-frame prediction unit 344 is functionally identical to inter-frame prediction unit 244 (especially motion compensation unit), and intra-frame prediction unit 354 is functionally identical to inter-frame prediction unit 254. It determines segmentation or partitioning and performs prediction based on segmentation and / or prediction parameters or corresponding information received from coded image data 21 (e.g., parsed and / or decoded by entropy decoding unit 304). Pattern application unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel (filtered or unfiltered), resulting in prediction block 365.

[0281] When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 generates a prediction block 365 for the current video slice based on the indicated intra-prediction mode and data from the previous decoded block of the current image. When a video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 generates a prediction block 365 for the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame lists 0 and 1 using the default construction technique based on the reference images stored in the DPB 330. In addition to slices (e.g., video slices) or as a substitute for slices, the same or similar processes can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), such as video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0282] The pattern application unit 360 is used to determine prediction information for video blocks in the current video slice by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video slice being decoded using the prediction information. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more reference image lists for the slice, motion vectors for each inter-frame coded video block in the slice, inter-frame prediction state for each inter-frame coded video block in the slice, and other information to decode video blocks within the current video slice. In addition to slices (e.g., video slices) or as alternatives to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, where video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0283] In one embodiment, Figure 3 The video encoder 30 can also be used to segment and / or decode images using slices (also called video slices), where images can be segmented or decoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coded blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0284] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slice / coded block groups (also known as video coded block groups) and / or coded blocks (also known as video coded blocks), wherein the image can be segmented or decoded using one or more slice / coded block groups (typically non-overlapping), each slice / coded block group may include one or more blocks (e.g., CTUs) or one or more coded blocks, wherein each coded block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0285] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0286] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering, such as clipping or shifting operations.

[0287] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, time motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on the representation bits of the motion vector. If the representation bits of the motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. Two methods for restricting motion vectors based on bitDepth are provided here.

[0288] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0289] This application provides a video decoding device. In one embodiment, the video decoding device may be a decoder, for example... Figure 1 The video decoder 30 in the text can also be an encoder, for example... Figure 1 The video encoder 20 in the middle.

[0290] The video decoding device includes: an input port (or inlet port) and a receiver unit (Rx) for receiving data; a processor, logic unit, or central processing unit (CPU) for processing data; for example, the processor may be a neural network processor; a transmitter unit (Tx) and an output port (or outlet port) for transmitting data; and a memory for storing data. The video decoding device may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port, receiver unit, transmitter unit, and output port for the entry or exit of optical or electrical signals.

[0291] The processor is implemented through hardware and software. The processor can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor communicates with ingress ports, receiver units, transmitter units, egress ports, and memory. The processor includes a decoding module (e.g., a neural network-based decoding module). The decoding module implements the embodiments disclosed above. For example, the decoding module performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module provides a substantial improvement to the functionality of the video decoding device and affects the switching of the video decoding device to different states. Alternatively, the decoding module can be implemented as instructions stored in memory and executed by the processor.

[0292] Memory, including one or more disks, tape drives, and solid-state drives, can be used as overflow data storage devices to store programs when an executable program is selected, and to store instructions and data read during program execution. Memory can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0293] This application provides an apparatus that may include a processor, a memory, and a bus. This apparatus can be used as... Figure 1 The source device 12 and the destination device 14 are either or both of them.

[0294] The processor in the device can be a central processing unit. Alternatively, the processor can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation can be implemented using a single processor, such as the one shown in the figure, using more than one processor is faster and more efficient.

[0295] In one implementation, the memory in the apparatus may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory. The memory may include code and data accessed by the processor via a bus. The memory may also include an operating system and application programs, the application programs including at least one program that allows the processor to perform the methods described herein. For example, the application programs may include applications 1 to N, and also include a video decoding application that performs the methods described herein.

[0296] The device may also include one or more output devices, such as a display. In one example, the display may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display may be coupled to the processor via a bus.

[0297] Although the bus in the device is described herein as a single bus, the bus can include multiple buses. Furthermore, auxiliary storage can be directly coupled to other components of the device or accessed via a network, and can include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, the device can have a wide variety of configurations.

[0298] Since the embodiments of this application involve the application of neural networks, for ease of understanding, some nouns or terms used in the embodiments of this application will be explained below, and these nouns or terms are also part of the content of the invention.

[0299] (1) Neural Network

[0300] A neural network (NN) is a machine learning model. A neural network can be composed of neural units, which are computational units that take xs and an intercept of 1 as input. The output of this computational unit can be:

[0301]

[0302] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0303] (2) Deep Neural Networks

[0304] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0305] (3) Convolutional Neural Network

[0306] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feedforward artificial neural network where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with a trainable filter and an input image or a convolutional feature map.

[0307] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. A convolutional layer can contain multiple convolution operators, also called kernels. In image processing, these operators act as filters, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" factors mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0308] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. This can be a single convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0309] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of desired class numbers of output. Therefore, the neural network can include multiple hidden layers, the parameters of which can be pre-trained based on training data relevant to a specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0310] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0311] (4) Recurrent Neural Network

[0312] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0313] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context, but how can machines do the same? This is where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0314] (5) Loss Function

[0315] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0316] (6) Backpropagation algorithm

[0317] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0318] (7) Generative Adversarial Networks

[0319] Generative adversarial networks (GANs) are a type of deep learning model. This model comprises at least two modules: a generative model and a discriminative model. These two modules learn from each other through a game-like interaction, resulting in better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GANs is as follows: Taking an image-generating GAN as an example, suppose there are two networks, G (Generator) and D (Discriminator). G is a network that generates images by receiving random noise z and using this noise, denoted as G(z). D is a discriminative network used to determine whether an image is "real." Its input parameter is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that the image is 100% real, while a value of 0 indicates that the image is impossible to be real. During the training of this generative adversarial network (GAN), the goal of the generative network G is to generate realistic images to deceive the discriminator network D, while the goal of the discriminator network D is to distinguish the images generated by G from real images as much as possible. Thus, G and D constitute a dynamic "game," which is the "adversarial" aspect of the GAN. Ideally, the game will result in G generating images G(z) that are sufficiently realistic, while D struggles to determine whether the images generated by G are real or not, i.e., D(G(z)) = 0.5. This yields a superior generative model G that can be used to generate images.

[0320] Figure 5 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Figure 5 Let's take images / videos as an example. In this application scenario, the device acquires images / videos, performs entropy encoding on the acquired images / videos to obtain a bitstream, and stores the bitstream. When images / videos are needed for subsequent output, the bitstream is entropy decoded to obtain the final images / videos. This device can integrate the functions of both the aforementioned source and destination devices.

[0321] like Figure 5As shown, the device includes an encoding network, a super-encoding network, an entropy encoding network, a storage module, a loading module, a super-decoding network, an entropy decoding network, and a decoding network. The encoding network extracts features from the input image / video to obtain a feature image / video with low redundancy. The super-encoding network estimates the probability value of each feature element in the feature image / video. The entropy encoding module then entropy-encodes the corresponding feature element based on its estimated probability value, obtaining a bitstream, which is stored by the storage module. The loading module loads this bitstream, and the super-decoding network estimates the probability value of the bitstream corresponding to each feature element. The entropy decoding module entropy-decodes the corresponding bitstream based on its estimated probability value for each feature element, obtaining the feature image / video. The decoding network performs inverse feature extraction on the feature image / video to obtain the final image / video.

[0322] It should be understood that the device compresses images / videos to save storage space. Optionally, the device can store the compressed images / videos in a photo album or a cloud photo album.

[0323] Figure 6 This application provides an illustration of another application scenario. Figure 6 Let's take images / videos as an example. The application scenario involves locally acquiring images / videos, encoding the acquired data using JPEG to obtain compressed images / videos, and then sending the compressed images / videos to the cloud. The cloud performs JPEG decoding on the compressed images / videos to obtain the final images / videos, then performs entropy encoding to obtain the bitstream and stores it. When the local machine needs to acquire images / videos from the cloud later, the cloud performs entropy decoding on the bitstream to obtain the final images / videos, then performs JPEG encoding on the final images / videos to obtain compressed images / videos, and sends the compressed images / videos back to the local machine. The local machine then performs JPEG decoding on the compressed images / videos to obtain the final images / videos. This cloud can integrate the functions of the aforementioned source and destination devices. For the structure of the cloud and the purpose of each module, please refer to [reference needed]. Figure 5 The structure and purpose of each module are not described in detail in the embodiments of this application.

[0324] It should be understood that JPEG encoding of acquired data, whether locally or in the cloud, is done to reduce transmission bandwidth, while cloud compression of images / videos is done to save storage space.

[0325] The method described in this application can be applied to end-to-end (ETO) codec architectures. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the encoder structure in an end-to-end codec architecture provided in an embodiment of this application. Figure 7As shown, the encoder includes an encoding network, a quantization module, a super-encoding network, a super-decoding network, and an entropy coding module. The encoding network extracts features from the input current data stream to obtain feature data. The quantization module quantizes the feature data, and the quantized feature data is then processed by the super-encoding network to obtain a bitstream 2 with side information. Bitstream 2 is then processed by the super-decoding network to obtain the side information. The entropy coding module uses the side information to entropy-encode the input feature data to obtain bitstream 1.

[0326] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the decoder structure in an end-to-end codec architecture provided in an embodiment of this application. Figure 8 As shown, the decoder includes a decoding network, an entropy decoding module, and a super-decoding network. Bitstream 2 is decoded by the super-decoding network to obtain side information. The entropy decoding module is used to perform entropy decoding on bitstream 1 based on the side information to obtain feature data. The decoding network is used to perform inverse feature extraction on the feature data to obtain the current data stream.

[0327] In the entropy encoding / decoding method provided in this application embodiment, the encoder can acquire reference information, then estimate the estimated probability distribution of the data to be encoded based on the reference information, and use the estimated probability distribution of the data to be encoded to perform entropy encoding on the data to be encoded to obtain a bitstream. The decoder can acquire reference information, then estimate the estimated probability distribution of the bitstream based on the reference information, and use the estimated probability distribution of the bitstream to perform entropy decoding on the bitstream. The reference information may include first context information and / or first side information; further, the reference information may also include second context information and second side information.

[0328] When performing entropy encoding on the data to be encoded in the current data stream, the current data stream includes multiple data points. The first context information is obtained by inputting at least one encoded data point from the multiple data points into the attention decoding network. The first side information is obtained by inputting multiple data points from the current data stream into the attention encoding network. The second context information is obtained by inputting at least one encoded data point that meets a preset condition into the masking convolutional network. The second side information is obtained by inputting at least one data point from the multiple data points that meets a preset condition into the supercoding network.

[0329] At least one data point in the at least one encoded data set that meets a preset condition may include at least one data point in the encoded data set that is the nearest neighbor to the data to be encoded. For one-dimensional data, the nearest neighbor to the data to be encoded may be the first m encoded data points of the data to be encoded, where m > 0. For two-dimensional data, the nearest neighbor to the data to be encoded may be adjacent data of the data to be encoded, or encoded data in the outer n rings of data of the data to be encoded, etc., where n > 0. In this embodiment, the nearest neighbor is not limited. Therefore, the first context information is obtained based on at least one encoded data point among multiple data sets, and the second context information is obtained based on at least one data point in the at least one encoded data set that is the nearest neighbor to the data to be encoded. Compared to the second context information, the first context information has a higher utilization rate of the encoded data and is more comprehensive in content.

[0330] At least one data point among multiple data sets that meets preset conditions may include at least one data point that is the nearest neighbor to the data to be encoded. For one-dimensional data, the nearest neighbor may be the first m1 bits and / or the last m2 bits of the data to be encoded, where m1, m2 > 0. For two-dimensional data, the nearest neighbor may be adjacent data to the data to be encoded, or data within the outer n rings of the data to be encoded, etc., where n > 0. This embodiment does not limit the definition of the nearest neighbor. Therefore, the first edge information is obtained based on multiple data sets, and the second edge information is obtained based on at least one data point that is the nearest neighbor to the data to be encoded. Compared to the second edge information, the first edge information has higher data utilization and more comprehensive content.

[0331] When performing entropy decoding on the bitstream, the first context information is obtained by inputting at least one decoded data into the attention decoding network, and the first side information is obtained by entropy decoding the bitstream based on the first side information. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into the masking convolutional network. The second side information is obtained by entropy decoding the bitstream based on the second side information.

[0332] The following examples illustrate the structure of the encoder and decoder using several methods of referencing information. In one implementation, the referencing information only includes the first context information; please refer to [link / reference]. Figure 9 and Figure 10 , Figure 9 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application. Figure 10 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application. Figure 9 As shown, the encoder includes an encoding network, a quantization module, a self-attention decoding network, and an entropy encoding module. For the aforementioned... Figure 8The same network or module performs the same function, and will not be described in detail here. The self-attention decoding network is used to extract first context information from the quantized feature data, and the entropy encoding module is used to perform entropy encoding on the quantized feature data according to the first context information to obtain the bitstream. For example... Figure 10 As shown, the decoder includes a self-attention decoding network, an entropy decoding module, and a decoding network. The self-attention decoding network is used to extract first context information from the decoded data, and the entropy decoding module is used to perform entropy decoding on the bitstream based on the first context information.

[0333] In another implementation, the reference information includes only the first side information; accordingly, please refer to... Figure 11 and Figure 12 , Figure 11 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application. Figure 12 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application. Figure 11 As shown, the encoder includes an encoding network, a self-attention encoding network, a quantization module, a decomposition entropy model, an entropy encoding module, an entropy decoding module, and a self-attention decoding network. The self-attention encoding network extracts first-side information from the feature data after feature extraction. The decomposition entropy model estimates the probability distribution of the obtained first-side information. The entropy encoding module entropy-encodes the first-side information according to its estimated probability distribution to obtain bitstream 2. The entropy decoding module entropy-decodes bitstream 2 according to its estimated probability distribution to obtain the first-side information. The self-attention decoding network estimates the probability distribution of the current data stream based on the first-side information. The entropy encoding module entropy-encodes the quantized feature data according to its estimated probability distribution to obtain bitstream 1.

[0334] like Figure 12 As shown, the decoder includes an entropy decoding module, a self-attention decoding network, and a decoding network. The entropy decoding module is used to perform entropy decoding on bitstream 2 to obtain the first side information. The self-attention decoding network is used to estimate the estimated probability distribution of bitstream 1 based on the first side information. The entropy decoding module is used to perform entropy decoding on bitstream 1 based on the estimated probability distribution of bitstream 1.

[0335] In another implementation, the reference information includes first context information and first side information; accordingly, please refer to... Figure 13 and Figure 14 , Figure 13 This is a schematic diagram of the structure of an encoder provided in an embodiment of this application. Figure 14 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application. Figure 13As shown, the encoder includes an encoding network, a self-attention encoding network, a quantization module, a decomposition entropy model, an entropy encoding module, an entropy decoding module, and a self-attention decoding network. The function of each module can be found in [reference needed]. Figure 9 and Figure 11 The functions of the corresponding modules are not elaborated here in this embodiment. The self-attention decoding network is used to extract the first context information from the quantized feature data, and to estimate the probability distribution of the current data stream based on the first context information and the first side information.

[0336] like Figure 14 As shown, the decoder includes an entropy decoding module, a self-attention decoding network, and a decoding network. The self-attention decoding network is used to extract first context information from the decoded data and estimate the probability distribution of bitstream 1 based on the first context information and the first side information.

[0337] Both self-attention decoding networks and self-attention encoding networks are neural networks that possess a self-attention mechanism (i.e., include a self-attention structure). The self-attention mechanism is a variant of the attention mechanism, which reduces dependence on external information and is able to better capture the internal correlations of data or features.

[0338] Please refer to Figure 15 , Figure 15 This diagram illustrates a self-attention structure provided in an embodiment of this application. The input to this self-attention structure includes three tensors: query (Q), key (K), and value (V). The self-attention structure includes matrix multiplication (MatMul) operations, scaling (Scale) operations, masking (Mask) operations, and exponential normalization (Softmax) operations.

[0339] Please refer to Figure 16 , Figure 16 This is a schematic diagram of the structure of a self-attention coding network provided in an embodiment of this application. The self-attention coding network includes an operation of encoding the input embedding position and an N1 part. The N1 part includes a multi-head attention mechanism operation, a summation and normalization operation, and a feedforward operation.

[0340] Please refer to Figure 17 , Figure 17 This is a schematic diagram of the structure of a self-attention decoding network provided in an embodiment of this application. The self-attention decoding network includes an operation of encoding the input embedding position and an N2 part. The N2 part includes a masked multi-head attention mechanism operation, a summation and normalization operation, and a feedforward operation.

[0341] Please refer to Figure 18 , Figure 18This is a flowchart of process 100 of the entropy encoding method provided in an embodiment of this application. Process 100 can be executed by an encoder, specifically by the entropy encoding unit of the encoder. Process 100 is described as a series of steps or operations. It should be understood that process 100 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 18 The execution order is shown. Assuming a current data stream with multiple data points is being encoded using an encoder, process 100 is executed to entropy encode the data, including the following steps: Process 800 may include:

[0342] Step 101: Obtain the data to be encoded. The data to be encoded is the non-first-order encoded data among the multiple data contained in the current data stream.

[0343] Step 102: Obtain reference information. The reference information includes at least one of first context information and first side information. The first context information is obtained by inputting at least one encoded data into the attention decoding network, and the first side information is obtained by inputting multiple data into the attention encoding network.

[0344] Step 103: Estimate the first estimated probability distribution based on the reference information.

[0345] Step 104: Perform entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain the first bitstream.

[0346] Please refer to Figure 19 , Figure 19 This is a flowchart of process 200 of the entropy decoding method provided in an embodiment of this application. Process 200 can be executed by a decoder, specifically by the entropy decoding unit of the decoder. Process 200 is described as a series of steps or operations. It should be understood that process 200 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 19 The execution order is shown. Assuming a current data stream with multiple data points is being decoded, process 200 is executed to entropy encode and decode the data, including the following steps: Process 200 may include:

[0347] Step 201: Obtain the first bitstream.

[0348] Step 202: Obtain reference information. The reference information includes at least one of first context information and decoded first side information. The first context information is obtained by inputting at least one decoded data into the attention decoding network. The decoded first side information is obtained by entropy decoding of the second bitstream.

[0349] Step 203: Estimate the first estimated probability distribution based on the reference information.

[0350] Step 204: Perform entropy decoding on the first bitstream according to the first estimated probability distribution to obtain decoded data. The decoded data is the non-first-order decoded data among the multiple data contained in the current datastream.

[0351] Please refer to Figure 20 , Figure 20 This is a flowchart of process 300 of the entropy encoding / decoding method provided in an embodiment of this application. Process 300 can be executed by an encoder and a decoder, specifically by the entropy encoding unit of the encoder and the entropy decoding unit of the decoder. Process 300 is described as a series of steps or operations. It should be understood that process 300 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 20 The execution order is shown. Assuming a current data stream with multiple data points is being encoded and decoded using an encoder and decoder, process 300 is executed to entropy encode and decode the data, including the following steps: Process 300 may include:

[0352] Step 301: The encoder obtains the data to be encoded, which is the non-first-order encoded data among the multiple data contained in the current data stream.

[0353] Multiple data points can also be referred to as multiple data units. These data points can include video data, image data, audio data, integer data, and other data requiring compression / decompression. This application does not limit the data type. Each data point corresponds to a location information point, and the data to be encoded is not located at the first position among the multiple data points.

[0354] The current data stream can be in one-dimensional or two-dimensional format, etc., and the embodiments of this application do not limit the format of the current data stream. Optionally, when the initial data stream is in a non-one-dimensional format (e.g., two-dimensional format), the encoder can directly use the initial data stream as the current data stream, or it can flatten the non-one-dimensional initial data stream into a one-dimensional format to obtain the current data stream. In this case, each data can be regarded as a "word" in the text.

[0355] For example, when flattening an initial data stream that is not in a one-dimensional format into a one-dimensional format, it can be flattened in a preset order. For example, when the initial data stream is in a two-dimensional format, the two-dimensional initial data stream can be flattened in a top-to-bottom and left-to-right order, or in a bottom-to-top and left-to-right order, or in a preset order. The embodiments of this application do not limit the flattening order.

[0356] In this embodiment, after acquiring the data to be encoded, quantization processing can be performed on the data to be encoded. This reduces the amount of data required to represent the data to be encoded, thereby reducing the code rate in the subsequent entropy encoding process and effectively reducing the entropy encoding overhead. As mentioned above, quantization processing can be performed by, for example, scalar quantization or vector quantization. This embodiment does not limit the method of quantization processing.

[0357] It should be noted that when entropy encoding multiple data points contained in the current data stream, the first data point is typically entropy encoded first, followed by the data to be encoded. For the first data point, a fourth estimated probability distribution can be estimated based on pre-set information. Alternatively, a learned model obtained through training can be used to estimate the fourth estimated probability distribution, and then entropy encoding is performed on the first data point based on the fourth estimated probability distribution to obtain the fourth bitstream. This application does not limit the method for obtaining the fourth estimated probability distribution.

[0358] Step 302: The encoder obtains the first context information.

[0359] The first context information is obtained by inputting at least one encoded data from multiple data streams included in the current data stream into the attention decoding network. This encoded data refers to data that the encoder has already entropy-encoded among the multiple data streams. Since no encoded data exists when entropy-encoding the first data of the current data stream, the data to be encoded needs to be non-first data of the current data stream to extract the first context information. The first context information obtained based on at least one encoded data from multiple data streams has less data redundancy and higher utilization of the encoded data. This improves the accuracy of the first estimated probability distribution when subsequently estimated using the first context information. Since a higher accuracy of the first estimated probability distribution results in a lower code rate during entropy encoding, inputting at least one encoded data from multiple data streams into the attention decoding network to obtain the first context information reduces the code rate during entropy encoding, thereby reducing the entropy encoding overhead. Here, the code rate is the average code length required to entropy-encode a unit of data.

[0360] A self-attention decoding network is a neural network with a self-attention mechanism (i.e., including a self-attention structure). It has a global receptive field and can obtain the correlation between all the encoded input data and the data to be encoded. This correlation can be represented as the weights of all the encoded input data relative to the data to be encoded. After obtaining the weights of all the encoded input data relative to the data to be encoded, the self-attention decoding network weights the corresponding encoded data according to the weights to obtain the first context information.

[0361] Optionally, the self-attention decoding network can weight all the encoded input data using appropriate weights to obtain the first context information. This improves the utilization rate of the encoded data during the acquisition of the first context information. When subsequently using the first context information to estimate the first estimated probability distribution, the accuracy of the obtained first estimated probability distribution can be further improved, further reducing the code rate in the entropy coding process, and thus further reducing the entropy coding overhead.

[0362] Alternatively, a self-attention decoding network can select a portion of the encoded input data based on the obtained weights and weight this portion of encoded data accordingly to obtain the first context information. For example, the obtained weights can be sorted in descending order, and the encoded data corresponding to the weights in the first i1 positions can be selected for weighting. Alternatively, the obtained weights can be sorted in ascending order, and the encoded data corresponding to the weights in the last i2 positions can be selected for weighting. Or, encoded data with weights greater than i2 can be selected for weighting. Here, i1 and i2 are both less than the total number of obtained weights. This improves the flexibility in obtaining the first context information. Furthermore, when selecting encoded data with higher weights for weighting, the utilization rate of encoded data with higher weights in obtaining the first context information can be guaranteed. This further improves the accuracy of the first estimated probability distribution when using the first context information to estimate it, further reducing the code rate in the entropy coding process, and thus further reducing the entropy coding overhead.

[0363] After the current data stream is input into the self-attention decoding network, the network performs an embedding operation on each data point in the stream. Embedding involves transforming each data point from its original data space to another. Then, positional encoding is performed on each data point to obtain its positional information, which is then combined with the data. Each data point has coordinate information; positional encoding extracts the positional information of each data point based on its coordinates. The positional information can be combined with the data using bitwise addition or concatenation, but this embodiment does not limit the method of positional encoding.

[0364] The structure of this self-attention decoding network can be referred to the aforementioned. Figure 17 The embodiments described in this application will not be repeated here. As mentioned above... Figure 17As shown, the input to the self-attention decoding network includes three tensors Q, K, and V. Q, K, and V sequentially pass through a masked multi-head self-attention mechanism, a summation and normalization operation, a multi-head attention mechanism, a summation and normalization operation, a feedforward operation, a summation and normalization operation, and a linearization operation, outputting the first context information. Q, K, and V refer to the tensors of the encoded data; for example, they can be the tensors obtained after embedding and positional encoding of the non-first-order encoded data in the aforementioned process.

[0365] Step 303: The encoder estimates the first estimated probability distribution based on the first context information.

[0366] The first estimated probability distribution may include at least one estimated probability parameter. For example, the at least one estimated probability parameter may include a mean and a scale, where the mean and scale form a Gaussian distribution.

[0367] Optionally, the first context information can be input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network. This probability distribution estimation network can be a single neural network or a structure within a self-attention decoding network; this embodiment of the application does not limit this.

[0368] Please refer to Figure 21 , Figure 21 This is a schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application. Figure 21 Taking an initial data stream in a two-dimensional format as an example, which needs to be flattened, the initial data stream 'a' includes 16 positions a1 to a16 arranged in a 4×4 pattern, with each position corresponding to one piece of data. The data corresponding to position a10 is the data to be encoded, the data corresponding to positions a1 to a9 are already encoded, and the data corresponding to the remaining positions are unencoded. Each encoded piece of data corresponds to a first estimated probability distribution. Flattening the initial data stream 'a' into a one-dimensional format from top to bottom and left to right yields the current data stream 'b', which includes the 16 sequentially arranged positions a1 to a16. The current data stream 'b' is input into a self-attention decoding network. The self-attention decoding network determines the position information of each piece of data in the current data stream 'b' and combines the position information with the data. Based on the encoded data in the data stream 'b' combined with position information (i.e., the data corresponding to positions a1 to a9), the self-attention decoding network outputs first context information. This first context information is input into a probability distribution estimation network, which outputs a first estimated probability distribution, i.e., the estimated probability distribution of the data corresponding to position a10. Figure 21 The process shown is merely illustrative and does not limit the process of obtaining the first estimated probability distribution.

[0369] Step 304: The encoder performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain the first bitstream.

[0370] The encoder can calculate the probability value of the data to be encoded based on a first estimated probability distribution, and then perform entropy encoding on the data to be encoded based on this probability value. The first bitstream can be in binary format.

[0371] Steps 301 to 304 described above are illustrated using the example of estimating a first estimated probability distribution and then performing entropy encoding on the data to be encoded based on that first estimated probability distribution to obtain a first bitstream. Each non-first-order data item in the current data stream can be taken as the data to be encoded. Following the process shown in steps 301 to 304, the first estimated probability distribution is obtained, and entropy encoding is performed based on this distribution to obtain the bitstream for each non-first-order data item. It should be noted that after each data item is encoded, it is added to the already encoded data.

[0372] Step 305: The encoder sends the first bitstream to the decoder.

[0373] As described in the foregoing embodiments, the encoder and decoder have communication interfaces that are connected, and the encoder can send a first bit stream to the communication interface of the decoder through the communication interface.

[0374] It should be noted that the encoder performs entropy encoding on each non-first-order encoded data in the current data stream to obtain a bitstream of each non-first-order data. Then, based on the bitstream of each non-first-order data, the current bitstream is obtained. The current bitstream includes multiple bitstreams of non-first-order encoded data arranged in the encoding order of the encoder. Of course, the current bitstream includes the first bitstream. The encoder can then send the current bitstream, including the first bitstream, to the decoder. For first-order encoded data, the encoder performs entropy encoding on the first-order encoded data according to the fourth estimated probability distribution to obtain a fourth bitstream, which can be included in the current bitstream for transmission to the decoder. Alternatively, the encoder can send the fourth bitstream to the decoder separately; this embodiment does not limit the method of sending the fourth bitstream.

[0375] Step 306: The decoder obtains the first context information.

[0376] As described in step 305 above, the first bitstream belongs to one of the current bitstreams received by the decoder. The decoded data obtained after decoding the first bitstream is the data that is not the first bit decoded among the multiple data contained in the current data stream. The first context information can be obtained by inputting at least one decoded data into the attention decoding network. The decoded data refers to the data obtained by entropy decoding before decoding the first bitstream. Since there is no decoded data when entropy decoding is performed on the fourth bitstream, the decoded data obtained after decoding the first bitstream is the data that is not the first bit decoded among the multiple data contained in the current data stream, so that the first context information can be extracted. The process of obtaining the first context information can refer to step 302 above, and will not be described in detail here.

[0377] It should be noted that when the decoder performs entropy decoding on each bitstream in the received current bitstream, it typically performs entropy decoding on the fourth bitstream first. The decoder can estimate the fourth estimated probability distribution based on pre-set information. Alternatively, it can estimate the fourth estimated probability distribution using a trained learnable model, and then perform entropy decoding on the fourth bitstream based on the fourth estimated probability distribution to obtain the decoded first bit data. The decoded first bit data is the first decoded data among multiple data. This application does not limit the method of obtaining the fourth estimated probability distribution in its embodiments.

[0378] The fourth estimated probability distribution obtained by the decoder needs to be consistent with the fourth estimated probability distribution obtained by the encoder. For example, when the encoder estimates the fourth estimated probability distribution based on pre-set information, the decoder estimates the fourth estimated probability distribution based on the same fixed information. When the encoder estimates the fourth estimated probability distribution using a learned model obtained through training, the decoder estimates the fourth estimated probability distribution based on the same learned model, and the estimated fourth estimated probability distributions are identical.

[0379] Step 307: The decoder estimates the first estimated probability distribution based on the first context information.

[0380] Optionally, the first context information can be input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network. This process can refer to step 303 above, and will not be repeated here in the embodiments of this application.

[0381] Step 308: The decoder performs entropy decoding on the first bitstream according to the first estimated probability distribution to obtain decoded data. The decoded data is the non-first-order decoded data among the multiple data contained in the current data stream.

[0382] The decoder can calculate the probability value of the first bitstream based on the first estimated probability distribution, and then perform entropy decoding on the first bitstream based on the probability value. It should be noted that after each decoded data is obtained, the decoded data is added to the already decoded data.

[0383] Steps 306 to 308 described above are illustrated using the example of estimating a first estimated probability distribution and performing entropy decoding on the first bitstream based on the first estimated probability distribution. Each bitstream included in the current bitstream can be taken as the first bitstream, and the first estimated probability distribution can be obtained according to the process shown in steps 306 to 308 above, and entropy decoding can be performed based on the first estimated probability distribution.

[0384] After entropy decoding of the current bitstream, the resulting decoded data is in one-dimensional format. The decoder can transform the format of the decoded data into one-dimensional data based on the two-dimensional distribution information of the decoded data, so as to obtain two-dimensional decoded data with the same arrangement as the current data stream acquired by the encoder. This two-dimensional distribution information may include the number and arrangement of decoded data arranged in the length and width directions of the two-dimensional plane, respectively. The two-dimensional distribution information can be pre-stored in the decoder or obtained by the encoder. This application embodiment does not limit the content and acquisition method of the two-dimensional distribution information, as long as it can ensure that the arrangement of the two-dimensional decoded data is the same as that of the current data stream acquired by the encoder.

[0385] In related technologies, during encoding, the adjacent encoded data of the data to be encoded are determined based on the position information of each data point. A masked convolutional neural network is then used to extract context information from the adjacent encoded data, and entropy encoding is performed on the data to be encoded based on this context information. During decoding, the adjacent decoded data of the data corresponding to each bitstream are determined based on the position information of the data to be decoded. A masked convolutional neural network is then used to extract context information from the adjacent decoded data, and entropy decoding is performed on the bitstream to be decoded based on this context information.

[0386] Because related technologies require extracting context information from adjacent encoded or decoded data, this process needs to be executed according to the position information of each data point. Therefore, the entropy encoding or decoding process for multiple data points needs to be executed serially according to the order of the data points, which is time-consuming and results in low efficiency for entropy encoding and decoding. In contrast, in the embodiments of this application, the first context information is extracted from at least one encoded or decoded data point without considering the position encoding of each data point. Therefore, the entropy encoding or decoding process for multiple data points can be executed in parallel, which is less time-consuming and improves the efficiency of entropy encoding and decoding compared to related technologies.

[0387] Furthermore, related technologies utilize masked convolutional neural networks to extract contextual information. However, this extraction only uses a local receptive field, resulting in low utilization of encoded or decoded data. This leads to low accuracy of the estimated probability distribution obtained from the contextual information, resulting in high overhead for entropy encoding and decoding. In contrast, this application's embodiment uses a self-attention decoding network with a self-attention mechanism to obtain the weights of all encoded or decoded input data. Then, the weighted average of some / all encoded or decoded input data is used to obtain the first contextual information. This improves the utilization of encoded or decoded data, reduces data redundancy in the extracted first contextual information, and further improves the accuracy of the estimated probability distribution. Compared to related technologies, this reduces the code rate in the entropy encoding process, thereby reducing the overhead of entropy encoding and decoding.

[0388] Please refer to Figure 22 , Figure 22 This is a schematic diagram illustrating the entropy coding performance provided in an embodiment of this application. Figure 22 Coordinate system (22a) shows the performance of entropy coding on the test set using the embodiments of this application and related techniques under the Multi-Scale Structural Similarity Index Measure (MS-SSIM) metric, and coordinate system (22b) shows the performance of entropy coding on the test set using the embodiments of this application and related techniques under the Peak Signal-to-Noise Ratio (PSNR) metric. The test set is the Kodak test set, which includes 24 images in Portable Network Graphics (PNG) format. The resolution of these 24 images can be 768×512 or 512×768. Figure 22 In the two coordinate systems, the horizontal axis represents pixel depth (Bits per pixel, BPP), and the vertical axis represents the bitrate. BPP represents the average number of bits used per pixel; a smaller value indicates a lower compression bitrate. MS-SSIM and PSNR are both objective standards for evaluating images; higher values ​​indicate better image quality. The polyline e1 in coordinate systems (12a) and (12b) represents an embodiment of this application, and the polyline e2 represents related technologies. Figure 22As can be seen, at various bitrates, the MSSSIM and PSNR indices of this application embodiment are higher than those of related technologies. Under the same compression quality, the bitrate of this application embodiment is lower than that of related technologies, typically 17% lower at low bitrates and 15% lower at high bitrates. That is, the compression performance of this application embodiment is higher than that of related technologies, and this application embodiment can improve the accuracy of the estimated probability distribution of the acquired data to be encoded or decoded.

[0389] In summary, the entropy encoding / decoding method provided in this application involves an encoder acquiring the current data stream and first context information, then estimating a first estimated probability distribution based on the first context information, and performing entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain a first bitstream. The encoder then sends the first bitstream to the decoder. The decoder acquires the first bitstream and the first context information, estimates the first estimated probability distribution based on the first context information, and then performs entropy decoding on the first bitstream according to the first estimated probability distribution. The first context information is obtained by inputting at least one encoded or decoded data into a self-attention decoding network. The self-attention decoding network can weight all input encoded data using appropriate weights to obtain the first context information. This improves the utilization rate of encoded data during the acquisition of the first context information. When estimating the first estimated probability distribution using the first context information, the accuracy of the obtained first estimated probability distribution can be improved, further reducing the bitrate in the entropy encoding process and further reducing the entropy encoding overhead. This reduces the bandwidth occupancy rate of the first bitstream transmission to the decoder and improves the transmission efficiency of the first bitstream to the decoding side. Furthermore, the position information of each data point does not need to be considered during the acquisition of the first context information. Therefore, the entropy encoding or entropy decoding processes of multiple data points can be executed in parallel. Parallel execution takes less time, which improves the efficiency of entropy encoding and entropy decoding.

[0390] The order of the methods provided in the embodiments of this application can be appropriately adjusted, and the steps can be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit this.

[0391] Please refer to Figure 23 , Figure 23 This is a flowchart of process 400 of the entropy encoding / decoding method provided in an embodiment of this application. Process 400 can be executed by an encoder and a decoder, specifically by the entropy encoding unit of the encoder and the entropy decoding unit of the decoder. Process 400 is described as a series of steps or operations. It should be understood that process 400 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 23The execution order is shown. Assuming a current data stream with multiple data points is being encoded and decoded using an encoder and decoder, process 400 is executed to entropy encode and decode the data, including the following steps: Process 400 may include:

[0392] Step 401: The encoder obtains the data to be encoded contained in the current data stream.

[0393] The data to be encoded can be the first encoded data or non-first encoded data among multiple data contained in the current data stream. This embodiment does not limit the position of the data to be encoded in the current data stream. This process can refer to step 301 above, and will not be repeated here.

[0394] Step 402: The encoder obtains the information of the first side.

[0395] The first-side information is obtained by inputting multiple data points into an attention encoding network. As mentioned above... Figure 21 Taking the initial data stream 'a' as an example, the data corresponding to positions a1 to a16 can be input into the self-attention coding network to obtain the first side information. The first side information obtained based on multiple data points is relatively comprehensive. When subsequently using the first side information to estimate the second estimated probability distribution, the accuracy of the obtained second estimated probability distribution can be improved, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0396] Self-attention coding networks are neural networks that possess a self-attention mechanism (i.e., include a self-attention structure). They have good feature transformation capabilities, and the quality of the extracted first-side information is relatively good. When using the first-side information to estimate the first estimated probability distribution, they can improve the accuracy of the obtained first estimated probability distribution, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0397] Self-attention encoding networks possess a global receptive field, enabling them to determine the correlation between all input data and the data to be encoded. This correlation can be represented as the weights of all input data relative to the data to be encoded. After obtaining the weights of all input data relative to the data to be encoded, the self-attention encoding network weights the corresponding data according to these weights to obtain the first-side information.

[0398] Optionally, the self-attention coding network can weight all input data using appropriate weights to obtain the first-side information. This improves the data utilization rate during the acquisition of the first-side information. When subsequently using the first-side information to estimate the first estimated probability distribution, the accuracy of the obtained first estimated probability distribution can be further improved, further reducing the code rate in the entropy coding process, thereby further reducing the entropy coding overhead.

[0399] Alternatively, a self-attention coding network can select a portion of the input data based on the obtained weights and weight the data accordingly to obtain the first-side information. This process can refer to step 302 described above, and will not be repeated here. This improves the flexibility in obtaining the first-side information. Furthermore, when selecting data with higher weights for weighting, it ensures the utilization rate of these data during the acquisition of the first-side information. This further improves the accuracy of the first estimated probability distribution when subsequently using the first-side information to estimate it, further reducing the code rate in the entropy coding process, and thus further reducing the entropy coding overhead.

[0400] In step 402, the structure of the self-attention encoding network can be referred to the above. Figure 16 The embodiments described in this application will not be repeated here. As mentioned above... Figure 16 As shown, the input to the self-attention encoding network includes three tensors Q, K, and V. Q, K, and V are sequentially processed through a multi-head attention mechanism, summation and normalization operations, feedforward operations, and summation and normalization operations, outputting the first-side information. Q, K, and V refer to the data tensors, such as the tensors obtained after embedding and positional encoding of the data in the current data stream in the aforementioned process.

[0401] Step 403: The encoder estimates the first estimated probability distribution based on the first side information.

[0402] Optionally, the first side information can be input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network. This probability distribution estimation network can be a single neural network or a structure within a self-attention decoding network; this embodiment of the application does not limit this.

[0403] Please refer to Figure 24 , Figure 24 This is a schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application. Figure 24 Taking the initial data stream as a two-dimensional format, which needs to be flattened, and the probability distribution estimation performed by the self-attention decoding network as an example, the explanation is as follows: The initial data stream 'a' consists of 16 positions a1 to a16 arranged in a 4×4 pattern, with each position corresponding to one data point. Flattening the initial data stream 'a' into a one-dimensional format in a top-to-bottom and left-to-right order yields the current data stream 'b', which includes the 16 sequentially arranged positions a1 to a16.

[0404] The current data stream b is input into a self-attention encoding network. The self-attention encoding network determines the position information of each data point in the current data stream b, and combines the position information of each data point with the data. Based on all the data in the data stream b with combined position information (i.e., the data corresponding to positions a1 to a16), the first side information is output. The entropy model is decomposed to estimate the second estimated probability distribution. The entropy encoding module uses the second estimated probability distribution to entropy encode the first side information to obtain the bitstream of the first side information. The entropy decoding module uses the second estimated probability distribution to entropy decode the bitstream of the first side information to obtain the first side information. The first side information is input into a self-attention decoding network, and the self-attention decoding network outputs the first estimated probability distribution (i.e., the estimated probability distribution of the data corresponding to position a10). This process can refer to the aforementioned step 303, and will not be repeated here in this embodiment. Figure 24 The process shown is merely illustrative and does not limit the process of obtaining the first estimated probability distribution.

[0405] Step 404: The encoder performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain the first bitstream.

[0406] The encoder can calculate the probability value of the data to be encoded based on the first estimated probability distribution, and then perform entropy encoding on the data to be encoded based on the probability value. The first bitstream can be in binary format. This process can refer to step 304 above, and will not be repeated here in the embodiments of this application.

[0407] Step 405: The encoder sends the first bitstream to the decoder.

[0408] This process can refer to step 305 above, and will not be repeated here in the embodiments of this application.

[0409] Step 406: The encoder estimates the second estimated probability distribution.

[0410] Optionally, the second estimated probability distribution can be estimated based on pre-set information. Alternatively, it can be estimated using a learned model obtained through training. This application does not limit the method for obtaining the second estimated probability distribution.

[0411] Step 407: The encoder performs entropy encoding on the first side information according to the second estimated probability distribution to obtain the second bitstream.

[0412] The encoder can calculate the probability value of the first side information based on the second estimated probability distribution, and then perform entropy encoding on the first side information based on the probability value. The second bitstream can be in binary format.

[0413] Step 408: The encoder sends the second bitstream to the decoder.

[0414] This process can refer to step 305 above, and will not be repeated here in the embodiments of this application.

[0415] Step 409: The decoder estimates the second estimated probability distribution.

[0416] Optionally, the second estimated probability distribution can be estimated based on pre-set information. Alternatively, it can be estimated using a trained learnable model. This application does not limit the method of obtaining the second estimated probability distribution. It should be noted that the second estimated probability distribution estimated by the decoder needs to be consistent with the second estimated probability distribution estimated by the encoder. For example, when the encoder estimates the second estimated probability distribution based on pre-set information, the decoder estimates the second estimated probability distribution based on the same fixed information. When the encoder estimates the second estimated probability distribution using a trained learnable model, the decoder estimates the second estimated probability distribution based on the same learnable model, and the estimated second estimated probability distributions are identical.

[0417] Step 410: The decoder performs entropy decoding on the second bitstream according to the second estimated probability distribution to obtain the decoded first side information.

[0418] The decoder can calculate the probability value of the second bitstream based on the second estimated probability distribution, and then perform entropy decoding on the second bitstream based on the probability value.

[0419] Step 411: The decoder estimates the first estimated probability distribution based on the decoded first side information.

[0420] Optionally, the decoded first-side information can be input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network. This process can refer to step 307 above, and will not be repeated here in the embodiments of this application.

[0421] Step 412: The decoder performs entropy decoding on the first bitstream according to the first estimated probability distribution to obtain the decoded data.

[0422] The decoder can calculate the probability value of the first bitstream based on the first estimated probability distribution, and then perform entropy decoding on the first bitstream based on the probability value. This process can be referred to step 308 above, and will not be repeated here in the embodiments of this application.

[0423] In summary, the entropy encoding / decoding method provided in this application involves an encoder acquiring the data to be encoded and first side information contained in the current data stream. Then, it estimates a first estimated probability distribution based on the first side information and performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain a first bitstream. The encoder then sends the first bitstream to the decoder. The encoder estimates a second estimated probability distribution and performs entropy encoding on the first side information according to the second estimated probability distribution to obtain a second bitstream. The encoder then sends the second bitstream to the decoder, which estimates a second estimated probability distribution and performs entropy decoding on the second bitstream to obtain the first side information. The encoder then estimates a first estimated probability distribution based on this first side information and performs entropy decoding on the first bitstream again. The first side information is obtained by inputting multiple data points into a self-attention encoding network. The self-attention encoding network can weight all input data using appropriate weights to obtain the first side information. This results in more comprehensive first side information. When using the first side information to estimate the first estimated probability distribution in the subsequent process, the accuracy of the obtained first estimated probability distribution can be improved, the code rate in the entropy coding process can be reduced, thereby reducing the entropy coding overhead and the bandwidth occupancy rate when the first bit stream is transmitted to the decoder, and improving the transmission efficiency of the first bit stream to the decoder.

[0424] The order of the methods provided in this application can be adjusted appropriately, and the steps can be added or removed as needed. For example, steps 403 to 405 and steps 406 to 408 can be executed simultaneously. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and this application does not limit them.

[0425] Please refer to Figure 25 , Figure 25 This is a flowchart of process 500 of the entropy encoding / decoding method provided in an embodiment of this application. Process 500 can be executed by an encoder and a decoder, specifically by the entropy encoding unit of the encoder and the entropy decoding unit of the decoder. Process 500 is described as a series of steps or operations. It should be understood that process 500 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 25 The execution order is shown. Assuming a current data stream with multiple data points is being encoded and decoded using an encoder and decoder, process 500 is executed to entropy encode and decode the data, including the following steps: Process 500 may include:

[0426] Step 501: The encoder obtains the data to be encoded, which is the non-first-order encoded data among the multiple data contained in the current data stream.

[0427] This process can refer to the aforementioned step 301, and will not be repeated here in the embodiments of this application.

[0428] Step 502: The encoder obtains the first context information and the first side information.

[0429] This process can refer to the aforementioned steps 302 and 402, and will not be repeated here in the embodiments of this application.

[0430] Step 503: The encoder estimates the first estimated probability distribution based on the first context information and the first side information.

[0431] The encoder can aggregate the first context information and the first edge information, and estimate the first estimated probability distribution based on the aggregated information. Optionally, the encoder can aggregate the first context information and the first edge information through an aggregation network. The aggregation network may include a self-attention decoding network, which has a self-attention mechanism and can fully utilize the complementarity of the first context information and the first edge information. Subsequently, it can use these two pieces of information to efficiently estimate the first estimated probability distribution, thereby improving the accuracy of the estimated first estimated probability distribution. The process of estimating the first estimated probability distribution based on the aggregated information can refer to the aforementioned step 303, and will not be repeated here in this embodiment.

[0432] Step 504: The encoder performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain the first bitstream.

[0433] This process can refer to step 304 above, and will not be repeated here in the embodiments of this application.

[0434] Step 505: The encoder sends the first bitstream to the decoder.

[0435] It should be noted that the encoder performs entropy encoding on each non-first-order encoded data in the current data stream to obtain the bitstream of each non-first-order data. Then, the current bitstream is obtained based on the bitstream of each non-first-order data. For first-order encoded data, its bitstream can be included in the current bitstream for transmission to the decoder. Alternatively, the encoder can send its bitstream to the decoder separately. This process can refer to step 305 above, and will not be elaborated upon here in this embodiment.

[0436] Step 506: The encoder estimates the second estimated probability distribution.

[0437] This process can refer to the aforementioned step 406, and will not be repeated here in the embodiments of this application.

[0438] Step 507: The encoder performs entropy encoding on the first side information according to the second estimated probability distribution to obtain the second bitstream.

[0439] This process can refer to the aforementioned step 407, and will not be repeated here in the embodiments of this application.

[0440] Step 508: The encoder sends the second bitstream to the decoder.

[0441] Optionally, the encoder can send the second bitstream to the decoder separately, or it can add the second bitstream to the first bitstream before sending it to the decoder. This embodiment does not limit the method of sending the second bitstream. This process can be referred to step 405 above, and will not be repeated here.

[0442] Step 509: The decoder obtains the first context information.

[0443] The method for obtaining the first context information can refer to the aforementioned step 306, and will not be repeated here in the embodiments of this application.

[0444] Step 510: The decoder estimates the second estimated probability distribution.

[0445] The second bitstream is the bitstream of the first side information. This process can be referred to step 409 above, and will not be repeated here in the embodiments of this application. It should be noted that the second estimated probability distribution estimated by the decoder needs to be consistent with the second estimated probability distribution estimated by the encoder.

[0446] Step 511: The decoder performs entropy decoding on the second bitstream according to the second estimated probability distribution to obtain the decoded first side information.

[0447] This process can refer to the aforementioned step 410, and will not be repeated here in the embodiments of this application.

[0448] Step 512: The decoder estimates the first estimated probability distribution based on the first context information and the decoded first side information.

[0449] This process can refer to the aforementioned steps 303 and 403, and will not be repeated here in the embodiments of this application.

[0450] Step 513: The decoder performs entropy decoding on the first bitstream according to the first estimated probability distribution to obtain decoded data. The decoded data is the non-first-order decoded data among the multiple data contained in the current data stream.

[0451] The process of entropy decoding of the first bitstream can refer to step 308 above, and will not be repeated here in this embodiment. It should be noted that after each piece of data is decoded, it is added to the decoded data.

[0452] In summary, the entropy encoding / decoding method provided in this application involves an encoder acquiring the current data stream, first context information, and first pass information. Then, it estimates a first estimated probability distribution based on the first context information and first side information, and performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain a first bitstream. The encoder then sends the first bitstream to the decoder. The encoder estimates a second estimated probability distribution, performs entropy encoding on the first side information according to the second estimated probability distribution to obtain a second bitstream, and sends the second bitstream to the decoder. The decoder estimates a second estimated probability distribution, performs entropy decoding on the second bitstream according to the second estimated probability distribution to obtain first side information. The decoder estimates the first estimated probability distribution based on the first context information and the first side information, and then performs entropy decoding on the first bitstream according to the first estimated probability distribution. The first context information is obtained by inputting at least one encoded or decoded data into a self-attention decoding network. The self-attention decoding network can weight all input encoded data using appropriate weights to obtain the first context information. The first side information is obtained by inputting multiple data into a self-attention encoding network. The self-attention encoding network can weight all input data using appropriate weights to obtain the first side information. This improves the utilization rate of encoded data during the acquisition of the first context information and provides more comprehensive first-side information. When estimating the first estimated probability distribution using the first context information and the first-side information, the accuracy of the estimated probability distribution is improved, further reducing the code rate in the entropy coding process and the entropy coding overhead. This reduces the entropy coding overhead and the bandwidth occupancy when transmitting the first bitstream to the decoder, thus improving the transmission efficiency of the first bitstream to the decoder. Furthermore, since the positional information of each data point does not need to be considered during the acquisition of the first context information, the entropy coding or entropy decoding processes for multiple data points can be executed in parallel. Parallel execution is less time-consuming, improving the efficiency of entropy coding and entropy decoding.

[0453] The order of the methods provided in the embodiments of this application can be appropriately adjusted, and the steps can be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit this.

[0454] Please refer to Figure 26 , Figure 26 This is a flowchart of process 600 of the entropy encoding / decoding method provided in an embodiment of this application. Process 600 can be executed by an encoder and a decoder, specifically by the entropy encoding unit of the encoder and the entropy decoding unit of the decoder. Process 600 is described as a series of steps or operations. It should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 26The execution order is shown. Assuming a current data stream with multiple data points is being encoded and decoded using an encoder and decoder, process 600 is executed to entropy encode and decode the data, including the following steps: Process 600 may include:

[0455] Step 601: The encoder obtains the data to be encoded, which is the non-first-order encoded data among the multiple data contained in the current data stream.

[0456] It should be noted that when entropy encoding multiple data points contained in the current data stream, the first data point is typically entropy encoded first, followed by the data to be encoded. For the first data point, a fourth estimated probability distribution can be estimated based on pre-set information. Alternatively, it can be estimated using a trained learnable model. Or, it can be estimated based on the first side information and / or the second side information. Then, entropy encoding is performed on the first data point based on the fourth estimated probability distribution to obtain the fourth bitstream. This application does not limit the method for obtaining the fourth estimated probability distribution.

[0457] This process can refer to the aforementioned step 301, and will not be repeated here in the embodiments of this application.

[0458] Step 602: The encoder obtains the first context information, the second context information, the first side information, and the second side information.

[0459] The method for obtaining the first context information can be referred to in step 302 above, and the method for obtaining the first side information can be referred to in step 402 above. The embodiments of this application will not be described in detail here.

[0460] The second context information is obtained by inputting at least one data point from at least one encoded data point that meets a preset condition into a masked convolution network. The at least one data point meeting the preset condition can be at least one data point from at least one encoded data point among multiple data points that is the nearest neighbor to the data to be encoded. Utilizing the encoded data in the process of obtaining the second context information improves the accuracy of the first estimated probability distribution obtained subsequently, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0461] Therefore, the aforementioned Figure 20 In the illustrated embodiment, the first context information is obtained based on at least one encoded data among a plurality of data, and the second context information in step 602 is obtained based on at least one data among the at least one encoded data that is adjacent to the data to be encoded. (As described above...) Figure 21Taking the initial data stream shown as an example, the first context information is obtained based on the data corresponding to positions a1 to a9, and the second context information is obtained based on at least one decoded data adjacent to position a10 (e.g., the data corresponding to positions a6 and a9). That is, compared with the second context information, the first context information has a higher utilization rate of the encoded data and is more comprehensive.

[0462] The second-side information is obtained by inputting at least one data point from a set of data that meets preset conditions into a hyper-encoder network. The at least one data point that meets the preset conditions can be at least one data point that is adjacent to the data to be encoded.

[0463] Therefore, the aforementioned Figure 23 In the illustrated embodiment, the first side information is obtained based on multiple data sets, and the second side information in step 602 is obtained based on at least one data set that is closest to the data to be encoded. Taking the initial data stream shown in Figure 21 as an example, the first side information is obtained based on the data corresponding to positions a1 to a16, and the second side information is obtained based on at least one data set that is closest to position a10 (e.g., the data corresponding to positions a6, a9, a11, and a14). That is, compared to the second side information, the first side information has a higher data utilization rate and more comprehensive content.

[0464] Masked convolutional networks or supercoding networks possess local receptive fields. Masked convolutional networks consist of masked convolutional layers or regular convolutional layers. Their input is at least one data point from at least one encoded data set that is the nearest neighbor to the data to be encoded. The output is the activation features of the convolution output, i.e., second contextual information. Supercoding networks consist of regular convolutional layers. Their input is at least one data point from multiple data sets that is the nearest neighbor to the data to be encoded. The output is the activation features of the convolution output, i.e., second side information.

[0465] The method by which the encoder obtains the second context information through the masking convolutional network, the method by which the encoder obtains the second side information through the supercoding network, and the architecture of the masking convolutional network and the supercoding network can all refer to the relevant content of the self-attention decoding network in step 302 above. The embodiments of this application will not be described in detail here.

[0466] In this embodiment, the first estimated probability distribution is subsequently estimated by combining the first context information, the first side information, the second context information, and the second side information. This can further improve the accuracy of the obtained first estimated probability distribution, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0467] Step 603: The encoder estimates the first estimated probability distribution based on the first context information, the second context information, the first side information, and the second side information.

[0468] The encoder can aggregate the first context information, the second context information, the first edge information, and the second edge information, and estimate the first estimated probability distribution based on the aggregated information. Optionally, the encoder can aggregate the first context information, the second context information, the first edge information, and the second edge information through an aggregation network. The aggregation network may include a self-attention decoding network, which has a self-attention mechanism and can fully utilize the complementarity of the first context information, the first edge information, the second context information, and the second edge information. Subsequently, it can use these four pieces of information to efficiently estimate the first estimated probability distribution, thereby improving the accuracy of the estimated first estimated probability distribution. The process of estimating the first estimated probability distribution based on the aggregated information can refer to the aforementioned step 303, and will not be repeated here in this embodiment.

[0469] Please refer to Figure 27 , Figure 27 This is a schematic diagram illustrating the process of obtaining a first estimated probability distribution provided in an embodiment of this application. Figure 27 by Figure 21 Taking the initial data stream shown as an example, the initial data stream a is flattened into a one-dimensional format in a top-down and left-to-right order, resulting in the current data stream b, which includes 16 sequentially arranged positions a1 to a16. The current data stream b is then input into a super-encoding network, a self-attention encoding network, a self-attention decoding network, and a masking convolutional network. The super-encoding network and the self-attention encoding network output the second-side information and the first-side information, respectively. The entropy model estimates the second estimated probability distribution, and the super-entropy model estimates the third estimated probability distribution. The entropy encoding module entropy-encodes the first-side information according to the second estimated probability distribution, and the entropy decoding module entropy-decodes the first-side information according to the second estimated probability distribution, inputting the entropy-decoded first-side information into the aggregation network. The entropy encoding module entropy-encodes the second-side information according to the third estimated probability distribution, and the entropy decoding module entropy-decodes the second-side information according to the third estimated probability distribution, inputting the entropy-decoded second-side information into the aggregation network. The self-attention decoding network and the masking convolutional network output the first and second context information, respectively, both of which are input into the aggregation network. The aggregation network aggregates the first context information, second context information, first side information, and second side information of the input, and outputs the first estimated probability distribution (i.e., the estimated probability distribution of the data corresponding to position a10). Figure 27 The process shown is merely illustrative and does not limit the process of obtaining the first estimated probability distribution.

[0470] Step 604: The encoder performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain the first bitstream.

[0471] The process of obtaining the first bitstream can be referred to step 304 above, and will not be repeated here in the embodiments of this application.

[0472] Step 605: The encoder sends the first bitstream to the decoder.

[0473] It should be noted that the encoder performs entropy encoding on each non-first-order encoded data in the current data stream to obtain the bitstream of each non-first-order data. Then, the current bitstream is obtained based on the bitstream of each non-first-order data. For first-order encoded data, the fourth bitstream can be included in the current bitstream for transmission to the decoder. Alternatively, the encoder can send the fourth bitstream to the decoder separately. This process can refer to step 305 above, and will not be repeated here in the embodiments of this application.

[0474] Step 606: The encoder estimates the second estimated probability distribution.

[0475] This process can refer to the aforementioned step 406, and will not be repeated here in the embodiments of this application.

[0476] Step 607: The encoder performs entropy encoding on the first side information according to the second estimated probability distribution to obtain the second bitstream.

[0477] This process can refer to the aforementioned step 407, and will not be repeated here in the embodiments of this application.

[0478] Step 608: The encoder sends the second bitstream to the decoder.

[0479] Optionally, the encoder can send the second bitstream to the decoder separately, or it can add the second bitstream to the first bitstream before sending it to the decoder. This embodiment does not limit the method of sending the second bitstream. This process can be referred to step 305 above, and will not be repeated here.

[0480] Step 609: The encoder estimates the third estimated probability distribution.

[0481] This process can refer to the aforementioned step 406, and will not be repeated here in the embodiments of this application.

[0482] Step 610: The encoder performs entropy encoding on the second side information according to the third estimated probability distribution to obtain the third bitstream.

[0483] This process can refer to the aforementioned step 407, and will not be repeated here in the embodiments of this application.

[0484] Step 611: The encoder sends the third bitstream to the decoder.

[0485] Optionally, the encoder can send the third bitstream to the decoder separately, or it can add the third bitstream to the first bitstream before sending it to the decoder. This embodiment does not limit the method of sending the third bitstream. This process can be referred to step 305 above, and will not be repeated here.

[0486] Step 612: The decoder obtains the first context information and the second context information.

[0487] The method for obtaining the first context information can refer to the aforementioned step 306, and will not be repeated here in the embodiments of this application.

[0488] The first bitstream is one of the current bitstreams received by the decoder, and its decoded data is the non-first-order decoded data among the multiple data contained in the current bitstream. The second context information can be obtained by inputting at least one of the decoded data and at least one data that meets a preset condition into a masking convolutional network. The masking convolutional network can refer to the aforementioned step 602, and will not be described in detail here.

[0489] It should be noted that when the decoder performs entropy decoding on each bitstream in the received current bitstream, it typically performs entropy decoding on the fourth bitstream first. The decoder can estimate the fourth estimated probability distribution based on pre-set information, or by using a trained learnable model, or by estimating the fourth estimated probability distribution based on the first and / or second side information. Then, entropy decoding is performed on the fourth bitstream based on the fourth estimated probability distribution to obtain the decoded first bit data, which is the first decoded data among multiple data streams.

[0490] The fourth estimated probability distribution obtained by the decoder needs to be consistent with the fourth estimated probability distribution obtained by the encoder. For example, when the encoder estimates the fourth estimated probability distribution based on pre-set information, the decoder estimates the fourth estimated probability distribution based on the same fixed information. When the encoder estimates the fourth estimated probability distribution using a learned model obtained through training, the decoder estimates the fourth estimated probability distribution based on the same learned model, and the estimated fourth estimated probability distributions are identical. When the encoder estimates the fourth estimated probability distribution based on the first side information and the second side information, the decoder estimates the fourth estimated probability distribution based on both the first side information and the second side information.

[0491] Step 613: The decoder estimates the second estimated probability distribution.

[0492] This process can refer to step 409 above, and will not be repeated here in the embodiments of this application. It should be noted that the second estimated probability distribution estimated by the decoder needs to be consistent with the second estimated probability distribution estimated by the encoder.

[0493] Step 614: The decoder performs entropy decoding on the second bitstream according to the second estimated probability distribution to obtain the decoded first side information.

[0494] This process can refer to the aforementioned step 410, and will not be repeated here in the embodiments of this application.

[0495] Step 615: The decoder estimates the third estimated probability distribution.

[0496] This process can refer to step 409 above, and will not be repeated here in the embodiments of this application. It should be noted that the third estimated probability distribution estimated by the decoder needs to be consistent with the third estimated probability distribution estimated by the encoder.

[0497] Step 616: The decoder performs entropy decoding on the third bitstream based on the third estimated probability distribution to obtain the decoded second-side information.

[0498] This process can refer to the aforementioned step 410, and will not be repeated here in the embodiments of this application.

[0499] Step 617: The decoder estimates the first estimated probability distribution based on the first context information, the second context information, the decoded first side information, and the decoded second side information.

[0500] This process can refer to the aforementioned steps 303 and 403, and will not be repeated here in the embodiments of this application.

[0501] Step 618: The decoder performs entropy decoding on the first bitstream according to the first estimated probability distribution to obtain decoded data. The decoded data is the non-first-order decoded data among the multiple data contained in the current data stream.

[0502] The process of entropy decoding of the first bitstream can refer to step 308 above, and will not be repeated here in this embodiment. It should be noted that after each piece of data is decoded, it is added to the decoded data.

[0503] In summary, the entropy encoding / decoding method provided in this application involves the encoder acquiring the data to be encoded, first context information, second context information, first side information, and second side information contained in the current data stream. Then, it estimates a first estimated probability distribution based on the first context information, second context information, first side information, and second side information. The encoder then performs entropy encoding on the data to be encoded according to the first estimated probability distribution to obtain a first bitstream. The encoder then sends the first bitstream to the decoder. The encoder estimates a second and third estimated probability distribution, and performs entropy encoding on the first and second side information according to the second and third estimated probability distributions to obtain a second bitstream and a third bitstream, respectively. The encoder then sends the second and third bitstreams to the decoder. The decoder acquires the first and second context information and estimates... The second and third estimated probability distributions are obtained. Entropy decoding is then performed on the second and third bitstreams based on these distributions to obtain first and second side information. Subsequently, a first estimated probability distribution is estimated based on this first context information, second context information, first side information, and second side information. Entropy decoding is then performed on the first bitstream based on this first estimated probability distribution. Combining the first context information, first side information, second context information, and second side information to estimate the first estimated probability distribution further improves the accuracy of the obtained first estimated probability distribution, reduces the bitrate in the entropy coding process, thereby reducing the entropy coding overhead and the bandwidth occupancy rate when the various data in the current data stream are transmitted to the decoder, and improving the transmission efficiency of the various data in the current data stream.

[0504] The order of the methods provided in this application can be adjusted appropriately, and the steps can be added or removed as needed. For example, steps 603 to 605, 606 to 608, and 609 to 611 can be executed simultaneously, as can steps 612, 613 to 614, and 615 to 616. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and this application does not limit the scope of protection in this regard.

[0505] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0506] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0507] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0508] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0509] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0510] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0511] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0512] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0513] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An entropy coding method, characterized in that, The method includes: Obtain the data to be encoded, which is the non-first-order encoded data among the multiple data contained in the current data stream; Obtain reference information, which includes at least one of first context information and first side information, wherein the first context information is obtained by inputting at least one encoded data into an attention decoding network, and the first side information is obtained by inputting the plurality of data into an attention encoding network; The first estimated probability distribution is obtained based on the reference information; The data to be encoded is entropy encoded according to the first estimated probability distribution to obtain a first bitstream.

2. The method according to claim 1, characterized in that, The reference information specifically includes the first context information and the first edge information; The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the first edge information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

3. The method according to claim 1, characterized in that, The reference information specifically includes the first context information and the second context information. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

4. The method according to claim 1, characterized in that, The reference information specifically includes the first context information, the first edge information, and the second context information. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the first edge information, and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

5. The method according to claim 1, characterized in that, The reference information specifically includes the first context information and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

6. The method according to claim 1, characterized in that, The reference information specifically includes the first context information, the first side information, and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the first edge information, and the second edge information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

7. The method according to claim 1, characterized in that, The reference information specifically includes the first context information, the second context information, and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the second context information, and the second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

8. The method according to claim 1, characterized in that, The reference information specifically includes the first context information, the first side information, the second context information, and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the first edge information, the second context information, and the second edge information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

9. The method according to claim 1, characterized in that, The reference information specifically includes the first side information and the second context information. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first edge information and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

10. The method according to claim 1, characterized in that, The reference information specifically includes the first side information and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The step of estimating the first estimated probability distribution based on the reference information includes: The first side information and the second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

11. The method according to claim 1, characterized in that, The reference information specifically includes the first side information, the second context information, and the second side information. The second side information is obtained by inputting at least one data that meets the preset conditions from the plurality of data into the supercoding network. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one encoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first edge information, the second context information, and the second edge information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

12. The method according to any one of claims 1-2, 4, 6, 8-11, characterized in that, The method further includes: The second estimated probability distribution is obtained; The first side information is entropy encoded according to the second estimated probability distribution to obtain the second bitstream.

13. The method according to any one of claims 5-8 and 10-11, characterized in that, The method further includes: The third estimated probability distribution is obtained; The second side information is entropy encoded according to the third estimated probability distribution to obtain the third bitstream.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: Obtain the first digit encoded data from the plurality of data; The fourth estimated probability distribution is obtained based on pre-set information; The first-bit encoded data is entropy encoded according to the fourth estimated probability distribution to obtain the fourth bitstream.

15. An entropy decoding method, characterized in that, The method includes: Obtain the first bitstream; Obtain reference information, which includes at least one of first context information and decoded first side information, wherein the first context information is obtained by inputting at least one decoded data into the attention decoding network, and the decoded first side information is obtained by entropy decoding of the second bitstream; The first estimated probability distribution is obtained based on the reference information; The first bitstream is entropy decoded according to the first estimated probability distribution to obtain decoded data, wherein the decoded data is the non-first-order decoded data among the multiple data contained in the current data stream.

16. The method according to claim 15, characterized in that, The reference information specifically includes the first context information and the decoded first side information; The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the decoded first side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

17. The method according to claim 15, characterized in that, The reference information specifically includes the first context information and the second context information. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one decoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

18. The method according to claim 15, characterized in that, The reference information specifically includes the first context information, the decoded first side information, and the second context information. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the decoded first side information, and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

19. The method according to claim 15, characterized in that, The reference information specifically includes the first context information and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream; The step of estimating the first estimated probability distribution based on the reference information includes: The first context information and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

20. The method according to claim 15, characterized in that, The reference information specifically includes the first context information, the decoded first-side information, and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the decoded first side information, and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

21. The method according to claim 15, characterized in that, The reference information specifically includes the first context information, the second context information, and the decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one decoded data into a masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the second context information, and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

22. The method according to claim 15, characterized in that, The reference information specifically includes the first context information, the decoded first side information, the second context information, and the decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The first context information, the decoded first side information, the second context information, and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

23. The method according to claim 15, characterized in that, The reference information specifically includes the decoded first side information and the second context information. The second context information is obtained by inputting at least one data that meets the preset conditions from the at least one decoded data into the masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The decoded first side information and the second context information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

24. The method according to claim 15, characterized in that, The reference information specifically includes the decoded first-side information and the decoded second-side information, wherein the decoded second-side information is obtained by entropy decoding of the third bitstream; The step of estimating the first estimated probability distribution based on the reference information includes: The decoded first side information and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

25. The method according to claim 15, characterized in that, The reference information specifically includes the decoded first side information, the second context information, and the decoded second side information. The decoded second side information is obtained by entropy decoding of the third bitstream. The second context information is obtained by inputting at least one data that meets a preset condition from the at least one decoded data into a masking convolutional network. The step of estimating the first estimated probability distribution based on the reference information includes: The decoded first side information, the second context information, and the decoded second side information are input into the probability distribution estimation network to obtain the first estimated probability distribution output by the probability distribution estimation network.

26. The method according to any one of claims 15-16, 18, 20, and 22-25, characterized in that, When the reference information includes the decoded first-side information, obtaining the reference information includes: Obtain the second bitstream; The second estimated probability distribution is obtained; The second bitstream is entropy-decoded according to the second estimated probability distribution to obtain the decoded first-side information.

27. The method according to any one of claims 19-22 and 24-25, characterized in that, When the reference information includes the decoded second-side information, obtaining the reference information further includes: Obtain the third bitstream; The third estimated probability distribution is obtained; The third bitstream is entropy-decoded according to the third estimated probability distribution to obtain the decoded second-side information.

28. The method according to any one of claims 15-27, characterized in that, The method further includes: Obtain the fourth bitstream; The fourth estimated probability distribution is obtained based on pre-set information; The fourth bitstream is entropy decoded according to the fourth estimated probability distribution to obtain the decoded first-order data, which is the first decoded data among the plurality of data.

29. An entropy coding device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-14.

30. An entropy decoding device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 15-28.

31. A computer-readable storage medium, characterized in that, It includes a computer program that, when executed on a computer, causes the computer to perform the method of any one of claims 1-28.

32. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1-28.

Images