Data coding method and related device

Abstract

The application provides a data coding and decoding method and related equipment, and relates to the field of data processing. In the coding method, side information features of a first feature map of current data are extracted and then quantized to obtain a first quantized feature map. Entropy coding is performed based on the first quantized feature map to obtain a first code stream of the current data. A scaling coefficient is obtained based on the first quantized feature map, the second feature map is scaled according to the scaling coefficient and then quantized to obtain a second quantized feature map. The first probability distribution parameter is scaled according to the scaling coefficient to obtain a second probability distribution parameter, and then the second quantized feature map is entropy coded according to the second probability distribution parameter to obtain a second code stream of the current data. The second feature map and the first probability distribution parameter are scaled by using the same scaling coefficient, so that the matching degree of the second probability distribution parameter and the second quantized feature map is higher, and thus the coding accuracy of the second quantized feature map can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, and in particular, to a data encoding and decoding method and related equipment. BACKGROUND

[0002] The purpose of data compression technology is to reduce the redundant information in the data so as to store and transmit the data in a more efficient format. In other words, data compression refers to representing the original data with fewer bits with loss or without loss. The reason why data can be compressed is that there is redundancy in the data. The purpose of data compression is to reduce the number of bits required to represent the data by removing these data redundancies.

[0003] How to improve the compression performance of the data compression algorithm is a hot spot that the technical personnel in the field are studying. SUMMARY

[0004] The present application provides a data encoding and decoding method and related equipment, which can improve the data compression performance.

[0005] In a first aspect, a data encoding method is provided, which is executed by a data encoding device. The method includes the following steps:

[0006] The first feature map of the current data is subjected to side information feature extraction to obtain a side information feature map. The side information feature map is subjected to quantization processing to obtain a first quantized feature map. The first quantized feature map is subjected to entropy encoding to obtain a first code stream of the current data. The second feature map is subjected to scaling processing based on a scaling coefficient to obtain a scaled feature map. The scaled feature map is subjected to quantization processing to obtain a second quantized feature map. The scaling coefficient is obtained based on the first quantized feature map. The first probability distribution parameter is subjected to scaling processing based on the scaling coefficient to obtain a second probability distribution parameter. The second quantized feature map is subjected to entropy encoding based on the second probability distribution parameter to obtain a second code stream of the current data.

[0007] The data includes at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data or text data. In this embodiment, the first feature map is a feature map obtained by feature extraction on the complete current data, and the second feature map is a feature map obtained based on the current data.

[0008] The side information refers to using the existing information Y to assist in encoding the information X, so that the encoding length of the information X is shorter, that is, the redundancy in the information X is reduced, and the information Y is the side information. In this embodiment, the side information refers to the part of information extracted from the first feature map for assisting the encoding and decoding of the first feature map. In addition, the code stream refers to the bit stream generated after the encoding processing.

[0009] In the scheme, on the one hand, entropy coding is performed based on the first quantized feature map to obtain a first code stream of the current data. The first code stream is a bit stream obtained by entropy coding the first quantized feature map. On the other hand, a scaling coefficient can be estimated based on the first quantized feature map. Then, the second feature map can be scaled according to the scaling coefficient to obtain a scaled feature map. The second quantized feature map can be obtained by quantizing the scaled feature map. The second probability distribution parameter is obtained by scaling the first probability distribution parameter according to the scaling coefficient. Finally, entropy coding is performed on the second quantized feature map according to the second probability distribution parameter to obtain a second code stream of the current data. The second code stream is a bit stream obtained by entropy coding the second quantized feature map. The first code stream and the second code stream together serve as the total code stream of the current data. The data coding method in the prior art only has the step of scaling the feature map. In the scheme, the same scaling coefficient is used to scale the second feature map and the first probability distribution parameter, so that the matching degree of the second probability distribution parameter and the second quantized feature map is higher, and the coding accuracy of the second quantized feature map is improved, that is, the data compression performance is improved.

[0010] In some possible embodiments of the first aspect, the second feature map is a residual feature map of the current data, which is obtained according to a third feature map of the current data and the first probability distribution parameter, and the third feature map is obtained by performing feature extraction on the current data.

[0011] In the embodiment, the third feature map is a feature map obtained by performing feature extraction on the complete current data, and the third feature map is different from or the same as the first feature map. According to the third feature map and the first probability distribution parameter, a residual of the third feature map, that is, a residual feature map, can be obtained, and the residual feature map is taken as the second feature map of the current data.

[0012] In the scheme, when encoding, after the residual feature map of the current data is scaled to obtain a scaled feature map, the scaled feature map is quantized, so that the quantization loss of the scaled feature map is smaller, that is, the information loss of the second code stream is smaller, which helps to improve the data quality of the reconstructed data obtained by decoding processing according to the second code stream. On the other hand, compared with the coding network structure in the prior art, since the coding network structure of the embodiment of the present application includes residual feature map scaling, after the training of the entire coding network is completed, the network parameters of the entire coding network can be optimized, including the network for generating the first code stream and the network for generating the second code stream. Therefore, by using the coding network structure of the embodiment of the present application, the data amount of the total code stream of the current data can be reduced, and the coding efficiency can be improved. In summary, by combining residual feature map scaling and first probability distribution parameter scaling, the data compression performance can be further improved.

[0013] In some possible embodiments of the first aspect, the second feature map is a feature map extracted from the current data. The first feature map and the second feature map are the same or different.

[0014] In some possible embodiments of the first aspect, the first probability distribution parameter comprises a mean and / or a variance.

[0015] In some possible embodiments of the first aspect, the first probability distribution parameter is obtained based on the first quantized feature map.

[0016] In some possible embodiments of the first aspect, the first probability distribution parameter is a preset probability distribution parameter.

[0017] In some possible embodiments of the first aspect, the data encoding method further comprises: transmitting the first code stream and the second code stream.

[0018] After the first code stream and the second code stream of the current data are obtained by using the data encoding method of the present solution, the first code stream and the second code stream can be transmitted to other devices as needed, so that the other devices can process the first code stream and the second code stream.

[0019] In some possible embodiments of the first aspect, the first code stream and the second code stream are saved in the form of a code stream file.

[0020] In a second aspect, the present application also provides a data encoding method, which is executed by a data encoding apparatus. The method comprises the following steps:

[0021] performing side information feature extraction on a first feature map of the current data to obtain a side information feature map, and performing quantization processing on the side information feature map to obtain a first quantized feature map; performing entropy encoding on the first quantized feature map to obtain a first code stream of the current data; performing scaling processing on a residual feature map according to a scaling coefficient to obtain a scaled feature map, and performing quantization processing on the scaled feature map to obtain a second quantized feature map, the residual feature map being obtained according to a second feature map of the current data and a first probability distribution parameter, the scaling coefficient being obtained based on the first quantized feature map; and performing entropy encoding on the second quantized feature map according to the first probability distribution parameter to obtain a second code stream of the current data.

[0022] In the scheme, the residual feature map of the current data is scaled to obtain a scaled feature map, and then the scaled feature map is quantized, so that the quantization loss of the scaled feature map is smaller, that is, the information loss of the second code stream is smaller, which helps to improve the data quality of the reconstructed data obtained by decoding the second code stream. On the other hand, compared with the coding network structure in the prior art, since the coding network structure comprising residual feature map scaling is adopted in the embodiment of the application, after the training of the entire coding network is completed, the network parameters of the entire coding network can be optimized, including the network for generating the first code stream and the network for generating the second code stream. Therefore, by using the coding network structure of the embodiment of the application, the data amount of the total code stream of the current data can be reduced, and the coding efficiency can be improved. In summary, by using the coding method of the embodiment, the data compression performance can be further improved.

[0023] In some possible embodiments of the second aspect, the first feature map and the second feature map are the same or different.

[0024] In some possible embodiments of the second aspect, the data comprises at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data, or text data.

[0025] In some possible embodiments of the second aspect, the first probability distribution parameter comprises a mean and / or a variance.

[0026] In some possible embodiments of the second aspect, the first probability distribution parameter is obtained based on the first quantized feature map.

[0027] In some possible embodiments of the second aspect, the first probability distribution parameter is a preset probability distribution parameter.

[0028] In a third aspect, the application further provides a data decoding method, which is executed by a data decoding apparatus. The method comprises the following steps:

[0029] performing entropy decoding on the first code stream of the current data to obtain a third feature map. A scaling coefficient is obtained based on the third feature map. The first probability distribution parameter is scaled based on the scaling coefficient to obtain a second probability distribution parameter. The second code stream of the current data is entropy decoded based on the second probability distribution parameter to obtain a fourth feature map. The fourth feature map is scaled based on the scaling coefficient to obtain a fifth feature map. The reconstructed data of the current data is obtained based on the fifth feature map.

[0030] When data is encoded, both the second feature map and the first probability distribution parameter are scaled. Correspondingly, during data decoding, the same scaling factor is used to process both the first probability distribution parameter and the fourth feature map to ensure the decoding accuracy of the fourth feature map. Furthermore, the fourth feature map is inversely scaled according to the scaling factor to obtain the fifth feature map. Reconstructed data can then be obtained based on the fifth feature map, resulting in higher accuracy and quality. In summary, combining scaling of the first probability distribution parameter and inverse scaling of the fourth feature map improves both the accuracy and quality of data decoding.

[0031] In some possible embodiments of the third aspect, the fourth feature map is a residual feature map, the fifth feature map is a scaled residual feature map, and the reconstructed data of the current data is obtained based on the fifth feature map, including:

[0032] The first probability distribution parameter and the fifth feature map are added together to obtain the sixth feature map. The reconstructed data of the current data is then obtained based on the sixth feature map.

[0033] When scaling and quantizing the residual feature map during data encoding, the data decoding method of this scheme has a smaller data volume in the first and second bitstreams compared to existing technologies, resulting in a smaller decoding processing volume and effectively improving decoding efficiency. Moreover, since the second bitstream has less information loss, the reconstructed data obtained using this scheme has higher data quality.

[0034] In some possible embodiments of the third aspect, the first probability distribution parameters include the mean and / or variance.

[0035] In some possible embodiments of the third aspect, the data includes at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

[0036] In some possible embodiments of the third aspect, the data decoding method further includes: receiving a first bitstream and a second bitstream of the current data.

[0037] In some possible embodiments of the third aspect, the data decoding method further includes: obtaining a first probability distribution parameter based on a third feature map.

[0038] In some possible embodiments of the third aspect, the first probability distribution parameter is a preset probability distribution parameter.

[0039] Fourthly, this application also provides a data decoding method, which is executed by a data decoding device. The method includes the following steps:

[0040] The third feature map is obtained by entropy decoding the first code stream of the current data. The scaling coefficient is obtained based on the third feature map. The fourth feature map is obtained by entropy decoding the second code stream of the current data based on the first probability distribution parameter. The fifth feature map is obtained by scaling the fourth feature map based on the scaling coefficient. The sixth feature map is obtained based on the first probability distribution parameter and the fifth feature map. The reconstructed data of the current data is obtained based on the sixth feature map.

[0041] In the case of performing the scaling and quantization operation on the residual feature map during data encoding, the data decoding method of the present solution has a smaller data amount of the first code stream and the second code stream compared with the prior art, and thus has a smaller decoding processing amount. The present solution can effectively improve the decoding efficiency. Moreover, the information loss of the second code stream is smaller, and thus the data quality of the reconstructed data obtained by the present solution is higher.

[0042] In some possible embodiments of the fourth aspect, the first feature map and the second feature map are the same or different.

[0043] In some possible embodiments of the fourth aspect, the data includes at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data, or text data.

[0044] In some possible embodiments of the fourth aspect, the first probability distribution parameter includes a mean value and / or a variance.

[0045] In some possible embodiments of the fourth aspect, the data decoding method further includes: obtaining the first probability distribution parameter based on the third feature map.

[0046] In some possible embodiments of the fourth aspect, the first probability distribution parameter is a preset probability distribution parameter.

[0047] In the fifth aspect, the present application further provides a data encoder including processing circuitry configured to perform the data encoding method according to any one of the embodiments of the first aspect or the second aspect.

[0048] In the sixth aspect, the present application further provides a computer-readable storage medium storing a code stream, wherein the code stream is generated according to the data encoding method of any one of the embodiments of the first aspect or the second aspect.

[0049] In the seventh aspect, the present application further provides a data decoder including processing circuitry configured to perform the data decoding method according to any one of the embodiments of the third aspect or the fourth aspect.

[0050] In an eighth aspect, the present application also provides a computer program product, comprising program codes for executing the method according to any one of the embodiments of the first aspect, the second aspect, the third aspect or the fourth aspect when the program codes are executed on a computer or a processor.

[0051] In a ninth aspect, the present application also provides a data encoder, comprising:

[0052] one or more processors;

[0053] a computer readable storage medium coupled to the one or more processors, the computer readable storage medium storing a program, wherein the program, when executed by the one or more processors, causes the data encoder to perform the data encoding method according to any one of the embodiments of the first aspect or the second aspect.

[0054] In a tenth aspect, the present application also provides a data decoder, comprising:

[0055] one or more processors;

[0056] a computer readable storage medium coupled to the one or more processors, the computer readable storage medium storing a program, wherein the program, when executed by the one or more processors, causes the data decoder to perform the data decoding method according to any one of the embodiments of the third aspect or the fourth aspect.

[0057] In an eleventh aspect, the present application also provides a computer readable storage medium, comprising program codes for executing the method according to any one of the embodiments of the first aspect, the second aspect, the third aspect or the fourth aspect when the program codes are executed by a computer device.

[0058] In a twelfth aspect, the present application also provides a computer readable storage medium, storing a code stream comprising program codes, which, when executed by one or more processors, causes a decoder to perform the data decoding method according to any one of the embodiments of the third aspect or the fourth aspect.

[0059] In a thirteenth aspect, the present application also provides a chip, comprising a processor and a data interface, the processor reading instructions stored on a memory through the data interface, and executing the method according to any one of the embodiments of the first aspect, the second aspect, the third aspect or the fourth aspect.

[0060] Optionally, as an implementation manner, the chip can further comprise a memory, the memory storing instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to execute the method according to any one of the embodiments of the first aspect, the second aspect, the third aspect or the fourth aspect. BRIEF DESCRIPTION OF DRAWINGS

[0061] The drawings for the embodiments of the present application are described as follows.

[0062] FIG. 1A is a schematic diagram of an architecture of a data coding system provided by an embodiment of the present application;

[0063] FIG. 1B is a block diagram of an example of a data coding system for implementing the embodiments of the present application;

[0064] FIG. 1C is a block diagram of another example of a data coding system for implementing the embodiments of the present application;

[0065] FIG. 2 is a schematic block diagram of a data coding device for implementing the embodiments of the present application;

[0066] FIG. 3 is a schematic block diagram of a data coding apparatus for implementing the embodiments of the present application;

[0067] FIG. 4A is a schematic diagram of a structure of a data encoder provided by an embodiment of the present application;

[0068] FIG. 4B is a schematic diagram of a structure of a data decoder provided by an embodiment of the present application;

[0069] FIG. 4C is a schematic diagram of another structure of a data encoder provided by an embodiment of the present application;

[0070] FIG. 4D is a schematic diagram of another structure of a data encoder provided by an embodiment of the present application;

[0071] FIG. 4E is a schematic diagram of another structure of a data decoder provided by an embodiment of the present application;

[0072] FIG. 4F is a schematic diagram of another structure of a data encoder provided by an embodiment of the present application;

[0073] FIG. 4G is a schematic diagram of another structure of a data decoder provided by an embodiment of the present application;

[0074] FIG. 4H is a schematic diagram of another structure of a data encoder provided by an embodiment of the present application;

[0075] FIG. 4I is a schematic diagram of another structure of a data encoder provided by an embodiment of the present application;

[0076] FIG. 4JFigure 2 is a structural schematic diagram of another data decoder provided by an embodiment of the present application;

[0077] FIG. 4K Figure 3 is a structural schematic diagram of another data encoder provided by an embodiment of the present application;

[0078] FIG. 5A Figure 4 is a structural schematic diagram of an encoding network provided by an embodiment of the present application;

[0079] FIG. 5B Figure 5 is a structural schematic diagram of another encoding network provided by an embodiment of the present application;

[0080] FIG. 5C Figure 6 is a structural schematic diagram of a hyper-prior encoding network provided by an embodiment of the present application;

[0081] FIG. 5D Figure 7 is a structural schematic diagram of a hyper-prior decoding network provided by an embodiment of the present application;

[0082] FIG. 5E Figure 8 is a structural schematic diagram of another hyper-prior decoding network provided by an embodiment of the present application;

[0083] FIG. 5F Figure 9 is a structural schematic diagram of another hyper-prior decoding network provided by an embodiment of the present application;

[0084] FIG. 5G Figure 10 is a structural schematic diagram of a decoding network provided by an embodiment of the present application;

[0085] FIG. 5H Figure 11 is a structural schematic diagram of a nonlinear unit provided by an embodiment of the present application;

[0086] FIG. 6A Figure 12 is a structural schematic diagram of another data coding system provided by an embodiment of the present application;

[0087] FIG. 6B Figure 13 is a structural schematic diagram of another data coding system provided by an embodiment of the present application;

[0088] FIG. 7 Figure 14 is a flow schematic diagram of a data encoding method provided by an embodiment of the present application;

[0089] FIG. 8 Figure 15 is a flow schematic diagram of another data encoding method provided by an embodiment of the present application;

[0090] FIG. 9 Figure 16 is a flow schematic diagram of a data decoding method provided by an embodiment of the present application;

[0091] FIG. 10 Figure 17 is a flow schematic diagram of another data decoding method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0092] The technical solutions in the present application will be described below with reference to the drawings.

[0093] Due to the application involved in the embodiments of the present application, in order to facilitate understanding, the related concepts of the related terms involved in the embodiments of the present application will be introduced first.

[0094] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design proposed as "exemplary" or "for example" in the present application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words such as "exemplary" or "for example" is intended to present concepts in a particular manner. The present application is not limited to the embodiments or designs described in the present application.

[0095] In the embodiments of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "At least one of the following (one)" or the like refers to any combination of these items, including any combination of single item (one) or multiple items. For example, at least one of a, b, or c can mean a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c can be single or multiple. "And / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. The character " / " generally represents a "or" relationship between the associated objects. The sequence numbers (such as step S1, step S21, etc.) of the steps of the embodiments of the present application are only used to distinguish different steps, and do not limit the execution order between the steps.

[0096] In addition, unless otherwise stated, the ordinal numbers "first", "second", etc. used in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the order, time sequence, priority or importance of the plurality of objects. For example, the first device and the second device are only for ease of description, and do not mean that the structures, importance, etc. of the first device and the second device are different. In some embodiments, the first device and the second device can also be the same device.

[0097] In the above embodiments, according to the context, the term "when" can be interpreted as meaning "if", "after", "in response to determining", or "in response to detecting". The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. within the concept and principle of the present application should be included in the protection scope of the present application.

[0098] For the convenience of understanding, the related terms and concepts that may be involved in the embodiments of the present application are introduced first as follows.

[0099] (1) Quantization

[0100] Quantization is used to convert continuous signals into discrete signals. In compression, continuous features are converted into discrete features. In entropy coding, the probability values of the probability distribution are usually converted from continuous values to discrete values.

[0101] (2) Entropy coding

[0102] Entropy coding is a coding process that does not lose any information according to the entropy principle. The information entropy is the average amount of information of the source (a measure of uncertainty). Common entropy coding includes: Shannon coding, Huffman coding, run-length coding, LZW coding and arithmetic coding. Among them, the LZW coding algorithm, also known as "string table compression algorithm", is to realize compression by establishing a string table and using shorter codes to represent longer strings.

[0103] (3) Neural network

[0104] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs and intercept 1 as inputs. The output of the operation unit can be:

[0105]

[0106] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a Sigmoid function. The neural network is a network formed by connecting multiple single neural units, i.e. the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0107] (4) Deep neural network

[0108] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0109] Although DNN looks very complex, it is actually not complex in terms of the work of each layer. In simple terms, it is expressed as the following linear relationship expression: wherein, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also known as coefficient), and a() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Due to the large number of layers in DNN, the number of coefficients W and offset vectors is also relatively large. These parameters in the DNN are defined as follows: taking the coefficient W as an example: assuming in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

[0110] In summary, the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as

[0111] It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and "capacity", which means it can complete more complex learning tasks. Training a deep neural network is a process of learning the weight matrix, and the ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vectors W).

[0112] (5) Convolutional Neural Network

[0113] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The CNN contains a feature extractor composed of convolutional layers and sub-sampling layers, which can be regarded as filters. A convolutional layer refers to a layer of neurons in the CNN that performs convolutional processing on an input signal. In the convolutional layer of the CNN, a neuron can be connected to only part of the adjacent layer neurons. A convolutional layer usually contains several feature planes, each of which can be composed of a number of rectangularly arranged neural units. The neural units in the same feature plane share weights, which are the convolutional kernels. The shared weights can be understood as being independent of the way and position of extracting image information. The convolutional kernels can be initialized in the form of a matrix of random size, and the convolutional kernels can obtain reasonable weights through learning in the training process of the CNN. In addition, the shared weights directly reduce the connections between the layers of the CNN and reduce the risk of overfitting.

[0114] (6) Loss function

[0115] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is actually intended to be predicted, the weight vectors of each layer of the neural network can be updated based on the difference between the predicted value of the current network and the target value that is actually intended to be predicted (of course, there is usually an initialization process before the first update, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the deep neural network can predict the target value that is actually intended to be predicted or a value very close to it. Therefore, it is necessary to define in advance “how to compare the difference between the predicted value and the target value”, which is the loss function or the objective function, which are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (Loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the Loss.

[0116] (7) Backpropagation algorithm

[0117] The neural network can adopt a back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal will produce an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal neural network model parameters, such as weight matrix.

[0118] In the prior art, the compression performance of the data compression algorithm needs to be improved, for example, the data occupies more bits after compression. Therefore, the data encoding and decoding method provided in the embodiments of the present application can effectively improve the data compression performance.

[0119] Exemplarily, the data in the data encoding method and / or the data decoding method in the embodiments of the present application includes at least one of the following: image data, video data, motion vector (MV) data of the video data, audio data, point cloud data or text data. The image data can be one or more than two images. The video data refers to a sequence of continuous images, which is essentially composed of a group of continuous images. Exemplarily, for image data or video data, the data encoding method and / or the data decoding method of the present application can be performed on at least one image in the image data or the video data to realize the encoding and decoding processing of the at least one image. Exemplarily, for the encoding and decoding of video data, the data encoding method and the data decoding method of the present application can be used to process a frame of image A, obtain a reconstructed frame corresponding to the image A, and then use the reconstructed frame to predict the next frame of image of the image A to obtain a predicted image of the next frame of image, and then compress the difference between the next frame of image and the predicted image. The reconstruction result obtained by decoding is the sum of the predicted image of the next frame of image and the reconstruction residual. For video data, the pixel data of each video frame is usually encoded as a block of pixels (also referred to as "pixel block", "coding unit" and "macro block" herein). The motion vector is used to describe the position of the macro block in the video frame relative to the offset vector in the reference frame. The motion vector data is at least one motion vector obtained based on the video data.

[0120] Point Cloud Data refers to a set of vectors in a three-dimensional coordinate system. The scanning data is recorded in the form of points, and each point contains three-dimensional coordinates, and some may contain color information or intensity information. In addition to geometric position, point cloud data also has color information. Color information is usually obtained by a camera to capture color images, and then the color information of the corresponding pixels is assigned to the corresponding points in the point cloud. Intensity information is the echo intensity collected by the receiving device of the laser scanner. This intensity information is related to the surface material, roughness, incident angle direction of the target, and the emission energy and laser wavelength of the instrument. In addition, text data refers to data including text, and the data format of text data can be a TXT file, a PDF file, a Word file, an Excel file, etc. Further, the application examples of the data encoding method and / or the data decoding method of the present application have cloud storage services, cloud monitoring, live streaming, etc.

[0121] FIG. 1A is a schematic diagram of a data coding system architecture provided by an embodiment of the present application, exemplarily, FIG. 1A The data coding system in the present application includes a data encoder and a data decoder. The data encoder includes an encoding unit, an entropy encoding unit, and a saving unit, and the data decoder includes a decoding unit, an entropy decoding unit, and a loading unit.

[0122] The encoding unit is used to transform the to-be-processed data into feature data with lower redundancy, and obtain the probability distribution corresponding to the feature data. The to-be-processed data includes at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

[0123] The entropy encoding unit is used to losslessly encode the feature data according to the probability corresponding to the feature data, and further reduce the data transmission amount in the compression process.

[0124] The saving unit is used to save the data file generated by the entropy encoding unit to the corresponding storage location.

[0125] The loading unit is used to load the data file from the corresponding storage location and input it to the entropy decoding unit.

[0126] The entropy decoding unit is used to entropy-decode the data file to obtain the processed data.

[0127] The decoding unit is used to inversely transform the processed data output by the entropy decoding unit and parse it into reconstructed data.

[0128] Exemplarily, after the data acquisition device acquires the to-be-processed data, the to-be-processed data is compressed, and the specific process is as follows: after the to-be-processed data is processed by the encoding unit, the to-be-encoded feature and the corresponding probability distribution are obtained, the to-be-encoded feature and the probability distribution are input into the entropy encoding unit for processing, and the code stream file is obtained, and the saving unit saves the code stream file. When the code stream file is decompressed, the specific process is as follows: the loading unit loads the code stream file from the saving position, and inputs the code stream file into the entropy decoding unit, and the entropy decoding unit and the decoding unit cooperate to obtain the reconstructed data corresponding to the code stream file. Further, exemplarily, the reconstructed data can be output, for example, display output.

[0129] In the following embodiment of the coding system 10, the data encoder 20 and the data decoder 30 are based on FIG. 1B to FIG. 1C are described.

[0130] FIG. 1B The coding system 10 is a schematic block diagram, for example, a data coding system 10 (or simply referred to as coding system 10) of the present application can be used. The data encoder 20 (or simply referred to as encoder 20) and the data decoder 30 (or simply referred to as decoder 30) in the coding system 10 represent devices and the like that can be used to perform various techniques based on various examples described in the present application.

[0131] As FIG. 1B shown, the coding system 10 includes a source device 12 configured to provide encoded data 21, such as encoded images, encoded video, or encoded audio, to a destination device 14 configured to decode the encoded data 21.

[0132] The source device 12 includes the data encoder 20, and in addition, optionally, a data source 16, a preprocessor (or pre-processing unit) 18, and a communication interface (or communication unit) 22.

[0133] The data source 16 can include or be any type of data acquisition device for acquiring data, and / or any type of data generation device. In this embodiment, the data includes at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

[0134] As an example, when the data is image data, the data source 16 can comprise or be any type of image capturing device for capturing real-world images, etc., and / or any type of image generating device, e.g., a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, Virtual Reality (VR) images and / or any combination thereof (e.g., Augmented Reality (AR) images). The data source can be any type of memory or storage for storing any of the above images.

[0135] As an example, when the data is video data, the data source 16 can comprise or be any type of video recording device for capturing real-world images, etc., to generate a video, and / or any type of video generating device, e.g., a computer graphics processor for generating computer-animated videos or any type of device for acquiring and / or providing real-world videos, computer-generated videos (e.g., screen content).

[0136] As an example, when the data is audio data, the data source 16 can comprise or be any type of audio capturing device for capturing sounds in the real world, etc., to generate audio, and / or any type of audio generating device, e.g., an audio processor for generating virtual audio (e.g., virtual vocals, etc.) or any type of device for acquiring and / or providing real-world audio, computer-generated audio (e.g., screen content).

[0137] As an example, when the data is point cloud data, the data source 16 can comprise or be any type of device for acquiring point cloud data, e.g., a 3D laser scanner or a photographic scanner, etc.

[0138] For the sake of distinguishing the processing performed by the pre-processor (or pre-processing unit) 18, the data 17 can also be referred to as raw data 17.

[0139] The pre-processor 18 is configured to receive the (raw) data 17 and to pre-process the data 17 to obtain pre-processed data 19. As an example, when the data is image data, the pre-processing performed by the pre-processor 18 can comprise cropping, color format conversion (e.g., from RGB to YCbCr), color adjustment or de-noising. It can be appreciated that the pre-processing unit 18 can be an optional component.

[0140] The data encoder 20 is configured to receive the pre-processed data 19 and to provide encoded data 21 (based on the encoding scheme 22, as will be explained in more detail below). FIG. 4A ,FIG. 4C 、 FIG. 4D 、 FIG. 4F 、 FIG. 4G 、 FIG. 4I and the like (further described below).

[0141] The communication interface 22 in the source device 12 can be configured to receive the encoded data 21 and transmit the encoded data 21 (or any other processed version thereof) to another device, such as the destination device 14, or any other device, over the communication channel 13 for storage or direct reconstruction.

[0142] The destination device 14 comprises a data decoder 30, and in addition, optionally, a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0143] The communication interface 28 in the destination device 14 is configured to receive the encoded data 21 (or any other processed version thereof) directly from the source device 12 or from any other source device, such as a storage device, e.g. an encoded data storage device, and provide the encoded data 21 to the data decoder 30.

[0144] The communication interface 22 and the communication interface 28 can be configured to transmit or receive the encoded data 21 over a direct communication link, such as a direct wired or wireless connection, between the source device 12 and the destination device 14, or over any type of network, such as a wired network, a wireless network or any combination thereof, any type of private and public network or any combination thereof.

[0145] For example, the communication interface 22 can be configured to encapsulate the encoded data 21 into a suitable format, such as a packet, and / or to process the encoded data 21 using any type of transmission encoding or processing for transmission over the communication link or the communication network.

[0146] The communication interface 28 corresponds to the communication interface 22 and can be configured to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or de-encapsulate the transmitted data to obtain the encoded data 21.

[0147] The communication interface 22 and the communication interface 28 can each be configured as a unidirectional communication interface, as indicated by the arrow of the corresponding communication channel 13 from the source device 12 to the destination device 14 in FIG. 1B , or as a bidirectional communication interface, and can be configured to transmit and receive messages and the like to establish a connection, confirm and exchange any other information, such as the encoded data, and the like.

[0148] The data decoder 30 is configured to receive the encoded data 21 and provide decoded data (or reconstructed data) 31 (further described below based on FIG. 4B 、 FIG. 4E 、 FIG. 4H and the like).

[0149] The post-processor 32 is used to post-process the decoded data 31 to obtain post-processed data 33. Taking image data as an example, the post-processing performed by the post-processor 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate post-processed data 33 for display by display device 34, etc.

[0150] Display device 34 is used to receive post-processed data 33 to display the data to a user or viewer. Display device 34 can be or includes any type of display for representing reconstructed data, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0151] The decoding system 10 also includes a training engine 25, which is used to train the neural network in the data encoder 20 or the data decoder 30 so that the data encoder 20 can obtain encoded data 21 from the input data 17 or preprocessed data 19, or that the data decoder 30 can obtain decoded data 31 from the input encoded data 21. Optionally, the input data also includes advanced prior information.

[0152] Training data can be stored in a database. FIG. 1B (Not illustrated) In this embodiment, the training engine 25 trains a neural network based on the training data. This neural network is either the neural network in the data encoder 20 or the data decoder 30. It should be noted that this application embodiment does not limit the source of the training data; for example, the training data can be obtained from the cloud or other places for neural network training.

[0153] The neural network trained by training engine 25 can be applied to decoding system 10 and decoding system 40, for example, applied to... FIG. 1B The source device 12 (e.g., data encoder 20) or the destination device 14 (e.g., data decoder 30) shown. For example, the training engine 25 can train the aforementioned neural network in the cloud, and then the decoding system 10 downloads and uses the neural network from the cloud.

[0154] although FIG. 1BThe source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0155] Based on the description, FIG. 1B The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary based on the actual device and application, which is obvious to those skilled in the art.

[0156] Data encoder 20 or data decoder 30 or both can be transmitted via, for example FIG. 1C The processing circuitry shown can be implemented, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Data encoder 20 can be implemented via processing circuitry 43 to include reference... FIG. 1C The various modules discussed in the data encoder 20 and / or any other decoder system or subsystem described herein. The data decoder 30 may be implemented via processing circuitry 43 to include references... FIG. 1C The various modules discussed in the data decoder 30 and / or any other decoder system or subsystem described herein. The processing circuitry 43 can be used to perform the various operations discussed below. FIG. 3 As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of this application. One of the data encoder 20 and data decoder 30 can be integrated into a single device as part of a combined codec (Encoder / Decoder, CODEC), such as... FIG. 1C As shown.

[0157] Source device 12 and destination device 14 can comprise any of a variety of devices including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a server, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast reception device, a broadcast transmission device, etc., and can not use or use any type of operating system. In some cases, source device 12 and destination device 14 can be equipped for wireless communication. Thus, source device 12 and destination device 14 can be wireless communication devices.

[0158] In some cases, FIG. 1B The illustrated coding system 10 is merely exemplary. The techniques provided herein can be applied to coding settings that do not necessarily involve any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local memory, transmitted across a network, etc. An encoding device can encode data and store the data to memory, and / or a decoding device can retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but instead only encode data to memory and / or retrieve and decode data from memory.

[0159] FIG. 1C is a diagram illustrating an example of a coding system 40 including a data encoder 20 and / or a data decoder 30, in accordance with an example embodiment. The coding system 40 can include the data encoder 20, the data decoder 30 (and / or a codec implemented by processing circuitry 43), an antenna 42, one or more memory storage 44, and / or a display device 45. The coding system can also include an imaging device 41, e.g., when the data is image data.

[0160] As FIG. 1C illustrated, the imaging device 41, the antenna 42, the processing circuitry 43, the data encoder 20, the data decoder 30, the memory storage 44, and / or the display device 45 can be in communication with each other. In different examples, the coding system 40 can include only the data encoder 20 or only the data decoder 30. Of course, the coding system 40 is not limited to FIG. 1C the illustrated components, and can include more or less components than those shown. FIG. 1C For example, the coding system 40 can include multiple encoders and / or multiple decoders.

[0161] In some examples, the antenna 42 can be used to transmit or receive an encoded bitstream of data. Additionally, in some examples, the display device 45 can be used to present reconstructed data. The processing circuit 43 can include Application-Specific Integrated Circuit (ASIC) logic, a graphics processor, a general purpose processor, or the like. Additionally, the memory storage 44 can be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or the like) or non-volatile memory (e.g., flash memory, or the like), or the like. In non-limiting examples, the memory storage 44 can be implemented by a cache memory. In other examples, the processing circuit 43 can include memory (e.g., a cache, or the like) for implementing an image buffer, or the like.

[0162] In some examples, the coding system 40 can also include a data decoder 30 coupled to the antenna 42 and used to decode the encoded bitstream of data. The display device 45 is used to present reconstructed data.

[0163] It should be appreciated that the data decoder 30 can be used to perform the inverse process for the examples described with respect to the reference data encoder 20 in the embodiments of the present application.

[0164] FIG. 2 A schematic diagram of a data coding device 200 is provided in the embodiments of the present application. The data coding device 200 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the data coding device 200 can be a decoder, such as the data decoder 30 in FIG. 1C , or can be an encoder, such as the data encoder 20 in FIG. 1C .

[0165] The data coding device 200 includes an ingress port 210 (or input port 210) and a receiver unit (Rx) 220 for receiving data, a processor, logic unit or central processing unit (CPU) 230 for processing data, e.g., the processor 230 can be a neural network processor 230 herein, a transmitter unit (Tx) 240 and an egress port 250 (or output port 250) for transmitting data, and a memory 260 for storing data. Exemplarily, the data coding device 200 can further include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress port 210, the receiver unit 220, the transmitter unit 240 and the egress port 250 for egress or ingress of optical signals or electrical signals.

[0166] The processor 230 is implemented by hardware and software. The processor 230 can be implemented as one or more processor chips, cores (e.g., a multi-core processor), FPGAs, ASICs and DSPs. The processor 230 is in communication with the ingress port 210, the receiver unit 220, the transmitter unit 240, the egress port 250 and the memory 260. The processor 230 includes a coding module 270 (e.g., a neural network NN based coding module 270). The coding module 270 implements the embodiments disclosed above. For example, the coding module 270 performs, processes, prepares or provides various encoding operations. Thus, the coding module 270 provides substantial improvement for the functions of the data coding device 200, and affects the switching of the data coding device 200 to different states. Alternatively, the coding module 270 is implemented in instructions stored in the memory 260 and executed by the processor 230.

[0167] The memory 260 includes one or more disks, tape drives and solid-state drives, can be used as an overflow data storage device, to store programs when such programs are selected for execution, and to store instructions and data read during program execution. The memory 260 can be volatile and / or non-volatile, can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM) and / or static random-access memory (SRAM).

[0168] FIG. 3A simplified block diagram of a data coding device 300 is provided for exemplary embodiments, which can be used as either or both of the source device 12 and the destination device 14 in FIG. 1. FIG. 1B

[0169] The processor 302 in the device 300 can be a central processing unit. Alternatively, the processor 302 can be any other type of device or multiple devices capable of manipulating or processing information now-existing or hereafter developed. Although a single processor 302 as shown in FIG. 3 can be used to implement the disclosed implementations, the use of more than one processor can be faster and more efficient. FIG. 3

[0170] In one implementation, the memory 304 in the device 300 can be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of memory device can be used as the memory 304. The memory 304 can include code and data 306 that are accessed by the processor 302 via the bus 312. The memory 304 can also include an operating system 308 and application programs 310, including at least one program for allowing the processor 302 to perform the methods described herein. For example, the application programs 310 can include applications 1 through N, and also include a data coding application, i.e., a data encoding and / or a data decoding, that performs the methods described herein.

[0171] The device 300 can also include one or more output devices, such as a display 314. In one example, the display 314 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 314 can be coupled to the processor 302 via the bus 312.

[0172] Although the bus 312 in the device 300 is described herein as a single bus, the bus 312 can include multiple buses. Also, auxiliary storage can be directly coupled to other components of the device 300 or accessed via a network, and can include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, the device 300 can have a wide variety of configurations.

[0173] FIG. 4A FIG. 2 is a structural schematic diagram of a data encoder according to an embodiment of the present application. In the following description, the data encoder is used as the source device 12 in FIG. 1. FIG. 4A ​​In the example, the data encoder 20 comprises an input end (or an input interface) 401, an encoding network (Encoder) 402, a hyper-prior encoding network (HyperEncoder) 403, a quantization unit 404, a quantization unit 405, an entropy encoding unit 406, an entropy encoding unit 407, an entropy estimation unit (Entropy) 408, a hyper-prior entropy estimation unit (HyperEntropy) 409, and a hyper-prior decoding network (HyperDecoder) 410. The quantization unit 404 and the quantization unit 405 can be the same quantization unit, or they are two independent quantization units. Similarly, the entropy encoding unit 406 and the entropy encoding unit 407 can be the same entropy encoding unit, or they are two independent entropy encoding units. The entropy estimation unit 408 is also referred to as an entropy parameter model unit, and the hyper-prior entropy estimation unit 409 is an entropy parameter model unit using a preset distribution.

[0174] The data encoder 20 can receive data to be encoded by the input end 401 and the like, wherein the data to be encoded comprises at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data, or text data.

[0175] The encoding network 402 is configured to extract features from the data to be encoded to obtain a first feature map y1 and a second feature map y2, wherein the first feature map y1 and the second feature map y2 are different, and the method for obtaining the first feature map y1 and the second feature map y2 is not particularly limited.

[0176] Taking the data as image data as an example, the first feature map and the second feature map output by the encoding network 402 can have a change in size relative to the original image, and the redundant information is removed to facilitate entropy encoding.

[0177] The hyper-prior encoding network 403 is configured to further extract summary information from the first feature map y1 to obtain a side information feature map z. Illustratively, the side information feature map z has a smaller size than the first feature map y1.

[0178] The quantization unit 404 is configured to quantize the side information feature map z to obtain integer type feature data, i.e., a first quantized feature map

[0179] The hyper-prior decoding network 410 is configured to obtain a first decoded feature map The scaling coefficient Δ and the first probability distribution parameter X are generated. The first probability distribution parameter X is used to represent the probability distribution of the second quantized feature map, and X includes but is not limited to the mean μ and / or the variance σ. The first probability distribution parameter X is a matrix with the same size as the second quantized feature map, and each element in the matrix represents the probability distribution parameter of each element in the second quantized feature map, and the probability distribution parameter of each element includes but is not limited to the mean and / or the variance, that is, one second quantized feature map corresponds to one mean matrix and / or one variance matrix.

[0180] In addition, the hyper-prior decoding network 410 can be used only to obtain the first quantized feature map y1 based on the first feature map x1. The scaling coefficient Δ is generated. The first probability distribution parameter X is a preset probability distribution parameter according to actual conditions, which is not particularly limited.

[0181] Based on the second feature map y2 and the first probability distribution parameter X, the residual of the second feature map, that is, the residual feature map C, can be obtained. Specifically, y2 minus X can obtain the residual feature map C. For example, the second feature map y2 minus the mean μ can obtain the residual feature map C.

[0182] The residual feature map C is scaled according to the scaling coefficient Δ to obtain the scaled residual feature map C. Specifically, the residual feature map C divided by the scaling coefficient Δ can obtain the scaled residual feature map C.

[0183] The quantization unit 405 is configured to quantize the scaled residual feature map C to obtain the integer type feature data, that is, the second quantized feature map.

[0184] The hyper-prior entropy estimation unit 409 is configured to estimate the probability distribution of the first quantized feature map y1 according to a preset distribution. The preset distribution can be a Gaussian distribution, a binomial distribution, a uniform distribution or a Laplace distribution, which is not particularly limited and can be set according to actual conditions.

[0185] The entropy coding unit 406 is configured to encode the first quantized feature map y1 into the first code stream A according to the probability distribution of the first quantized feature map y1 estimated by the hyper-prior entropy estimation unit 409.

[0186] The entropy estimation unit 408 is configured to obtain the probability distribution of the second quantized feature map according to the first probability distribution parameter X.

[0187] The entropy coding unit 407 is configured to entropy encode the second quantized feature map by using the probability estimated by the entropy estimation unit 408 to obtain the second code stream B. The first code stream A and the second code stream B are the total code streams of the data to be encoded. The data encoder can output the total code streams through an output end (or an output interface) (​​​FIG. 4A The total code stream of data is outputted (not shown in the figure).

[0188] FIG. 4B FIG. 1 is a structural schematic diagram of a data decoder provided by an embodiment of the present application. FIG. 4B The schematic data decoder is used for decoding FIG. 4A The schematic data decoder is used for decoding

[0189] The data decoder 30 can acquire the total code stream to be decoded (i.e., the first code stream A and the second code stream B) through an input end (or an input interface) (not shown in the figure). FIG. 4B

[0190] The hyper-prior entropy estimation unit 409 is used for estimating the probability distribution of the first quantized feature map Y according to a preset distribution.

[0191] The entropy decoding unit 413 is used for entropy decoding the first code stream A according to the probability estimated by the hyper-prior entropy estimation unit 409, to obtain a third feature map. The entropy decoding unit 413 uses the same distribution as the entropy encoding unit 406 for entropy decoding.

[0192] The hyper-prior decoding network 410 is used for generating the scaling coefficient Δ and the first probability distribution parameter X based on the third feature map. The first probability distribution parameter X is used for representing the probability distribution of the fourth feature map, and X includes but is not limited to the mean μ and / or the variance σ. The first probability distribution parameter X is a matrix with the same scale as the fourth feature map, each element in the matrix represents the probability distribution parameter of each element in the fourth feature map, and each element's probability distribution parameter includes but is not limited to the mean and / or the variance, i.e., one fourth feature map corresponds to one mean matrix and / or one variance matrix.

[0193] In addition, the hyper-prior decoding network 410 can be used only for generating the scaling coefficient Δ based on the third feature map, and the first probability distribution parameter X is a preset probability distribution parameter according to actual conditions, which is not particularly limited.

[0194] The entropy estimation unit 408 is used for obtaining the probability distribution of the fourth feature map according to the first probability distribution parameter X.

[0195] The entropy decoding unit 414 is used for entropy decoding the second code stream B according to the probability estimated by the entropy estimation unit 408, to obtain the fourth feature map.

[0196] ​​The fifth feature map can be obtained according to the scaling coefficient Δ and the fourth feature map. That is, the second quantized feature map and the scaling coefficient Δ are multiplied to obtain the fifth feature map.

[0197] The sixth feature map can be obtained according to the fifth feature map and the first probability distribution parameter X. That is, the fifth feature map and the first probability distribution parameter X are added to obtain the sixth feature map. Exemplarily, when the fifth feature map corresponds to the first probability distribution parameter X, the fifth feature map plus the mean μ can obtain the sixth feature map. FIG. 4A

[0198] The decoding network 412 is configured to reflect map the sixth feature map into the reconstructed data.

[0199] The output end 411 is configured to output the reconstructed data.

[0200] FIG. 4C FIG. 20 is a structural schematic diagram of another data encoder provided by an embodiment of the present application. FIG. 4C The data encoder shown in FIG. 20 is the same as the data encoder shown in FIG. 1, except that the encoding network 402 is configured to perform feature extraction on the data to be processed to obtain the feature map y (that is, the first feature map y1 and the second feature map y2 are the same at this time). FIG. 4A The data encoder shown in FIG. 20 is the same as the data encoder shown in FIG. 1, except that the encoding network 402 is configured to perform feature extraction on the data to be processed to obtain the feature map y (that is, the first feature map y1 and the second feature map y2 are the same at this time). FIG. 4C The data decoder configured to perform decoding processing on the total code stream of the processed data is the same in structure as the data decoder shown in FIG. 1. FIG. 4B The data decoder configured to perform decoding processing on the total code stream of the processed data is the same in structure as the data decoder shown in FIG. 1.

[0201] FIG. 4D FIG. 20 is a structural schematic diagram of another data encoder provided by an embodiment of the present application. FIG. 4D In the example of FIG. 20, the data encoder 20 includes an input end (or input interface) 401, an encoding network (Encoder) 402, a hyper-prior encoding network (HyperEncoder) 403, a quantization unit 404, a quantization unit 405, an entropy encoding unit 406, an entropy encoding unit 407, a hyper-entropy estimation unit (HyperEntropy) 409, a hyper-prior decoding network (HyperDecoder) 410, and a hyper-prior variance decoding network (Hyper Scale Decoder) 415. The quantization unit 404 and the quantization unit 405 can be the same quantization unit, or they are two independent quantization units. Similarly, the entropy encoding unit 406 and the entropy encoding unit 407 can be the same entropy encoding unit, or they are two independent entropy encoding units. The hyper-entropy estimation unit 409 is an entropy parameter model unit adopting a preset distribution.

[0202] The data encoder 20 can receive data to be encoded and processed through the input end 401, and the data to be encoded and processed includes at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data, or text data.​

[0203] The encoding network 402 is configured to extract features from the data to be encoded to obtain a feature map y.

[0204] The hyper-prior encoding network 403 is configured to further extract summary information from the feature map y to obtain an edge information feature map z. Exemplarily, the edge information feature map z has a smaller size than the feature map y.

[0205] The quantization unit 404 is configured to quantize the edge information feature map z to obtain integer type feature data, i.e., a first quantized feature map

[0206] The hyper-prior entropy estimation unit 409 estimates a probability distribution of the first quantized feature map according to a preset distribution. The preset distribution can be a Gaussian distribution, a binomial distribution, a uniform distribution, or a Laplace distribution, and is not particularly limited and can be set according to actual conditions.

[0207] The entropy encoding unit 406 encodes the first quantized feature map into a first code stream A according to the probability distribution of the first quantized feature map estimated by the hyper-prior entropy estimation unit 409.

[0208] The hyper-prior decoding network 410 is configured to generate a scaling coefficient Δ and a mean value μ based on the first quantized feature map , and the mean value μ is used to represent a probability distribution of a second quantized feature map.

[0209] Based on the feature map y and the mean value μ, a residual feature map of the feature map y can be obtained. Specifically, the residual feature map can be obtained by subtracting the mean value μ from the feature map y.

[0210] The residual feature map is scaled according to the scaling coefficient Δ to obtain a scaled residual feature map. Specifically, the scaled residual feature map can be obtained by dividing the residual feature map by the scaling coefficient Δ.

[0211] The quantization unit 405 is configured to quantize the scaled residual feature map to obtain integer type feature data, i.e., a second quantized feature map.

[0212] The hyper-prior variance decoding network 415 performs feature transformation on the first quantized feature map to obtain a variance σ used to represent the probability distribution of the second quantized feature map.

[0213] The entropy encoding unit 407 is configured to perform entropy encoding on the second quantized feature map using the variance σ to obtain a second code stream B. The first code stream A and the second code stream B are used as a total code stream of the data to be encoded. The data encoder can output the total code stream through an output end (or an output interface)FIG. 4D The total code stream of data is outputted (not shown in the figure).

[0214] FIG. 4E FIG. 3 is a structural schematic diagram of another data decoder provided by the embodiment of the present application. FIG. 4E The schematic data decoder is used for decoding the total code stream of data. FIG. 4D The schematic data decoder is used for decoding the total code stream of data.

[0215] The data decoder 30 can acquire the total code stream to be decoded (i.e. the first code stream A and the second code stream B) through an input end (or an input interface) (not shown in the figure). FIG. 4E The data decoder 30 can acquire the total code stream to be decoded (i.e. the first code stream A and the second code stream B) through an input end (or an input interface) (not shown in the figure).

[0216] The hyper-prior entropy estimation unit 409 is used for estimating the probability distribution of the first quantized feature map X according to a preset distribution. The hyper-prior entropy estimation unit 409 is used for estimating the probability distribution of the first quantized feature map X according to a preset distribution.

[0217] The entropy decoding unit 413 is used for entropy decoding the first code stream A according to the probability estimated by the hyper-prior entropy estimation unit 409, so as to obtain a third feature map. The entropy decoding unit 413 uses the same distribution as the entropy encoding unit 406 for entropy decoding.

[0218] The hyper-prior decoding network 410 is used for generating a scaling coefficient Δ and a mean value μ based on the third feature map. The mean value μ is used for representing the probability distribution of a fourth feature map. The mean value μ is a matrix with the same scale as the fourth feature map, and each element in the matrix represents the mean value μ of each element in the fourth feature map, i.e. one fourth feature map corresponds to one mean value matrix.

[0219] The hyper-prior variance decoding network 415 is used for performing feature transformation based on the third feature map, so as to obtain a variance σ used for representing the probability distribution of each image feature data in the fourth feature map.

[0220] The entropy decoding unit 414 is used for entropy decoding the second code stream B according to the variance σ, so as to obtain the fourth feature map.

[0221] The fifth feature map can be obtained according to the scaling coefficient Δ and the fourth feature map. That is, the fifth feature map can be obtained by multiplying the fourth feature map and the scaling coefficient Δ.

[0222] The sixth feature map can be obtained according to the fifth feature map and the mean value μ. That is, the sixth feature map can be obtained by adding the fifth feature map and the mean value μ. The decoding network 412 is used for reflecting the sixth feature map into reconstructed data.

[0223] The output terminal 411 is configured to output the reconstructed data.

[0224] FIG. 4F is a structural schematic diagram of another data encoder provided by an embodiment of the present application. FIG. 4F The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that FIG. 4A The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that FIG. 4F The data encoder not only has the operation of scaling and quantizing the residual feature map, but also has the operation of scaling the probability distribution parameter. Specifically, after the hyper-prior decoding network 410 obtains the first probability distribution parameter X and the scaling coefficient Δ, the first probability distribution parameter X is scaled according to the scaling coefficient Δ, that is, the first probability distribution parameter X is divided by the scaling coefficient Δ to obtain the second probability distribution parameter (X / Δ). The entropy encoding unit 407 is configured to perform entropy encoding on the second quantized feature map according to the second probability distribution parameter (X / Δ) in cooperation with the entropy estimation unit 408, to obtain the second code stream B.

[0225] FIG. 4G is a structural schematic diagram of another data decoder provided by an embodiment of the present application. FIG. 4G The data decoder shown in the structural schematic diagram is configured to decode FIG. 4F The data decoder shown in the structural schematic diagram is configured to decode FIG. 4G The data decoder shown in the structural schematic diagram is different from the data decoder shown in the structural schematic diagram in that FIG. 4B The data decoder shown in the structural schematic diagram is different from the data decoder shown in the structural schematic diagram in that, after the hyper-prior decoding network 410 obtains the first probability distribution parameter X and the scaling coefficient Δ, the first probability distribution parameter X is scaled according to the scaling coefficient Δ, that is, the first probability distribution parameter X is divided by the scaling coefficient Δ to obtain the second probability distribution parameter (X / Δ). The entropy decoding unit 414 is configured to perform entropy decoding on the second code stream B according to the second probability distribution parameter (X / Δ) in cooperation with the entropy estimation unit 408, to obtain the fourth feature map.

[0226] FIG. 4H is a structural schematic diagram of another data encoder provided by an embodiment of the present application. FIG. 4H The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that FIG. 4F The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that, at this time, the first feature map y1 and the second feature map y2 are the same, that is, the encoding network 402 performs feature extraction on the data to be processed to obtain a feature map y. At this time, the data decoder configured to decode FIG. 4F The data decoder configured to decode the total code stream of the data processed by the data encoder has the same structure as the data decoder FIG. 4G

[0227] FIG. 4I is a structural schematic diagram of another data encoder provided by an embodiment of the present application. The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that FIG. 4F The data encoder shown in the structural schematic diagram is different from the data encoder shown in the structural schematic diagram in that FIG. 4I ​The illustrated data encoder does not generate a residual feature map and does not have an operation of scaling the residual feature map, FIG. 4I In the embodiment, after the encoding network 402 obtains the second feature map y2 of the data to be processed, the second feature map y2 is scaled according to the scaling coefficient Δ, that is, the second feature map y2 is divided by the scaling coefficient Δ to obtain a scaled feature map, and the quantization unit 405 quantizes the scaled feature map to obtain the second quantized feature map. Reference FIG. 4J , FIG. 4J FIG. 3 is a structural schematic diagram of another data decoder provided by an embodiment of the present application; FIG. 4J The illustrated data decoder is used for decoding the total code stream of the data processed by the data encoder. FIG. 4I The illustrated data encoder processes the data to obtain a total code stream. The data decoder 30 includes an entropy estimation unit 408, an entropy decoding unit 414, a hyper-prior entropy estimation unit 409, an entropy decoding unit 413, a hyper-prior decoding network 410, a decoding network 412, and an output end (or output interface) 411.

[0228] The data decoder 30 can obtain the total code stream to be decoded (that is, the first code stream A and the second code stream B) through an input end (or input interface) (not shown in the figure). FIG. 4J The data decoder 30 can obtain the total code stream to be decoded (that is, the first code stream A and the second code stream B) through an input end (or input interface) (not shown in the figure).

[0229] The entropy decoding unit 413 is configured to perform entropy decoding on the first code stream A in cooperation with the hyper-prior entropy estimation unit 409 to obtain a third feature map.

[0230] The hyper-prior decoding network 410 is configured to generate the scaling coefficient Δ and a first probability distribution parameter X based on the third feature map. The first probability distribution parameter X is used to represent the probability distribution of the fourth feature map, and X includes but is not limited to the mean μ and / or the variance σ.

[0231] In addition, the hyper-prior decoding network 410 can be used to generate only the scaling coefficient Δ based on the third feature map. The first probability distribution parameter X is a probability distribution parameter preset according to actual conditions, and is not particularly limited.

[0232] The first probability distribution parameter X is scaled according to the scaling coefficient Δ, that is, the first probability distribution parameter X is divided by the scaling coefficient Δ to obtain a second probability distribution parameter (X / Δ).

[0233] The entropy decoding unit 414 is configured to perform entropy decoding on the second code stream B according to the second probability distribution parameter (X / Δ) in cooperation with the entropy estimation unit 408 to obtain a fourth feature map.

[0234] The fourth feature map and the scaling coefficient Δ are multiplied to obtain a fifth feature map based on the scaling coefficient Δ and the fourth feature map.

[0235] The decoding network 412 is configured to reflect map the fifth feature map into the reconstructed data.

[0236] The output end 411 is configured to output the reconstructed data.

[0237] FIG. 4K FIG. 4 is a structural schematic diagram of another data encoder provided by an embodiment of the present application. FIG. 4K The data encoder shown is similar to the data encoder shown in FIG. 3, except that FIG. 4I the first feature map y1 and the second feature map y2 are the same in the data encoder, i.e., the feature map y obtained after the encoding network 402 performs feature extraction on the data to be processed. FIG. 4K The data encoder shown is similar to the data encoder shown in FIG. 3, except that FIG. 4K The data decoder shown in FIG. 4 is configured to decode the total code stream of the data processed by the data encoder. FIG. 4J The data decoder shown in FIG. 4 is configured to decode the total code stream of the data processed by the data encoder.

[0238] It is explained that at least one of the encoding network, the hyper-prior encoding network, the quantization unit, the entropy encoding unit, the entropy decoding unit, the hyper-prior entropy estimation unit, the entropy estimation unit, the hyper-prior decoding network, and the decoding network can be implemented by a neural network, such as a convolutional neural network.

[0239] Exemplarily, for the data encoder shown in FIG. 3, FIG. 4C , FIG. 4H or FIG. 4K the specific structure of the encoding network 402 can refer to the structure shown in FIG. 5. FIG. 5A The encoding network 402 shown in FIG. 5 includes a first convolution (Conv) layer, a first nonlinear unit (ResAU) layer, a second convolution layer, a second nonlinear unit (ResAU) layer, a third convolution layer, a third nonlinear unit (ResAU) layer, and a fourth convolution layer. Exemplarily, the specific parameters of the first convolution layer, the second convolution layer, the third convolution layer, and / or the fourth convolution layer are 192x5x5 / 2↓, where the number of channels is 192, the size of the convolution kernel is 5x5, and the step is 2. FIG. 5A The encoding network shown in FIG. 5 can obtain the feature map y of the data to be processed.

[0240] Exemplarily, for the data encoder shown in FIG. 3, FIG. 4A , FIG. 4F or FIG. 4I the specific structure of the encoding network 402 can refer to the structure shown in FIG. 6. FIG. 5B The encoding network 402 shown in FIG. 6 includes a first encoder, a second encoder, and a third encoder, where the first encoder is configured to perform first feature extraction on the data to be processed, and the second encoder and the third encoder are respectively configured to perform feature extraction on the feature map extracted by the first encoder to obtain the first feature map y1 and the second feature map y2, respectively.

[0241] Further exemplarily, for the data encoder shown in any one of FIG. 4A , FIG. 4F or FIG. 4I , the specific structure of the encoding network 402 can be that the feature map y is obtained by feature extraction using the encoding network shown in FIG. 5A , and then the channel separation is performed on the feature map y to obtain the first feature map y1 and the second feature map y2. The specific method of the channel separation is not limited. For example, FIG. 5A the feature map y obtained has 384 channels, and a (a is less than 384) channels in the feature map y can be taken as the first feature map y1, and the remaining (384-a) channels in the feature map y can be taken as the second feature map y2. For example, the first 192 channels of the feature map y can be taken as the first feature map y1, and the last 192 channels of the feature map y can be taken as the second feature map y2.

[0242] Exemplarily, for the data encoder shown in any one of FIG. 4A , FIG. 4C , FIG. 4F , FIG. 4H , FIG. 4I or FIG. 4K , the specific structure of the hyper-prior encoding network can refer to the structure shown in FIG. 5C . The hyper-prior encoding network includes a first LeakyRelu layer, a first convolutional layer, a second LeakyRelu layer, a second convolutional layer, a third LeakyRelu layer and a third convolutional layer. Exemplarily, the specific parameters of the first convolutional layer are 192x3x3, where the number of channels is 192, and the size of the convolution kernel is 3x3. The specific parameters of the second convolutional layer and / or the third convolutional layer are 192x5x5 / 2↓, where the number of channels is 192, the size of the convolution kernel is 5x5, and the step is 2.

[0243] Exemplarily, for any one of FIG. 4A to FIG. 4K , the specific structure of the hyper-prior decoding network can refer to the structure shown in FIG. 5D . The hyper-prior decoding network includes a first convolutional layer, a first LeakyRelu layer, a second convolutional layer, a second LeakyRelu layer and a third convolutional layer. Exemplarily, the specific parameters of the first convolutional layer are 384x3x3, where the number of channels is 384, and the size of the convolution kernel is 3x3. The specific parameters of the second convolutional layer are 288x5x5 / 2↑, where the number of channels is 288, the size of the convolution kernel is 5x5, and the step is 2. The specific parameters of the third convolutional layer are 192x5x5 / 2↑, where the number of channels is 192, the size of the convolution kernel is 5x5, and the step is 2.

[0244] Further exemplarily, for the data encoder shown in FIG. 4D orFIG. 4E The specific structure of the hyper-prior decoding network in the data decoder can refer to the structure shown in the following table. FIG. 5E In the data encoder, the input of the hyper-prior decoding network is the first quantized feature map. In the data decoder, the input of the hyper-prior decoding network is the third feature map. The hyper-prior decoding network includes a first convolutional layer, a first up-sampling convolutional layer, a first Crop layer, a first Leaky Relu layer, a second convolutional layer, a second up-sampling convolutional layer, a second Crop layer, a second Leaky Relu layer, a third convolutional layer, a fourth convolutional layer, and a third Leaky Relu layer.

[0245] For example, the first convolutional layer corresponds to “Conv128x1x1S1” in the following table, and the specific parameters are as follows: the convolution kernel size is 1x1, the output channel number is 128, and S1 represents 1 times up-sampling. FIG. 5E

[0246] The up-sampling convolutional layer corresponds to “DConv128x4x4S2” in the following table, and the specific parameters are as follows: the convolution kernel size is 4x4, the output channel number is 128, and S2 represents 2 times up-sampling. FIG. 5E

[0247] The second convolutional layer corresponds to “Conv128x3x3S1” in the following table, and the specific parameters are as follows: the convolution kernel size is 3x3, the output channel number is 128, and S1 represents 1 times up-sampling. FIG. 5E

[0248] The second up-sampling convolutional layer corresponds to “DConv128x4x4S2” in the following table, and the specific parameters are as follows: the convolution kernel size is 4x4, the output channel number is 128, and S2 represents 2 times up-sampling. FIG. 5E

[0249] The third convolutional layer corresponds to “Conv128x1x1S1” in the following table, which outputs the scaling coefficient Δ of the third Leaky Relu layer, and the specific parameters are as follows: the convolution kernel size is 1x1, the output channel number is 128, and S1 represents 1 times up-sampling. FIG. 5E

[0250] The third Leaky Relu layer outputs the mean μ, and the fourth convolutional layer corresponds to “Conv128x1x1S1” in the following table, which is connected to the third Leaky Relu layer, and the specific parameters are as follows: the convolution kernel size is 1x1, the output channel number is 128, and S1 represents 1 times up-sampling. FIG. 5E

[0251] The first Leaky Relu layer, the second Leaky Relu layer, and the third Leaky Relu layer can be replaced by a Relu layer.

[0252] ​​​​​​For example, for FIG. 4D The data encoder shown or FIG. 4E For the data decoder shown, the specific structure of the advanced prior decoding network can be found in [reference needed]. FIG. 5F The schematic structure is shown. In the data encoder, the input to the hyperprior decoding network is the first quantized feature map. In the data decoder, the input to the super-prior decoding network is the third feature map. The super-prior decoding network includes a first convolutional module, a second convolutional module, a third convolutional module, a first linear module, a second linear module, a grouped convolutional module, a matrix processing module, a pixel shuffle module, and a decomposition module.

[0253] For example, the first convolutional module, the second convolutional module, and the third convolutional module may each include one or more convolutional layers. The number of convolutional layers in the first convolutional module, the second convolutional module, and the third convolutional module may be the same or different. The structures of the convolutional layers in the first convolutional module, the second convolutional module, and the third convolutional module are the same. Taking the first convolutional module as an example, the size of the convolutional kernel in the first convolutional module is k×k, where k is an integer. The number of input channels of the convolutional layer is C1, the number of output channels of the convolutional layer is C2, the stride of the convolutional layer is 1, and the grouping of the convolutional layer is 1.

[0254] The first and second linear modules are linear units, which can be implemented using Leaky ReLU or ReLU, for example.

[0255] The grouped convolutional module may include one or more convolutional layers. The kernel size of each convolutional layer in the grouped convolutional module is k×k, where k is an integer. The number of input channels is C1, the number of output channels is C2, the stride is 1, and the number of groups is C, where C is an integer. C1 and C2 in the grouped convolutional module may be the same as or different from C1 and C2 in the first convolutional module described above.

[0256] The matrix processing module is used to process matrices. For example, the matrix processing module can be a matrix multiplication module or a matrix addition module.

[0257] The pixel recombination module is used to reconstruct low-resolution feature maps through multi-channel recombination to obtain high-resolution feature maps. For example, if the size of the input feature map is [k... 2If the size of the input feature map is [C1, h, w], the size of the output feature map is [C1+C2, kh, kw]. k is the multiple of the high resolution size relative to the low resolution size, C1 is the number of channels of the second code stream B, and C2 is the number of channels of the scaling coefficient Δ. C1 and C2 here can be different, that is, only part of the channel features and probabilities are scaled. h is the height of the feature map, and w is the width of the feature map.

[0258] The decomposition module is configured to decompose the output of the pixel reorganization module to obtain the mean μ and the scaling coefficient Δ. For example, for the data decoder shown in FIG. 4B , FIG. 4G or FIG. 4J , the specific structure of the decoding network can refer to the structure shown in FIG. 5G . The decoding network includes a first convolutional layer, a first ResAU layer, a second convolutional layer, a second ResAU layer, a third convolutional layer, a third ResAU layer, and a fourth convolutional layer. For example, the specific parameters of the first convolutional layer, the second convolutional layer, and / or the third convolutional layer are 192x5x5 / 2↑, where the number of channels is 192, the size of the convolution kernel is 5x5, and the step is 2. The specific parameters of the fourth convolutional layer are 384x5x5 / 2↑, where the number of channels is 384, the size of the convolution kernel is 5x5, and the step is 2.

[0259] For example, for the specific structure of the ResAU layer in the structure shown in FIG. 5A and / or FIG. 5G , the specific structure of the ResAU layer can refer to FIG. 5H , where the ResAU layer includes a Leaky Relu layer, a convolutional layer, and a Tanh layer. In addition, for any embodiment in the present application, the number of channels C1 of the mean μ and the number of channels C2 of the scaling coefficient Δ can be different or the same, and C2≤C1. The number of channels of the mean μ is the same as the number of channels of the second code stream B. When C2 is equal to C1, there is one scaling coefficient for each channel, and the features and probabilities of each channel are scaled. When C2 is less than C1, the scaling coefficient of the channel without the scaling coefficient is 1, that is, the features and probabilities of the channel are not scaled.

[0260] The following takes the data decoding system composed of FIG. 4A , FIG. 4B as an example (hereinafter referred to as the first data decoding system) to compare and illustrate the data decoding system shown in FIG. 6A (hereinafter referred to as the second data decoding system). The first data decoding system has a residual feature map scaling operation, and the second data decoding system has a feature map scaling operation.

[0261] Data is taken as an example of image data, and the Kodak test set is taken as data to be processed, which contains 24 PNG images with a resolution of 768x512 or 512x768. Each test image is processed by using the first data coding system and the second data coding system respectively, and BPP (Bits per pixel), PSNR (Peak Signal to Noise Ratio) and BD Rate (Delta bit Rate) are obtained (see Table 1 for details). The BPP and PSNR in Table 1 are the average values of the 24 images. The BPP represents the average number of bits used for one pixel, and the smaller the value, the smaller the compression rate. The PSNR is an objective standard for evaluating image quality, and the higher the value, the better the image quality. The BD Rate represents the compression rate saving (or performance improvement) of the two compared methods under the same image quality.

[0262] From Table 1, it can be seen that the results of the first data coding system are better than those of the second data coding system. The BD Rate of the first data coding system is better, and the performance is improved by 3.87%. It can be seen that scaling and quantizing the residual feature map can improve the compression performance.

[0263] Table 1 Performance comparison of the first data coding system and the second data coding system

[0264]

[0265] Next, the data coding system (hereinafter referred to as the third data coding system) is taken as an example, and the data coding system is compared with the second data coding system shown in FIG. 6B . The second data coding system is a feature map scaling operation, while the third data coding system has a feature map scaling operation and a probability distribution parameter scaling operation. For example, the third data coding system scales both the mean μ and the variance σ. FIG. 6A

[0266] Data is taken as an example of image data, and the Kodak test set is taken as data to be processed, which contains 24 PNG images with a resolution of 768x512 or 512x768. Each test image is processed by using the second data coding system and the third data coding system respectively, and BPP, PSNR and BD Rate are obtained (see Table 2 for details). Similarly, the BPP and PSNR in Table 2 are the average values of the 24 images.

[0267] From Table 2, it can be seen that the results of the third data coding system are better than those of the second data coding system. The BD Rate of the third data coding system is better, and the performance is improved by 2.71%. It can be seen that scaling the probability distribution parameters can improve the compression performance.​​

[0268] Table 2 Performance comparison of the second data coding system and the third data coding system

[0269]

[0270] The first data coding system will be compared with a data coding system composed of the residual feature map scaling operation and the probability distribution parameter scaling operation (hereinafter referred to as the fourth data coding system). The first data coding system has the residual feature map scaling operation, and the fourth data coding system has the residual feature map scaling operation and the probability distribution parameter scaling operation. For example, the fourth data coding system scales the variance σ. FIG. 4D 、 FIG. 4E

[0271] The data is taken as image data, and the Kodak test set is taken as the data to be processed, which contains 24 PNG images with a resolution of 768x512 or 512x768. The first data coding system and the fourth data coding system are respectively used to process each test image, and the BPP, PSNR and BD Rate are obtained. For details, refer to Table 3. Similarly, the BPP and PSNR in Table 3 are the average values of the 24 images.

[0272] As can be seen from Table 3, the results of the fourth data coding system are better than those of the first data coding system. The BD Rate of the fourth data coding system is better, and the performance is improved by 0.65%. It can be seen that scaling the probability distribution parameter can improve the compression performance.

[0273] Table 3 Performance comparison of the first data coding system and the fourth data coding system

[0274]

[0275] FIG. 7 FIG. 7 is a flowchart of a data encoding method according to an embodiment of the present application. The data encoding method 700 is performed by the data encoder 20. FIG. 7 The method shown is described as a series of steps or operations. It should be understood that the method can be performed in various orders and / or simultaneously, and is not limited to the order shown. FIG. 7 The order of execution shown.

[0276] As shown in FIG. 7, the data encoding method 700 includes: FIG. 7

[0277] 701, performing side information feature extraction on the first feature map of the current data to obtain a side information feature map.

[0278] ​​The data includes at least one of image data, video data, motion vector data of the video data, audio data, point cloud data, or text data. The side information is information used to assist in encoding of the information X, so that the length of the encoded information X is shorter, that is, the redundancy in the information X is reduced. In this embodiment of the application, the side information is the part of information extracted from the first feature map and used to assist in encoding and decoding of the first feature map.

[0279] 702. The side information feature map is quantized to obtain a first quantized feature map.

[0280] 703. The first quantized feature map is entropy encoded to obtain a first bitstream of the current data.

[0281] The bitstream is a bitstream generated after encoding. At this time, the first bitstream is a bitstream obtained after entropy encoding of the first quantized feature map.

[0282] 704. The residual feature map is scaled according to the scaling coefficient to obtain a scaled feature map, and the scaled feature map is quantized to obtain a second quantized feature map. The residual feature map is obtained according to the second feature map of the current data and the first probability distribution parameter, and the scaling coefficient is obtained based on the first quantized feature map. At this time, the scaling is a scaling down.

[0283] In this embodiment, the first feature map and the second feature map are different feature maps obtained by feature extraction on complete current data.

[0284] 705. The second quantized feature map is entropy encoded according to the first probability distribution parameter to obtain a second bitstream of the current data.

[0285] The first probability distribution parameter is a matrix with the same size as the second quantized feature map. Each element in the matrix represents a probability distribution parameter of each element in the second quantized feature map. The probability distribution parameter of each element includes but is not limited to mean and / or variance, that is, one second quantized feature map corresponds to one mean matrix and / or one variance matrix. The second bitstream is a bitstream obtained after entropy encoding of the second quantized feature map.

[0286] In the scheme, the residual feature map of the current data is scaled to obtain a scaled feature map, and then the scaled feature map is quantized, so that the quantization loss of the scaled feature map is smaller, that is, the information loss of the second code stream is smaller, which helps to improve the data quality of the reconstructed data obtained by decoding the second code stream. On the other hand, compared with the coding network structure in the prior art, since the coding network structure comprising residual feature map scaling is adopted in the embodiment of the application, after the training of the entire coding network is completed, the network parameters of the entire coding network can be optimized, including the network for generating the first code stream and the network for generating the second code stream. Therefore, by using the coding network structure of the embodiment of the application, the data amount of the total code stream of the current data can be reduced, and the coding efficiency can be improved. In general, by using the coding method of the embodiment, the data compression performance can be further improved.

[0287] Exemplarily, the first quantized feature map is input into the hyper-prior decoding network, and the first probability distribution parameter and the scaling coefficient can be predicted. The first probability distribution parameter is used to represent the probability distribution of the second quantized feature map.

[0288] Exemplarily, the first quantized feature map is input into the hyper-prior decoding network, and the scaling coefficient can be predicted. The first probability distribution parameter is a preset probability distribution parameter.

[0289] In some possible embodiments, the first probability distribution parameter comprises a mean and / or a variance.

[0290] In some possible embodiments, the first feature map and the second feature map are the same.

[0291] FIG. 8 is a flowchart of another data encoding method provided by the embodiment of the application. The data encoding method 800 is executed by the data encoder 20. FIG. 8 The method shown is described as a series of steps or operations. It should be understood that the method can be executed in various orders and / or simultaneously, and is not limited to FIG. 8 The execution order shown.

[0292] As FIG. 8 shown, the data encoding method 800 comprises:

[0293] 801, edge information feature extraction is performed on the first feature map of the current data to obtain an edge information feature map.

[0294] The data includes at least one of image data, video data, motion vector data of the video data, audio data, point cloud data, or text data. The side information is used to assist in encoding the information X by using the existing information Y, so that the encoding length of the information X is shorter, that is, the redundancy in the information X is reduced. In the embodiment of the application, the side information refers to the part of information extracted from the first feature map for assisting in the encoding and decoding of the first feature map.

[0295] 802. Perform quantization processing on the side information feature map to obtain a first quantized feature map.

[0296] 803. Perform entropy encoding on the first quantized feature map to obtain a first code stream of the current data.

[0297] The code stream refers to a bit stream generated after encoding processing. The first code stream is a bit stream obtained after entropy encoding on the first quantized feature map.

[0298] 804. Perform scaling processing on the second feature map according to the scaling coefficient to obtain a scaled feature map.

[0299] The scaling coefficient is obtained based on the first quantized feature map. In the embodiment, the first feature map and the second feature map are different feature maps obtained by feature extraction on the complete current data. For example, when the first feature map and the second feature map are different, the first feature map ( FIG. 5B y1 in the first feature map) and the second feature map ( FIG. 5B y2 in the second feature map) can refer to the related description of FIG. 5B . At this time, the scaling processing is a scaling down processing.

[0300] 805. Perform quantization processing on the scaled feature map to obtain a second quantized feature map.

[0301] 806. Perform scaling processing on the first probability distribution parameter according to the scaling coefficient to obtain a second probability distribution parameter.

[0302] The first probability distribution parameter is obtained based on the first quantized feature map. Alternatively, the first probability distribution parameter is a preset probability distribution parameter. The first probability distribution parameter is a matrix with the same scale as the second quantized feature map, and each element in the matrix represents a probability distribution parameter of each element in the second quantized feature map. The probability distribution parameter of each element includes but is not limited to mean and / or variance, that is, one second quantized feature map corresponds to one mean matrix and / or one variance matrix.

[0303] 807. Perform entropy encoding on the second quantized feature map according to the second probability distribution parameter to obtain a second code stream of the current data.

[0304] The second code stream is a bit stream obtained by entropy encoding the second quantized feature map.

[0305] In the present scheme, on the one hand, the first code stream of the current data can be obtained by entropy encoding the first quantized feature map. On the other hand, the scaling coefficient and the first probability distribution parameter can be estimated based on the first quantized feature map. Thus, the second feature map can be scaled based on the scaling coefficient to obtain a scaled feature map. The second quantized feature map can be obtained by quantizing the scaled feature map. The second probability distribution parameter can be obtained by scaling the first probability distribution parameter based on the scaling coefficient. Finally, the second code stream of the current data can be obtained by entropy encoding the second quantized feature map based on the second probability distribution parameter. The first code stream and the second code stream together serve as the total code stream of the current data. The data encoding method in the prior art only has the step of scaling the feature map. However, the present scheme scales the second feature map and the first probability distribution parameter using the same scaling coefficient, so that the second probability distribution parameter has a higher matching degree with the second quantized feature map, thereby improving the encoding accuracy of the second quantized feature map, i.e., improving the data compression performance. In some possible embodiments, the second feature map is a residual feature map of the current data, which is obtained based on a third feature map of the current data and the first probability distribution parameter. The third feature map is obtained by feature extraction on the current data.

[0306] In the present embodiment, the second feature map is a feature map obtained based on the current data, i.e., a residual feature map. In the present embodiment, the third feature map is a feature map obtained by feature extraction on the complete current data, which is different from the first feature map. The residual of the third feature map, i.e., the residual feature map, can be obtained by subtracting the first probability distribution parameter from the third feature map, and the residual feature map is taken as the second feature map of the current data. For example, when the first feature map and the third feature map are different, the first feature map (y1) and the third feature map (y2) can be obtained by feature extraction on the same data. FIG. 5B The specific method for obtaining the first feature map (y1) and the third feature map (y2) can be referred to the related description of the first feature map (y1) and the third feature map (y2) in the method for encoding the data. FIG. 5B FIG. 5B

[0307] ​​In the scheme, the residual feature map of the current data is scaled to obtain a scaled feature map, and then the scaled feature map is quantized, so that the quantization loss of the scaled feature map is smaller, that is, the information loss of the second code stream is smaller, which helps to improve the data quality of the reconstructed data obtained by decoding the second code stream. On the other hand, compared with the coding network structure in the prior art, since the coding network structure comprising residual feature map scaling is adopted in the embodiment of the application, after the training of the entire coding network is completed, the network parameters of the entire coding network can be optimized, including the network for generating the first code stream and the network for generating the second code stream. Therefore, by adopting the coding network structure of the embodiment of the application, the data amount of the total code stream of the current data can be reduced, and the coding efficiency can be improved. In summary, by combining residual feature map scaling and first probability distribution parameter scaling, the data compression performance can be further improved.

[0308] Further, in some possible embodiments, the first feature map and the third feature map are the same. For example, the specific acquisition method of the first feature map and the third feature map can refer to the related description of FIG. 5A .

[0309] In some possible embodiments, the second feature map is a feature map obtained by feature extraction on the current data. The first feature map and the second feature map are the same or different. For example, the specific acquisition method of the first feature map and the second feature map can refer to the related description of FIG. 5A .

[0310] In some possible embodiments, the first probability distribution parameter comprises a mean and / or a variance.

[0311] In some possible embodiments, the data encoding method further comprises:

[0312] The first code stream and the second code stream are transmitted.

[0313] After the first code stream and the second code stream of the current data are obtained by using the data encoding method of the scheme, the first code stream and the second code stream can be transmitted to other devices as needed, so that the other devices can process the first code stream and the second code stream.

[0314] In some possible embodiments, the first code stream and the second code stream are saved in the form of a code stream file.

[0315] FIG. 9 is a flowchart of a data decoding method provided by the embodiment of the application. The data decoding method 900 is executed by the data decoder 30. FIG. 9 The method shown is described as a series of steps or operations, and it should be understood that the method can be executed in various orders and / or simultaneously, and is not limited to the execution order shown in FIG. 9 .

[0316] As shown in FIG. 9 the data decoding method 900 includes:

[0317] 901. Entropy decoding a first code stream of current data to obtain a third feature map.

[0318] 902. Obtaining a scaling coefficient based on the third feature map.

[0319] 903. Entropy decoding a second code stream of the current data based on a first probability distribution parameter to obtain a fourth feature map.

[0320] wherein the first probability distribution parameter is a matrix with a same size as the fourth feature map, each element in the matrix represents a probability distribution parameter of each element in the fourth feature map, and the probability distribution parameter of each element includes but is not limited to a mean and / or a variance, i.e., one fourth feature map corresponds to one mean matrix and / or one variance matrix.

[0321] 904. Scaling the fourth feature map based on the scaling coefficient to obtain a fifth feature map. At this time, the scaling is an amplification.

[0322] 905. Obtaining a sixth feature map based on the first probability distribution parameter and the fifth feature map.

[0323] 906. Obtaining reconstructed data of the current data based on the sixth feature map.

[0324] In the case of scaling and quantizing the residual feature map during data encoding, the data decoding method of the present scheme can effectively improve the decoding efficiency because the data amount of the first code stream and the second code stream is smaller than that of the prior art, and the data quality of the reconstructed data obtained by the present scheme is higher because the information loss of the second code stream is smaller.

[0325] In some possible embodiments, the first feature map and the second feature map are the same or different.

[0326] In some possible embodiments, the data includes at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

[0327] In some possible embodiments, the first probability distribution parameter includes a mean and / or a variance.

[0328] In some possible embodiments, the data decoding method further includes:

[0329] obtaining the first probability distribution parameter based on the third feature map.

[0330] In some possible embodiments, the first probability distribution parameter is a preset probability distribution parameter.

[0331] FIG. 10 is a flowchart of another data decoding method provided by an embodiment of the present application. The data decoding method 1000 is performed by the data decoder 30. FIG. 10 The method shown is described as a series of steps or operations. It should be understood that the method can be performed in various sequences and / or simultaneously, and is not limited to FIG. 10 the execution sequence shown.

[0332] As FIG. 10 shown, the data decoding method 1000 includes:

[0333] 1001. Entropy decoding a first code stream of current data to obtain a third feature map.

[0334] 1002. Obtaining a scaling coefficient based on the third feature map.

[0335] 1003. Scaling the first probability distribution parameter according to the scaling coefficient to obtain a second probability distribution parameter.

[0336] The first probability distribution parameter is a matrix with a same scale as that of the fourth feature map. Each element in the matrix represents a probability distribution parameter of each element in the fourth feature map. The probability distribution parameter of each element includes but is not limited to a mean and / or a variance, that is, one fourth feature map corresponds to one mean matrix and / or one variance matrix.

[0337] 1004. Entropy decoding a second code stream of the current data based on the second probability distribution parameter to obtain a fourth feature map.

[0338] 1005. Scaling the fourth feature map according to the scaling coefficient to obtain a fifth feature map. At this time, the scaling processing is amplification processing.

[0339] 1006. Obtaining reconstructed data of the current data based on the fifth feature map.

[0340] When data is encoded, the scaling processing of the second feature map and the scaling processing of the first probability distribution parameter are performed. Correspondingly, when data is decoded, the same scaling coefficient is used to process the first probability distribution parameter and the fourth feature map, so as to guarantee the decoding accuracy of the fourth feature map. In addition, the fourth feature map is inversely scaled according to the scaling coefficient to obtain the fifth feature map, and the reconstructed data can be obtained based on the fifth feature map. The precision and quality of the reconstructed data are higher. In general, the combination of the first probability distribution parameter scaling and the fourth feature map inverse scaling processing can improve the precision and quality of data decoding.

[0341] In some possible embodiments, the fourth feature map is a residual feature map, and the reconstructed data of the current data is obtained based on the fifth feature map, including:

[0342] The first probability distribution parameter and the fifth feature map are added to obtain a sixth feature map.

[0343] The reconstructed data of the current data is obtained based on the sixth feature map.

[0344] In the case of performing the scaling and quantization operation on the residual feature map when encoding the data, the data decoding method of the present solution has a smaller data amount of the first code stream and the second code stream compared with the prior art, and thus has a smaller decoding processing amount. The present solution can effectively improve the decoding efficiency. Moreover, the information loss of the second code stream is smaller, and thus the data quality of the reconstructed data obtained by using the present solution is higher.

[0345] In some possible embodiments, the first probability distribution parameter includes a mean value and / or a variance.

[0346] In some possible embodiments, the data includes at least one of the following: image data, video data, motion vector data of the video data, audio data, point cloud data, or text data.

[0347] In some possible embodiments, the data decoding method further includes:

[0348] The first code stream and the second code stream of the current data are received.

[0349] In some possible embodiments, the data decoding method further includes:

[0350] The first probability distribution parameter is obtained based on the third feature map.

[0351] In some possible embodiments, the first probability distribution parameter is a preset probability distribution parameter.

[0352] If the method functions of any embodiments of the present application are realized in the form of software function units and sold or used as independent products, the method functions can be stored in a computer readable storage medium. Based on such understanding, the data encoding method and / or the data decoding method of the present application can be embodied in the form of a computer program product, which is stored in a storage medium and includes a plurality of instructions for causing an electronic device to execute all or part of the steps of the method described in the embodiments of the present application. The storage medium described above includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0353] The application further provides a computer readable storage medium, which stores a code stream generated by the data encoding method according to any one of the above embodiments.

[0354] The application further provides a computer readable storage medium, which stores a code stream containing program code, when the program code is executed by one or more processors, the decoder performs the data decoding method according to any one of the above embodiments.

[0355] The application further provides a chip applied to an electronic device, the chip comprising one or more processors configured to invoke computer instructions to cause the electronic device to perform the data encoding method and / or the data decoding method according to any one of the above embodiments.

[0356] The application further provides a computer program product containing instructions, when the computer program product is run on an electronic device, the electronic device performs the data encoding method and / or the data decoding method according to any one of the above embodiments.

[0357] It can be understood that the computer storage medium, the chip and the computer program product provided above are all used to perform the data encoding method and / or the data decoding method according to any one of the above embodiments. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects of the data encoding method and / or the data decoding method according to any one of the above embodiments, which will not be described here again.

[0358] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solutions. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0359] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the above-described apparatus embodiments are only schematic, for example, the division of units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0360] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0361] In addition, each functional unit in each embodiment of the present patent application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0362] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data encoding method characterized by, The method comprises the following steps: performing side information feature extraction on a first feature map of current data to obtain a side information feature map; performing quantization processing on the side information feature map to obtain a first quantized feature map; performing entropy coding on the first quantized feature map to obtain a first code stream of the current data; performing scaling processing on a second feature map according to a scaling coefficient to obtain a scaled feature map, the scaling coefficient being obtained based on the first quantized feature map; performing quantization processing on the scaled feature map to obtain a second quantized feature map; performing scaling processing on a first probability distribution parameter according to the scaling coefficient to obtain a second probability distribution parameter; performing entropy coding on the second quantized feature map according to the second probability distribution parameter to obtain a second code stream of the current data.

2. The method of claim 1, wherein, The second feature map is a residual feature map of the current data obtained based on a third feature map of the current data and the first probability distribution parameter, the third feature map being obtained by performing feature extraction on the current data.

3. The method of claim 1, wherein, The second feature map is a feature map obtained by performing feature extraction on the current data.

4. The method of claim 2, wherein, The first feature map and the third feature map are the same.

5. The method of claim 3, wherein, The first feature map and the second feature map are the same.

6. The method according to any one of claims 1 to 5, characterized in that, The first probability distribution parameter comprises a mean value and / or a variance.

7. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: sending the first code stream and the second code stream.

8. The method according to any one of claims 1 to 5, characterized in that, The data comprises at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

9. A data decoding method characterized by comprising: The method comprises the following steps: performing entropy decoding on a first code stream of current data to obtain a third feature map; obtaining a scaling coefficient based on the third feature map; performing scaling processing on a first probability distribution parameter according to the scaling coefficient to obtain a second probability distribution parameter; performing entropy decoding on a second code stream of the current data based on the second probability distribution parameter to obtain a fourth feature map; performing scaling processing on the fourth feature map according to the scaling coefficient to obtain a fifth feature map; obtaining reconstructed data of the current data based on the fifth feature map.

10. The method of claim 9, wherein, The fourth feature map is a residual feature map, and the fifth feature map is the scaled residual feature map. Obtaining the reconstructed data of the current data based on the fifth feature map comprises: adding the first probability distribution parameter and the fifth feature map to obtain a sixth feature map; obtaining the reconstructed data of the current data based on the sixth feature map.

11. The method according to claim 9 or 10, characterized in that, The first probability distribution parameter comprises a mean value and / or a variance.

12. The method of claim 9 or 10, wherein, The data comprises at least one of the following: image data, video data, motion vector data of video data, audio data, point cloud data, or text data.

13. The method of claim 9 or 10, wherein, The method further comprises: receiving the first code stream and the second code stream of the current data.

14. A data encoder, characterized by The processing circuit is configured to perform the data coding method according to any one of claims 1 to 8.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program and a code stream, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 8 to generate the code stream.

16. A data decoder, characterized by comprising processing circuitry configured to perform the data decoding method of any one of claims 9-13.

17. A computer program product, characterised in that, comprising program code configured to perform the method of any one of claims 1-13 when executed on a computer or processor.

18. A data encoder, comprising: comprising: one or more processors; a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium having stored thereon a program, wherein the program, when executed by the one or more processors, causes the data encoder to perform the data encoding method of any one of claims 1-8.

19. A data decoder, characterized by comprising: one or more processors; a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium having stored thereon a program, wherein the program, when executed by the one or more processors, causes the data decoder to perform the data decoding method of any one of claims 9-13.

20. A computer-readable storage medium, characterized in that, the computer-readable storage medium having stored thereon program code configured to perform the method of any one of claims 1-13 when executed on a computer device.

Images