Hierarchical coding and decoding method and model training method, device and equipment and medium

Abstract

This application relates to the field of artificial intelligence, providing a hierarchical encoding and decoding method, model training method, apparatus, device, and medium, comprising: receiving an input bitstream and encoding rate requirement information; determining an N-layer mature deep learning network based on the encoding rate requirement information; N being a natural number greater than or equal to 1 and less than or equal to M; calling the encoder network and the quantization network of the Nth-layer mature deep learning network to encode the input bitstream to obtain an Nth encoded bitstream; and calling the decoder network of the Nth-layer mature deep learning network to decode the Nth encoded bitstream to obtain an output bitstream. This application improves the utilization and training efficiency of deep learning networks while reducing the consumption of storage and computing resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a hierarchical encoding and decoding method, model training method, apparatus, device and medium. Background Technology

[0002] In the typical encoding and decoding process of audio and video streams, it is usually necessary to prepare multiple bitrates according to different transmission / storage conditions. This bitrate refers to the compression bitrate used to compress the input stream. Under good conditions, a compression algorithm with a higher bitrate is used, and under poor conditions, a compression algorithm with a lower bitrate is used.

[0003] Current deep learning-based compression algorithms all employ a retraining method with different bitrates. This means that a deep learning network is designed and trained for each bitrate, requiring multiple deep learning networks to be trained for different bitrates. However, the inventors discovered that this approach not only results in low utilization of each deep learning network due to their lack of interdependence, but also wastes storage and computational resources by repeatedly training multiple deep learning networks. Summary of the Invention

[0004] This application provides a hierarchical encoding and decoding method, model training method, apparatus, device, and medium to solve the problem.

[0005] In a first aspect, this application provides a hierarchical encoding and decoding method based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0006] The hierarchical encoding and decoding method includes:

[0007] Receive the input bitstream and encoding rate requirement information, and determine an N-layer mature deep learning network based on the encoding rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M;

[0008] The encoder network and the quantization network of the Nth layer mature deep learning network are invoked to encode the input bitstream to obtain the Nth encoded bitstream;

[0009] The decoder network in the Nth layer of the mature deep learning network is invoked to decode the Nth encoded bitstream to obtain the output bitstream.

[0010] In the above scheme, the encoder network and the quantization network of the Nth layer mature deep learning network are invoked to encode the input bitstream to obtain the Nth encoded bitstream, including:

[0011] The encoder networks, from the first layer of the mature deep learning network to the Nth layer of the mature deep learning network, are sequentially invoked to perform dimensionality reduction processing on the input bitstream to obtain the Nth encoding vector; wherein, the first layer of the mature deep learning network is the first mature deep learning network in the M layers of mature deep learning networks; and the Nth layer of the mature deep learning network is the last mature deep learning network in the N layers of mature deep learning networks.

[0012] The quantization network of the mature deep learning network at layer N is invoked to quantize the Nth encoding vector to obtain the Nth encoded bitstream.

[0013] In the above scheme, the quantization network includes: a convolutional neural network and a residual quantization network;

[0014] The Nth layer mature deep learning network is invoked to perform quantization processing on the Nth encoding vector to obtain the Nth encoded bitstream, including:

[0015] The Nth encoding vector is convolved by the convolutional neural network of the mature deep learning network at the Nth layer to obtain the Nth convolutional vector; wherein the Nth convolutional vector has at least one Nth convolutional sub-vector.

[0016] The index corresponding to each of the Nth convolutional sub-vectors is obtained from the code table of the residual quantization network, and the indexes corresponding to each of the Nth convolutional sub-vectors are summarized to obtain the Nth encoded code stream; the code table is an index codeword table pre-set in the residual quantization network, the index codeword table records at least one layer of indexes and codewords corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used for comparison with the Nth convolutional sub-vector.

[0017] In the above scheme, the decoder network in the Nth layer of the mature deep learning network is invoked to decode the Nth encoded bitstream to obtain the output bitstream, including:

[0018] The Nth encoded bitstream is dequantized using a preset code table to obtain the Nth code table vector.

[0019] The decoder network, from the Nth layer mature deep learning network to the first layer mature deep learning network, is called sequentially to perform dimensionality upscaling on the Nth code table vector to obtain the output bitstream.

[0020] In the above scheme, the Nth encoded bitstream is dequantized using a preset code table to obtain the Nth code table vector, including:

[0021] Obtain the codeword of each index in the code table in the Nth encoded bitstream;

[0022] The Nth code table vector is obtained by summing the codewords corresponding to each index.

[0023] Secondly, this application provides a model training method for a multi-level deep learning network model, wherein the multi-level deep learning network model includes an M-layer initial deep learning network; the model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0024] The model training method includes:

[0025] The training input bitstream is obtained, and the encoder network in the (L-1)th layer mature deep learning network, as well as the encoder and quantization networks of the Lth layer initial deep learning network, are invoked to encode the training input bitstream to obtain the Lth encoded bitstream. Here, L refers to the number of layers in the initial deep learning network used for encoding and decoding the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M.

[0026] The decoder network in the mature deep learning network of layer L-1 and the decoder network in the initial deep learning network of layer L are invoked to decode the Lth encoded bitstream to obtain the Lth training output bitstream;

[0027] Based on the vector loss, the training input bitstream, and the Lth training output bitstream, the model parameters of the network units in the Lth initial deep learning network are adjusted to transform the Lth initial deep learning network into an Lth mature deep learning network. The vector loss is the difference between the Lth convolutional vector and the vector represented by the Lth encoded bitstream; the Lth convolutional vector is the input vector used by the Lth initial deep learning network to generate the Lth encoded bitstream; the network units include one or more of encoder networks, quantization networks, and decoder networks; the model parameters include the weights and / or biases of the network units.

[0028] A multi-level deep learning network model is established by completely transforming the initial deep learning network of M layers into a mature deep learning network of M layers. This model includes a mature deep learning network of M layers. The multi-level deep learning network model includes a mature deep learning network of M layers and is used to receive the input bitstream and encoding rate requirement information. Based on the encoding rate requirement information, an N-layer mature deep learning network is determined. N is a natural number greater than or equal to 1 and less than or equal to M. The encoder network of the N-layer mature deep learning network and the quantization network of the Nth-layer mature deep learning network are called to encode the input bitstream to obtain the Nth encoded bitstream. The decoder network of the N-layer mature deep learning network is called to decode the Nth encoded bitstream to obtain the output bitstream.

[0029] In the above scheme, the encoder network in the mature deep learning network of layer L-1, as well as the encoder network and quantization network of the initial deep learning network of layer L, are invoked to encode the training input bitstream to obtain the Lth encoded bitstream, including:

[0030] The encoder networks from the first mature deep learning network to the (L-1)th mature deep learning network, and the encoder network of the Lth initial deep learning network are sequentially invoked to perform dimensionality reduction processing on the training input bitstream to obtain the Lth training vector; wherein, the first initial deep learning network is the first initial deep learning network in the M layers of initial deep learning networks; the Lth initial deep learning network is the last initial deep learning network in the L layers of initial deep learning networks;

[0031] The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L.

[0032] In the above scheme, the quantization network includes: a convolutional neural network and a residual quantization network;

[0033] The process involves calling the quantization network of the initial deep learning network at layer L to quantize the training vectors at layer L to obtain the encoded bitstream at layer L, including:

[0034] The convolutional neural network of the Lth initial deep learning network is invoked to perform convolution processing on the Lth training vector to obtain the Lth convolution vector; wherein the Lth convolution vector has at least one Lth convolution sub-vector.

[0035] The index corresponding to each Lth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Lth convolutional subvector are summarized to obtain the Lth encoded code stream; wherein, the code table is an index codeword table pre-set in the residual quantization network, the index codeword table records at least one layer of indexes and codewords corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used for comparison with the Nth convolutional subvector.

[0036] In the above scheme, the decoder network in the mature deep learning network of layer L-1 and the decoder network in the initial deep learning network of layer L are invoked to decode the Lth encoded bitstream to obtain the Lth training output bitstream, including:

[0037] The Lth encoded bitstream is dequantized using a preset code table to obtain the Lth code table vector.

[0038] The decoder network of the initial deep learning network at layer L and the decoder network from the initial deep learning network at layer L-1 to the initial deep learning network at layer 1 are sequentially called to perform dimensionality upscaling on the Lth code table vector to obtain the Lth training output bitstream.

[0039] In the above scheme, the Lth encoded bitstream is dequantized using a preset code table to obtain the Lth code table vector, including:

[0040] Obtain the codeword of each index in the code table in the Lth encoded bitstream;

[0041] The Lth code table vector is obtained by summing the codewords corresponding to each index.

[0042] In the above scheme, the model parameters of the network units in the initial deep learning network of the Lth layer are adjusted according to the vector loss, the training input bitstream, and the Lth training output bitstream, so that the initial deep learning network of the Lth layer is transformed into a mature deep learning network of the Lth layer, including:

[0043] The first loss function is called to obtain the first loss value based on the training input bitstream and the Lth training output bitstream; wherein, the first loss value represents the difference between the training input bitstream and the Lth training output bitstream;

[0044] The second loss function is called to obtain the second loss value based on the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to the preset code table and the Lth convolution vector.

[0045] The model parameters of the network units in the initial deep learning network of the Lth layer are adjusted by gradient descent based on the first loss value and the second loss value, so that the adjusted initial deep learning network of the Lth layer is transformed into the mature learning network of the Lth layer.

[0046] Thirdly, this application provides a hierarchical encoding and decoding device based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1.

[0047] The hierarchical encoding and decoding device includes: an encoding input module, an encoding processing module, and a decoding processing module;

[0048] The encoding input module is used to receive the input bitstream and encoding bitrate requirement information, and to determine an N-layer mature deep learning network based on the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M.

[0049] The encoding input module is connected to the encoding processing module. The encoding processing module is used to call the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream.

[0050] The encoding processing module is connected to the decoding processing module. The decoding processing module is used to call the decoder network in the N-layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

[0051] Fourthly, this application provides a model training apparatus for a multi-level deep learning network model, wherein the multi-level deep learning network model includes an M-layer initial deep learning network; the model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0052] The model training device includes: a training encoding module, a training decoding module, a training adjustment module, and a training integration module;

[0053] The training encoding module is used to acquire the training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder and quantization networks of the Lth layer initial deep learning network to encode the training input bitstream to obtain the Lth encoded bitstream; where L refers to the number of layers in the initial deep learning network used to encode and decode the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the Lth layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M;

[0054] The training encoding module is connected to the training decoding module. The training decoding module is used to call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream.

[0055] The training adjustment module is connected to the training decoding module. The training decoding module is used to adjust the model parameters of the network units in the Lth layer initial deep learning network according to the vector loss, the training input bitstream, and the Lth training output bitstream, so that the Lth layer initial deep learning network is transformed into the Lth layer mature deep learning network. The vector loss is the difference between the Lth convolutional vector and the vector represented by the Lth encoded bitstream; the Lth convolutional vector is the input vector used by the Lth layer initial deep learning network to generate the Lth encoded bitstream; the network unit includes one or more of an encoder network, a quantization network, and a decoder network; the model parameters include the weights and / or biases of the network units.

[0056] The training adjustment module is connected to the training integration module. The training integration module is used to completely transform the initial M-layer deep learning network into a multi-level deep learning network model containing M mature deep learning networks. This multi-level deep learning network model includes M mature deep learning networks. The multi-level deep learning network model includes M mature deep learning networks and is used to receive input bitstream and encoding rate requirement information. Based on the encoding rate requirement information, it determines an N-layer mature deep learning network. N is a natural number greater than or equal to 1 and less than or equal to M. The module calls the encoder network and the quantization network of the Nth mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream. Finally, it calls the decoder network of the Nth mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

[0057] Fifthly, this application provides a computer device, including: a processor and a memory communicatively connected to the processor;

[0058] The memory stores computer-executed instructions;

[0059] The processor executes computer execution instructions stored in the memory to implement the method as described above.

[0060] Sixthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described method.

[0061] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0062] This application provides a hierarchical encoding and decoding method, model training method, apparatus, device, and medium. By determining an N-layer mature deep learning network based on the encoding bitrate requirement information, and calling the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream, and then calling the decoder network of the Nth layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream, this method allows for encoding and decoding of an input bitstream with a given bitrate by calling the encoder and decoder networks of multiple mature deep learning networks, and also by calling the quantization network of the Nth layer mature deep learning network corresponding to that bitrate. This achieves the goal of calling multiple mature deep learning networks to encode and decode an input bitstream with a given bitrate, improving the utilization rate of each layer of deep learning networks.

[0063] Meanwhile, training each initial deep learning network can leverage previously trained mature deep learning networks to provide encoder and decoder network support for the currently being trained initial deep learning network. Based on the Nth training output bitstream and training input bitstream obtained with this support, training of the initial deep learning network can be completed by adjusting the model parameters of the currently being trained initial deep learning network. This improves the training efficiency of deep learning networks, avoids the situation of repeatedly training multiple deep learning networks, and greatly reduces the consumption of storage and computing resources. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0066] Figure 2A flowchart of Embodiment 1 of a hierarchical encoding and decoding method provided in this application;

[0067] Figure 3 In Embodiment 1 of the hierarchical encoding and decoding method provided in this application, taking N as 3 as an example, a schematic block diagram of the structure of the input bitstream being encoded and decoded to obtain the output bitstream is shown.

[0068] Figure 4 A flowchart of Embodiment 2 of a model training method provided in this application;

[0069] Figure 5 In Embodiment 2 of the model training method provided in this application, a schematic block diagram of the structure for training the first layer of the initial deep learning network is shown, taking the training of the first layer as an example.

[0070] Figure 6 In Embodiment 2 of the model training method provided in this application, a schematic block diagram of the structure for training the initial deep learning network of the second layer is shown, taking the training of the second layer as an example.

[0071] Figure 7 In Embodiment 2 of the model training method provided in this application, a schematic block diagram of the structure for training the initial deep learning network of the third layer is shown, taking the training of the third layer as an example.

[0072] Figure 8 A schematic diagram of the program modules of Embodiment 3 of a hierarchical encoding and decoding device provided by the present invention;

[0073] Figure 9 A schematic diagram of the program modules of Embodiment 4 of the model training device provided by the present invention;

[0074] Figure 10 This is a schematic diagram of the hardware structure of the computer device in the computer device of the present invention.

[0075] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0077] Please see Figure 1 The specific application scenario of this application is as follows:

[0078] A server 11 running hierarchical encoding and decoding methods and model training methods is installed with a multi-level deep learning network model 12. The multi-level deep learning network model 12 includes an M-layer mature deep learning network 13. The M-layer mature deep learning network 13 is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates. The mature deep learning network 13 includes an encoder network 131, a quantization network 132, and a decoder network 133. The quantization network 132 includes a convolutional neural network 1321 and a residual quantization network 1322.

[0079] Server 11 receives the input bitstream and encoding bitrate requirement information, and determines an N-layer mature deep learning network based on the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M.

[0080] Server 11 calls the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream;

[0081] Server 11 calls the decoder network in the N-layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

[0082] In this application, the encoder network and quantization network are used to compress the data, that is, to encode the input bitstream (e.g., images, videos, audio signals or files) into an encoded bitstream (a data stream that has been compressed); the decoder network is used to decompress the encoded bitstream (data stream) into an output bitstream (e.g., images, videos, audio signals or files).

[0083] Between the quantization network and the decoder network is a communication connection for data transmission. The two ends of the communication connection are connected to the first terminal and the second terminal, respectively. The first terminal can call the encoder network and the quantization network, and the second terminal can call the decoder network. The first terminal sends the encoded bitstream to the second terminal through the communication connection to achieve the compression and output of the input bitstream, and to decompress the compressed input bitstream to obtain the output bitstream.

[0084] The technical solution of this application and how the technical solution of this application solves the problems of the prior art will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0085] Example 1:

[0086] Please see Figure 2 This application provides a hierarchical encoding and decoding method based on a multi-level deep learning network model. The multi-level deep learning network model includes M layers of mature deep learning networks. The M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1.

[0087] Hierarchical encoding and decoding methods include:

[0088] S201: Receive the input bitstream and encoding rate requirement information, and determine an N-layer mature deep learning network based on the encoding rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M.

[0089] In this step, the encoding bitrate requirement information records the compression bitrate required to compress the input bitstream. This compression bitrate corresponds to the bitrate of the Nth layer of a mature M-layer deep learning network. Therefore, by determining the Nth layer of the mature deep learning network, the number of layers used to provide encoding and decoding processing for the input bitstream is determined. The bitrate of the Nth layer of the deep learning network refers to the compression bitrate used by the Nth layer for data compression.

[0090] S202: Call the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream.

[0091] In this step, by using a multi-layer mature deep learning network to encode the input bitstream, the technical effect of processing input bitstreams with multiple bitrates based on a multi-layer network is achieved, thereby improving the utilization rate of the multi-level deep learning network model.

[0092] In a preferred embodiment, the encoder network and the quantization network of the Nth mature deep learning network are invoked to encode the input bitstream to obtain the Nth encoded bitstream, including:

[0093] The encoder networks, from the first layer to the Nth layer of the mature deep learning network, are sequentially called to reduce the dimensionality of the input bitstream and obtain the Nth encoded vector. The first layer of the mature deep learning network is the first mature deep learning network in the M layers of mature deep learning networks, and the Nth layer of the mature deep learning network is the last mature deep learning network in the N layers of mature deep learning networks.

[0094] The quantization network of the mature deep learning network at layer N is invoked to quantize the Nth encoded vector to obtain the Nth encoded bitstream.

[0095] The input bitstream is reduced in dimensionality by using an N-layer encoder network, which allows the Nth encoding vector to be adapted to the quantization network. The Nth encoding vector is then quantized by the quantization network, ensuring the accuracy of quantization processing for input bitstreams with different bit rates.

[0096] For example, taking N=3 as an example, it represents that the input bitstream needs to be processed by the encoder network and decoder network of a three-layer mature deep learning network, as well as the quantization network in the third-layer mature deep learning network.

[0097] The encoder network 1 of the first layer mature deep learning network, the encoder network 2 of the second layer mature deep learning network, and the encoder network 3 of the third layer mature deep learning network are called in sequence to perform dimensionality reduction processing on the input bitstream to obtain the corresponding third encoding vector; then the third encoding vector is quantized through the quantization network of the third layer mature deep learning network to obtain the third encoded bitstream.

[0098] Furthermore, quantization networks include: convolutional neural networks and residual quantization networks;

[0099] The quantization network of the Nth layer mature deep learning network is invoked to quantize the Nth encoding vector to obtain the Nth encoded bitstream, including:

[0100] The Nth encoding vector is convolved by the convolutional neural network of the Nth layer of a mature deep learning network to obtain the Nth convolutional vector; wherein the Nth convolutional vector has at least one Nth convolutional sub-vector.

[0101] The index corresponding to each Nth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Nth convolutional subvector are summarized to obtain the Nth encoded code stream. The code table is an index codeword table pre-set in the residual quantization network. The index codeword table records at least one level of index and the codeword corresponding to each index. The index is the quantization value used to describe the codeword. The codeword is a multi-dimensional vector used to compare with the Nth convolutional subvector.

[0102] First, the Nth encoded vector is convolved to make the length of the output convolution vector a specified length. Then, according to the code table in the residual quantization network, the index of each convolution sub-vector of the specified length is determined to quantize the convolution vector. The resulting encoded bitstream is the data used to represent the convolution vector based on the code table.

[0103] In the example, the dimension of the residual quantization code table for each layer is (1024, 64), that is, the code table consists of 1024 vectors of length 64.

[0104] For example, please refer to Figure 3Taking N=3 as an example, its representation requires a three-layer mature deep learning network, namely the encoder network and decoder network, as well as the quantization network in the third layer of the mature deep learning network to process the input bitstream.

[0105] Assuming the input bitstream has a length of 96,000 samples per channel, it will be represented as (1, 96,000). Simultaneously, assuming the temporal dimensionality reduction factors of the encoder networks of the first, second, third, and fourth mature deep learning networks are 3, 4, 5, and 5 respectively; the input bitstream (1, 96,000) passes through the first, second, and third mature deep learning networks... The encoder networks 1, 2, and 3 are successively reduced in dimensionality to multidimensional vectors of (64, 32000), (128, 8000), and (256, 1600). At this point, the third encoded vector is (256, 1600). After the third encoded vector is processed by the convolutional neural network CNN3 of the third mature deep learning network, it will be transformed into a third convolutional vector of (64, 1600). The third convolutional vector represents 1600 multidimensional vectors of length 64.

[0106] From the code table of the residual quantization network 3 of the third-layer mature deep learning network, the index corresponding to each third convolutional sub-vector is obtained. The index corresponding to each third convolutional sub-vector refers to the index in the code table where the codeword is closest to the third convolutional sub-vector. Therefore, a total of 1600 10-bit indices are obtained, where each index in the code table consists of 10 bits (10 bits cover the index range of the code table). This generates a 1600x10-bit third-encoded bitstream, which is the compressed bitstream.

[0107] S203: Call the decoder network in the N-layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

[0108] By determining an N-layer mature deep learning network based on the required encoding bitrate, and then calling the encoder network and quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream, and finally calling the decoder network of the Nth layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream, this method allows for encoding and decoding of an input bitstream at a certain bitrate by calling the encoder and decoder networks of multiple mature deep learning networks, and also calling the quantization network of the Nth layer mature deep learning network corresponding to that bitrate to encode the input bitstream. This achieves the goal of calling multiple mature deep learning networks to encode and decode an input bitstream at a certain bitrate, thus improving the utilization rate of each layer of deep learning networks.

[0109] In a preferred embodiment, the decoder network in an N-layer mature deep learning network is invoked to decode the Nth encoded bitstream to obtain the output bitstream, including:

[0110] The Nth coded bitstream is dequantized using a preset code table to obtain the Nth code table vector.

[0111] The decoder network is called sequentially from the Nth layer mature deep learning network to the first layer mature deep learning network to perform dimensionality upscaling on the Nth code table vector to obtain the output bitstream.

[0112] By calling the decoder network of an N-layer mature deep learning network to perform dimensionality upscaling on the Nth code table vector, the participation of each mature deep learning network is increased, thereby improving the utilization rate of multi-level deep learning network models.

[0113] For example, based on the above example, the third encoded bitstream is dequantized by the quantization network 3 of the third mature neural network to obtain the third code table vector. The encoder network 3, 3 in the third mature neural network, the encoder network 2 in the second mature neural network, and the encoder network 1 in the first mature neural network are called in sequence to perform dimensionality increase processing on the third code table vector to obtain the output bitstream.

[0114] Furthermore, the Nth encoded bitstream is dequantized using a preset code table to obtain the Nth code table vector, which includes:

[0115] Obtain the codeword in the code table for each index in the Nth encoded bitstream;

[0116] The Nth code table vector is obtained by summing the codewords corresponding to each index.

[0117] For example, based on the above examples, please refer to Figure 3 From the code table of the residual quantization network 3, 1600 vectors (64, 1600) are obtained from the 1600*10-bit code stream of the third encoding code stream. The third encoding code stream is dequantized to obtain the third code table vector (64, 1600). The vectors are then passed through the decoder network 3 of the third mature deep learning network, the decoder network 2 of the second mature deep learning network, and the decoder network 1 of the first mature deep learning network. The third code table vector (64, 1600) is then multidimensionally multiplied to obtain multidimensional vectors (128, 8000), (64, 32000), and (1, 96000). The vector (1, 96000) is then used as the output code stream.

[0118] Example 2:

[0119] Please see Figure 4This application provides a model training method for a multi-level deep learning network model, which includes an M-layer initial deep learning network. The model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network. The M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates. Here, M is a natural number greater than 1.

[0120] Model training methods include:

[0121] S401: Obtain the training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder and quantization networks of the Lth layer initial deep learning network to encode the training input bitstream to obtain the Lth encoded bitstream; where L refers to the number of layers in the initial deep learning network used to encode and decode the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the Lth layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M.

[0122] By encoding the training input code using the encoder network in the mature deep learning network of layer L-1, as well as the encoder and quantization networks of the initial deep learning network of layer L, the training results based on the mature deep learning network of layer L-1 are achieved, thus improving the technical effect of training the initial deep learning network of layer L and improving training efficiency.

[0123] S402: Call the decoder network in the mature deep learning network of layer L-1 and the decoder network in the initial deep learning network of layer L to decode the L-th encoded bitstream to obtain the L-th training output bitstream.

[0124] By using the decoder network in the mature deep learning network at layer L-1 and the decoder network in the initial deep learning network at layer L, the technical effect of training the initial deep learning network at layer L based on the training results of the mature deep learning network at layer L-1 is achieved, thereby improving training efficiency.

[0125] In a preferred embodiment, the encoder network in the L-1 layer mature deep learning network, as well as the encoder network and quantization network of the Lth layer initial deep learning network, are invoked to encode the training input bitstream to obtain the Lth encoded bitstream, including:

[0126] The encoder networks from the first mature deep learning network to the (L-1)th mature deep learning network, and the encoder network of the Lth initial deep learning network are called sequentially to reduce the dimensionality of the training input bitstream to obtain the Lth training vector; where the first initial deep learning network is the first initial deep learning network in the M initial deep learning networks; the Lth initial deep learning network is the last initial deep learning network in the L initial deep learning networks.

[0127] The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L.

[0128] Furthermore, quantization networks include: convolutional neural networks and residual quantization networks;

[0129] The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L, including:

[0130] The Lth training vector is convolved by the convolutional neural network of the Lth initial deep learning network to obtain the Lth convolutional vector; wherein the Lth convolutional vector has at least one Lth convolutional sub-vector.

[0131] The index corresponding to each Lth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Lth convolutional subvector are summarized to obtain the Lth encoded bitstream. Here, the code table is an index codeword table pre-set in the residual quantization network. The index codeword table records at least one level of index and the codeword corresponding to each index. The index is the quantization value used to describe the codeword. The codeword is a multidimensional vector used for comparison with the Nth convolutional subvector.

[0132] For example, a mono audio sampled at 48kHz is used as the training input bitstream, and the encoder network and decoder network are respectively set as convolutional neural networks. Therefore, the encoder network 1, encoder network 2, encoder network 3, and encoder network 4 of the initial neural network from the first layer to the fourth layer are set as encoder network CNN1, encoder network CNN2, encoder network CNN3, and encoder network CNN4, respectively; the decoder network 1, decoder network 2, decoder network 3, and decoder network 4 are set as decoder network CNN1, decoder network CNN2, decoder network CNN3, and decoder network CNN4, respectively.

[0133] The temporal dimensionality reduction factors of encoder networks 1, 2, 3, and 4 are 3, 4, 5, and 5, respectively; the temporal dimensionality increase factors of the decoder networks are the same as those of the corresponding encoder networks. The quantization network 1 of the initial neural network from the first layer includes: Convolutional Neural Network CNN1 and Residual Quantization Network 1; the quantization network 2 of the initial neural network from the second layer includes: Convolutional Neural Network CNN2 and Residual Quantization Network 2; the quantization network 3 of the initial neural network from the third layer includes: Convolutional Neural Network CNN3 and Residual Quantization Network 3; and the quantization network 4 of the initial neural network from the fourth layer includes: Convolutional Neural Network CNN4 and Residual Quantization Network 4.

[0134] Each residual quantization network has a code table of (1024, 64), which consists of 1024 multidimensional vectors of length 64. The role of the convolutional neural network (CNN) is to further transform the output of the encoder network to be equal to the length of the multidimensional vectors in the code table (length 64).

[0135] The first to fourth encoded bitstreams correspond to four bitrates from high to low, namely: the training input bitstreams of the bitrate requirement information-bitrate 1, bitrate requirement information-bitrate 2, bitrate requirement information-bitrate 3, and bitrate requirement information-bitrate 4. The first to fourth encoded bitstreams are the index values ​​of the code table corresponding to the multidimensional vectors output by the convolutional neural network (CNN).

[0136] Please see Figure 5 Taking the first coding bitrate as an example, it is used to train the highest bitrate. The input signal is processed and truncated into (1, 9600), i.e., mono, with 9600 sample points, as the training input bitstream. The training input bitstream is reduced to a first training vector of (64, 3200) by encoder network 1. The first training vector is transformed into (64, 3200), i.e., 3200 first convolution vectors of length 64, by convolutional neural network CNN1. The index of the codeword closest to each first convolution subvector in the first convolution vector is found from the code table in residual quantization 1. 3200 indices are found from the 3200 multidimensional vectors of length 64, as the first coding bitstream. Each index in the first coding bitstream consists of 10 bits (10 bits cover the index range of the code table). The 3200 multidimensional vectors of length 64 are transformed into 3200 10-bit vectors = 32000 bits. The first coding bitstream with 3200 10-bit indices is output.

[0137] Please see Figure 6Taking the second coding rate as an example, based on the first layer of the mature deep learning network, the weights of encoder network 1 and decoder network 1 in the first layer of the mature deep learning network are weights that have been trained and solidified. By processing the input signal, it is truncated into a segment of (1, 9600), i.e., mono, with 9600 sample points, as the training input bitstream; the training input bitstream is dimensionality-reduced by encoder network 1 in the first layer of the mature deep learning network to obtain a multidimensional vector of (64, 3200); the multidimensional vector of (64, 3200) is then dimensionality-reduced again by encoder network 2 in the second layer of the initial deep learning network to obtain a second training vector of (128, 800); the second training vector is transformed by the convolutional neural network CNN2 to (64, 800). That is, 800 second convolution vectors of length 64; the second convolution vectors find the index of the codeword closest to each second convolution sub-vector in the code table in residual quantization 2, and find 800 indices from the 800 multidimensional vectors of length 64 to obtain the second encoded bitstream; each index in the second encoded bitstream consists of 10 bits (10 bits cover the index range of the code table); the 800 multidimensional vectors of length 64 are transformed into 800 10-bit vectors = 8000 bits; the second encoded bitstream with 800 10-bit indices is output.

[0138] Please see Figure 7Taking the third coding rate as an example, based on the first and second layers of mature deep learning networks, the weights of encoder network 1 and decoder network 1 in the first layer of mature deep learning networks are weights that have been trained and solidified; the weights of encoder network 2 and decoder network 2 in the second layer of mature deep learning networks are weights that have been trained and solidified. The input signal is processed and truncated into a segment of (1, 9600), i.e., mono, with 9600 sample points, as the training input bitstream; the training input bitstream is reduced to a multidimensional vector of (64, 3200) by encoder network 1 of the first layer of mature deep learning networks; the multidimensional vector of (64, 3200) is further reduced to a multidimensional vector of (128, 800) by encoder network 2 of the second layer of mature deep learning networks; the multidimensional vector of (128, 800) is further reduced to a third training vector of (256, 160) by encoder network 3 of the third initial deep learning network; the third training vector... The CNN3 network transforms the data into (64, 160), which consists of 160 third convolutional vectors of length 64. The third convolutional vectors then find the index of the codeword closest to each third convolutional sub-vector in the code table of residual quantization 3. These 160 multidimensional vectors of length 64 generate a third encoded bitstream with 160 indices. Each index of the third encoded bitstream consists of 10 bits (10 bits cover the index range of the code table). The 160 multidimensional vectors of length 64 are transformed into 160 10-bit vectors = 1600 bits. The output is a third encoded bitstream with 160 10-bit indices.

[0139] Taking the fourth coding rate as an example, based on the above principle, the fourth white horse code stream is obtained based on the first, second and third layers of mature deep learning networks.

[0140] In a preferred embodiment, the decoder network in the L-1 layer mature deep learning network and the decoder network in the Lth layer initial deep learning network are invoked to decode the Lth encoded bitstream to obtain the Lth training output bitstream, including:

[0141] The Lth coded bitstream is dequantized using a preset code table to obtain the Lth code table vector.

[0142] The decoder network of the initial deep learning network at layer L and the decoder network from the initial deep learning network at layer L-1 to the initial deep learning network at layer 1 are called sequentially to perform dimensionality upscaling on the Lth code table vector to obtain the Lth training output bitstream.

[0143] Furthermore, the Lth encoded bitstream is dequantized using a preset code table to obtain the Lth code table vector, which includes:

[0144] Obtain the codeword in the code table for each index in the Lth encoded bitstream;

[0145] The Lth code table vector is obtained by summing the multidimensional vectors corresponding to each index.

[0146] For example, if a first encoded bitstream is obtained, the multi-dimensional vector corresponding to each index in the first encoded bitstream is found from the code table of the residual quantization network 1, and the vectors are aggregated to form a first code table vector; the first code table vector is output to the decoder network 1, the decoder network 1 performs dimensionality upscaling on the first code table vector, and outputs a first training output bitstream with a single-channel sample point count of 9600.

[0147] If the second encoded bitstream is obtained, the multidimensional vector corresponding to each index in the second encoded bitstream is found from the code table of the residual quantization network 2, and the vectors are summarized to form the second code table vector. The second code table vector is output to the decoder network 2, and the second code table vector is sequentially processed by the decoder network 2 and the decoder network 1 to perform dimensionality increase, and the second training output bitstream with a multidimensional vector of 9600 mono sample points is output.

[0148] If the third encoded bitstream is obtained, the multidimensional vector corresponding to each index in the third encoded bitstream is found from the code table of the residual quantization network 3, and the vectors are summarized to form the third code table vector. The third code table vector is output to the decoder network 3, and the third code table vector is sequentially processed by the decoder network 3, decoder network 2 and decoder network 1 to perform dimensionality increase processing, and the third training output bitstream with a multidimensional vector of 9600 mono sample points is output.

[0149] If the fourth encoded bitstream is obtained, the fourth training output bitstream is obtained according to the above principle.

[0150] S403: Based on the vector loss, the training input bitstream, and the Lth training output bitstream, adjust the model parameters of the network units in the Lth initial deep learning network to transform the Lth initial deep learning network into the Lth mature deep learning network; whereby, the vector loss is the difference between the Lth convolutional vector and the multidimensional vector represented by the Lth encoded bitstream; the Lth convolutional vector is the input vector used by the Lth initial deep learning network to generate the Lth encoded bitstream; the network units include one or more of the encoder network, quantization network, and decoder network; the model parameters include the weights and / or biases of the network units.

[0151] In an exemplary embodiment, assume there are n types of bitrate requirements, including bitrate requirement 1, bitrate requirement 2, ..., where encoded bitstream 1 corresponds to the highest bitrate, encoded bitstream 2 corresponds to the second highest bitrate, encoded bitstream 3 corresponds to the third highest bitrate, and so on, with encoded bitrate n corresponding to the lowest bitrate. In the training network, encoder network 1, encoder network 2, encoder network 3, ... represent a progressive dimensionality reduction process to achieve a bitrate from high to low, while decoder network n... decoder network 3, decoder network 2, decoder network 1 represent a progressive dimensionality increase process, corresponding one-to-one with the encoder network. During training, the encoder network 1 + quantization network 1 + decoder network 1 corresponding to the highest bitrate is trained first; then the second highest bitrate is trained, using the previously trained encoder network 1 + decoder network 1, and only encoder network 2 + quantization network 2 + decoder network 2 is trained; the third highest bitrate is trained again, using the previously trained encoder network 1 + encoder network 2 + decoder network 2 + decoder network 1, and only encoder network 3 + quantization network 3 + decoder network 3 is trained; and so on. For the lowest coding bitrate n, only encoder network n + quantization network n + decoder network n is needed, and the other networks use the pre-trained networks.

[0152] Therefore, the training process of this application is as follows: training the first-layer initial deep learning network; training the encoder network 1 + quantization network 1 + decoder network 1 in the first-layer initial deep learning network, so that the first-layer initial deep learning network is transformed into the first-layer mature deep learning network. The training input stream is pre-prepared audio or image.

[0153] The second initial deep learning network is trained; specifically, the encoder network 1 of the first mature deep learning network, the encoder network 2 of the second initial deep learning network, and the quantization network 2 are used to encode the training input bitstream to obtain a second encoded bitstream; the decoder network 2 of the second initial deep learning network and the decoder network 3 of the first mature deep learning network are used to decode the second encoded bitstream to obtain a second training output bitstream; based on the training input bitstream and the second training output bitstream, the encoder network 2 + quantization network 2 + decoder network 2 of the second initial deep learning network are trained. The training input bitstream is pre-prepared audio or image.

[0154] Following this principle, the third initial deep learning network, the fourth initial deep learning network, and so on, are trained sequentially until the Mth initial deep learning network is reached.

[0155] In a preferred embodiment, the model parameters of the network units in the initial deep learning network at layer L are adjusted based on the vector loss, the training input bitstream, and the training output bitstream at layer L, so that the initial deep learning network at layer L is transformed into a mature deep learning network at layer L, including:

[0156] The first loss function is called to obtain the first loss value based on the training input bitstream and the Lth training output bitstream; whereby the first loss value characterizes the difference between the training input bitstream and the Lth training output bitstream.

[0157] The second loss function is called to obtain the second loss value based on the vector loss; where the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to the preset code table and the Lth convolution vector.

[0158] The model parameters of the network units in the initial deep learning network of layer L are adjusted by gradient descent based on the first and second loss values, so that the adjusted initial deep learning network of layer L is transformed into the mature learning network of layer L.

[0159] For example, the first loss function is:

[0160]

[0161] Where, x i It is the element value of the i-th sample point in the training input bitstream, y i is the element value of the i-th sample point in the L-th training output bitstream, where i is a natural number greater than or equal to 1 and less than or equal to M; n is the number of sample points in the training input bitstream, and n is also the number of sample points in the L-th training output bitstream; Loss1 is the first loss value.

[0162] The second loss function is:

[0163]

[0164] Among them, e i It is the element value of the i-th sample point in the L-th convolution vector, g i is the element value of the i-th sample point in the L-th code table vector, where i is a natural number greater than or equal to 1 and less than or equal to M; n is the number of sample points in the L-th convolution vector, and n is also the number of sample points in the L-th code table vector; Loss2 is the second loss value.

[0165] Calculate the sum of the first loss value and the second loss value (Loss = Loss1 + Loss2), or perform a weighted summation of the first loss value and the second loss value (Loss = a × Loss1 + b × Loss2, where a and b are preset weight parameters), or calculate the average of the first loss value and the second loss value (Loss = (Loss1 + Loss2) / 2) to obtain the reconstruction loss value Loss.

[0166] The model parameters of the network units in the initial deep learning network of layer L are adjusted based on the reconstruction loss value using the gradient descent method, so that the adjusted initial deep learning network of layer L is transformed into the mature learning network of layer L.

[0167] The objective function of gradient descent is: This objective function adjusts the model parameters of the network units, where θ1 represents the model parameters to be adjusted, and Loss represents the reconstruction loss. The symbol is for partial differential, and θ2 represents the adjusted model parameters.

[0168] Therefore, by using gradient descent, the model parameters θ1 of one or more of the network units in the initial deep learning network of layer L, namely the encoder network, quantization network, and decoder network in the initial deep learning network of layer L, are adjusted to θ2, so as to adjust the initial deep learning network of layer L and transform it into the mature learning network of layer L.

[0169] S404: A multi-level deep learning network model is established by completely transforming the initial M-layer deep learning network into an M-layer mature deep learning network. This model includes an M-layer mature deep learning network and is used to receive the input bitstream and encoding rate requirement information. Based on the encoding rate requirement information, an N-layer mature deep learning network is determined. N is a natural number greater than or equal to 1 and less than or equal to M. The encoder network and the quantization network of the Nth mature deep learning network are called to encode the input bitstream to obtain the Nth encoded bitstream. The decoder network of the Nth mature deep learning network is called to decode the Nth encoded bitstream to obtain the output bitstream.

[0170] In this embodiment, by training each initial deep learning network layer, the existing mature deep learning networks can be used to provide encoder and decoder network support for the currently being trained initial deep learning network. Based on the Nth training output bitstream and training input bitstream obtained under this support, the training of the initial deep learning network can be completed by adjusting the model parameters of the currently being trained initial deep learning network. This improves the training efficiency of deep learning networks, avoids the situation of repeatedly training multiple deep learning networks, and greatly reduces the consumption of storage and computing resources.

[0171] Example 3:

[0172] Please see Figure 8This application provides a hierarchical encoding and decoding device 8 based on a multi-level deep learning network model. The multi-level deep learning network model includes M layers of mature deep learning networks. The M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1.

[0173] The hierarchical encoding and decoding device 8 includes: an encoding input module 81, an encoding processing module 82, and a decoding processing module 83;

[0174] The encoding input module 81 is used to receive the input bitstream and encoding bitrate requirement information, and to determine an N-layer mature deep learning network based on the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M.

[0175] The encoding input module 81 is connected to the encoding processing module 82. The encoding processing module 82 is used to call the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream.

[0176] The encoding processing module 82 is connected to the decoding processing module 83. The decoding processing module 83 is used to call the decoder network in the N-layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

[0177] Optional, encoding processing module 82, specifically used for:

[0178] The encoder networks, from the first layer to the Nth layer of the mature deep learning network, are sequentially called to reduce the dimensionality of the input bitstream and obtain the Nth encoded vector. The first layer of the mature deep learning network is the first mature deep learning network in the M layers of mature deep learning networks, and the Nth layer of the mature deep learning network is the last mature deep learning network in the N layers of mature deep learning networks.

[0179] The quantization network of the mature deep learning network at layer N is invoked to quantize the Nth encoded vector to obtain the Nth encoded bitstream.

[0180] Optionally, the quantization network includes: convolutional neural networks and residual quantization networks;

[0181] Encoding processing module 82 is specifically used for:

[0182] The quantization network of the Nth layer mature deep learning network is invoked to quantize the Nth encoding vector to obtain the Nth encoded bitstream, including:

[0183] The Nth encoding vector is convolved by the convolutional neural network of the Nth layer of a mature deep learning network to obtain the Nth convolutional vector; wherein the Nth convolutional vector has at least one Nth convolutional sub-vector.

[0184] The index corresponding to each Nth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Nth convolutional subvector are summarized to obtain the Nth encoded code stream. The code table is an index codeword table pre-set in the residual quantization network. The index codeword table records at least one level of index and the codeword corresponding to each index. The index is the quantization value used to describe the codeword. The codeword is a vector used to compare with the Nth convolutional subvector.

[0185] Optional, the decoding processing module 83 is specifically used for:

[0186] The Nth coded bitstream is dequantized using a preset code table to obtain the Nth code table vector.

[0187] The decoder network is called sequentially from the Nth layer mature deep learning network to the first layer mature deep learning network to perform dimensionality upscaling on the Nth code table vector to obtain the output bitstream.

[0188] Optional, the decoding processing module 83 is specifically used for:

[0189] Obtain the codeword in the code table for each index in the Nth encoded bitstream;

[0190] The Nth code table vector is obtained by summing the codewords corresponding to each index.

[0191] Example 4:

[0192] Please see Figure 9 This application provides a model training device 9 for a multi-level deep learning network model, the multi-level deep learning network model including an M-layer initial deep learning network; a model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1;

[0193] The model training device 9 includes: a training encoding module 91, a training decoding module 92, a training adjustment module 93, and a training integration module 94;

[0194] The training encoding module 91 is used to acquire the training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder and quantization networks of the Lth layer initial deep learning network to encode the training input bitstream to obtain the Lth encoded bitstream; where L refers to the number of layers in the initial deep learning network used to encode and decode the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the Lth layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M;

[0195] The training encoding module 91 is connected to the training decoding module 92. The training decoding module 92 is used to call the decoder network in the mature deep learning network of layer L-1 and the decoder network in the initial deep learning network of layer L to decode the L encoded bitstream to obtain the L training output bitstream.

[0196] The training adjustment module 92 is connected to the training decoding module 93. The training decoding module 93 is used to adjust the model parameters of the network units in the Lth layer initial deep learning network according to the vector loss, the training input bitstream, and the Lth training output bitstream, so that the Lth layer initial deep learning network is transformed into the Lth layer mature deep learning network. Here, the vector loss is the difference between the Lth convolution vector and the vector represented by the Lth encoded bitstream; the Lth convolution vector is the input vector used by the Lth layer initial deep learning network to generate the Lth encoded bitstream; the network unit includes one or more of the encoder network, quantization network, and decoder network; the model parameters include the weights and / or bias values ​​of the network units.

[0197] The training adjustment module 93 is connected to the training integration module 94. The training integration module 94 is used to completely transform the initial M-layer deep learning network into a multi-level deep learning network model containing an M-layer mature deep learning network. This multi-level deep learning network model, which includes an M-layer mature deep learning network, is used to receive the input bitstream and encoding rate requirement information, and determine an N-layer mature deep learning network based on the encoding rate requirement information. Here, N is a natural number greater than or equal to 1 and less than or equal to M. The encoder network and the quantization network of the Nth mature deep learning network are called to encode the input bitstream to obtain the Nth encoded bitstream. The decoder network of the Nth mature deep learning network is called to decode the Nth encoded bitstream to obtain the output bitstream.

[0198] Optionally, training encoding module 91, training decoding module 92, training adjustment module 93, and training integration module 94;

[0199] Optional, training encoding module 91, specifically used for:

[0200] The encoder networks from the first mature deep learning network to the (L-1)th mature deep learning network, and the encoder network of the Lth initial deep learning network are called sequentially to reduce the dimensionality of the training input bitstream to obtain the Lth training vector; where the first initial deep learning network is the first initial deep learning network in the M initial deep learning networks; the Lth initial deep learning network is the last initial deep learning network in the L initial deep learning networks.

[0201] The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L.

[0202] Optionally, the quantization network includes: convolutional neural networks and residual quantization networks;

[0203] Training encoding module 91, specifically used for:

[0204] The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L, including:

[0205] The Lth training vector is convolved by the convolutional neural network of the Lth initial deep learning network to obtain the Lth convolutional vector; wherein the Lth convolutional vector has at least one Lth convolutional sub-vector.

[0206] The index corresponding to each Lth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Lth convolutional subvector are summarized to obtain the Lth encoded bitstream. Here, the code table is an index codeword table pre-set in the residual quantization network, which records at least one level of index and the codeword corresponding to each index. The index is the quantization value used to describe the codeword. The codeword is a vector used to compare with the Nth convolutional subvector.

[0207] Optionally, train decoding module 92, specifically for:

[0208] The Lth coded bitstream is dequantized using a preset code table to obtain the Lth code table vector.

[0209] The decoder network of the initial deep learning network at layer L and the decoder network from the initial deep learning network at layer L-1 to the initial deep learning network at layer 1 are called sequentially to perform dimensionality upscaling on the Lth code table vector to obtain the Lth training output bitstream.

[0210] Optionally, train decoding module 92, specifically for:

[0211] Obtain the codeword in the code table for each index in the Lth encoded bitstream;

[0212] The Lth code table vector is obtained by summing the codewords corresponding to each index.

[0213] Optional, training tuning module 93, specifically used for:

[0214] The first loss function is called to obtain the first loss value based on the training input bitstream and the Lth training output bitstream; whereby the first loss value characterizes the difference between the training input bitstream and the Lth training output bitstream.

[0215] The second loss function is called to obtain the second loss value based on the vector loss; where the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to the preset code table and the Lth convolution vector.

[0216] The model parameters of the network units in the initial deep learning network of layer L are adjusted by gradient descent based on the first and second loss values, so that the adjusted initial deep learning network of layer L is transformed into the mature learning network of layer L.

[0217] Example 5:

[0218] Please see Figure 10 To achieve the above objectives, this application also provides a computer device 10, including: a processor 102 and a memory 101 communicatively connected to the processor 102; the memory stores computer execution instructions;

[0219] The processor executes computer execution instructions stored in memory 101 to implement the aforementioned multi-level encoding / decoding method and model training method. The components of the multi-level encoding / decoding device and model training device can be distributed across different computer devices. Computer device 10 can be a smartphone, tablet, laptop, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple application servers), etc. The computer device in this embodiment includes, but is not limited to, memory 101 and processor 102, which can be interconnected via a system bus. Figure 10 As shown. It should be noted that, Figure 10Only computer devices with components are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead. In this embodiment, memory 101 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, memory 101 may be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, memory 101 may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, flash card, etc. Of course, memory 101 may also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 101 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the multi-level encoding / decoding device and model training device in Embodiment 3. Furthermore, the memory 101 can also be used to temporarily store various types of data that have been output or will be output. In some embodiments, the processor 102 can be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 102 is typically used to control the overall operation of the computer device. In this embodiment, the processor 102 is used to run the program code stored in the memory 101 or process data, for example, to run the multi-level encoding / decoding device and model training device to implement the multi-level encoding / decoding method and model training method of the above embodiments.

[0220] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. It should be understood that the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disk, etc.

[0221] To achieve the above objectives, this application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores computer-executable instructions. When the program is executed by the processor 102, it implements the corresponding functions. The computer-readable storage medium of this embodiment is used to store computer-executable instructions for implementing multi-level encoding / decoding methods and model training methods. When executed by the processor 102, it implements the multi-level encoding / decoding methods and model training methods of the above embodiments.

[0222] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0223] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0224] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described multi-level encoding / decoding method and model training method.

[0225] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0226] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0227] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A hierarchical encoding and decoding method based on a multi-level deep learning network model, characterized in that, The multi-level deep learning network model includes M layers of mature deep learning networks. The M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1. The hierarchical encoding and decoding method includes: Receive the input bitstream and encoding rate requirement information, and determine an N-layer mature deep learning network based on the encoding rate requirement information; where N is a natural number greater than or equal to 1 and less than or equal to M; The encoder network and the quantization network of the Nth layer mature deep learning network are invoked to encode the input bitstream to obtain the Nth encoded bitstream; The decoder network in the Nth layer of the mature deep learning network is invoked to decode the Nth encoded bitstream to obtain the output bitstream.

2. The hierarchical encoding and decoding method according to claim 1, characterized in that, The input bitstream is encoded by calling the encoder network and the quantization network of the Nth layer mature deep learning network to obtain the Nth encoded bitstream, including: The encoder networks, from the first layer of the mature deep learning network to the Nth layer of the mature deep learning network, are sequentially invoked to perform dimensionality reduction processing on the input bitstream to obtain the Nth encoding vector; wherein, the first layer of the mature deep learning network is the first mature deep learning network in the M layers of mature deep learning networks; and the Nth layer of the mature deep learning network is the last mature deep learning network in the N layers of mature deep learning networks. The quantization network of the mature deep learning network at layer N is invoked to quantize the Nth encoding vector to obtain the Nth encoded bitstream.

3. The hierarchical encoding and decoding method according to claim 2, characterized in that, The quantization network includes: a convolutional neural network and a residual quantization network; The Nth layer mature deep learning network is invoked to perform quantization processing on the Nth encoding vector to obtain the Nth encoded bitstream, including: The Nth encoding vector is convolved by the convolutional neural network of the mature deep learning network at the Nth layer to obtain the Nth convolutional vector; wherein the Nth convolutional vector has at least one Nth convolutional sub-vector. The index corresponding to each of the Nth convolutional sub-vectors is obtained from the code table of the residual quantization network, and the indexes corresponding to each of the Nth convolutional sub-vectors are summarized to obtain the Nth encoded code stream; the code table is an index codeword table pre-set in the residual quantization network, the index codeword table records at least one layer of indexes and codewords corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used for comparison with the Nth convolutional sub-vector.

4. The hierarchical encoding and decoding method according to claim 1, characterized in that, Calling the decoder network in the Nth layer of the mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream includes: The Nth encoded bitstream is dequantized using a preset code table to obtain the Nth code table vector. The decoder network, from the Nth layer mature deep learning network to the first layer mature deep learning network, is called sequentially to perform dimensionality upscaling on the Nth code table vector to obtain the output bitstream.

5. The hierarchical encoding and decoding method according to claim 4, characterized in that, The Nth encoded bitstream is dequantized using a preset code table to obtain the Nth code table vector, which includes: Obtain the codeword of each index in the code table in the Nth encoded bitstream; The Nth code table vector is obtained by summing the codewords corresponding to each index.

6. A method for training a multi-level deep learning network model, characterized in that, The multi-level deep learning network model includes an initial deep learning network of M layers; the model training method is used to train the initial deep learning network of M layers into a mature deep learning network of M layers; the mature deep learning network of M layers is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1. The model training method includes: The training input bitstream is obtained, and the encoder network in the (L-1)th layer mature deep learning network, as well as the encoder and quantization networks of the Lth layer initial deep learning network, are invoked to encode the training input bitstream to obtain the Lth encoded bitstream. Here, L refers to the number of layers in the initial deep learning network used for encoding and decoding the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M. The decoder network in the mature deep learning network of layer L-1 and the decoder network in the initial deep learning network of layer L are invoked to decode the Lth encoded bitstream to obtain the Lth training output bitstream; Based on the vector loss, the training input bitstream, and the Lth training output bitstream, the model parameters of the network units in the Lth initial deep learning network are adjusted to transform the Lth initial deep learning network into an Lth mature deep learning network. The vector loss is the difference between the Lth convolutional vector and the vector represented by the Lth encoded bitstream; the Lth convolutional vector is the input vector used by the Lth initial deep learning network to generate the Lth encoded bitstream; the network units include one or more of encoder networks, quantization networks, and decoder networks; the model parameters include the weights and / or biases of the network units. A multi-level deep learning network model is established by completely transforming the initial deep learning network of M layers into a mature deep learning network of M layers. This model includes a mature deep learning network of M layers. The multi-level deep learning network model includes a mature deep learning network of M layers and is used to receive the input bitstream and encoding rate requirement information. Based on the encoding rate requirement, an N-layer mature deep learning network is determined. N is a natural number greater than or equal to 1 and less than or equal to M. The encoder network of the N-layer mature deep learning network and the quantization network of the Nth-layer mature deep learning network are called to encode the input bitstream to obtain the Nth encoded bitstream. The decoder network of the N-layer mature deep learning network is called to decode the Nth encoded bitstream to obtain the output bitstream.

7. The model training method according to claim 6, characterized in that, The encoder network in the L-1 layer mature deep learning network, as well as the encoder and quantization networks of the Lth layer initial deep learning network, are invoked to encode the training input bitstream to obtain the Lth encoded bitstream, including: The encoder networks from the first mature deep learning network to the (L-1)th mature deep learning network, and the encoder network of the Lth initial deep learning network are sequentially invoked to perform dimensionality reduction processing on the training input bitstream to obtain the Lth training vector; wherein, the first initial deep learning network is the first initial deep learning network in the M layers of initial deep learning networks; the Lth initial deep learning network is the last initial deep learning network in the L layers of initial deep learning networks; The quantization network of the initial deep learning network at layer L is invoked to quantize the training vector at layer L to obtain the encoded bitstream at layer L.

8. The model training method according to claim 7, characterized in that, The quantization network includes: a convolutional neural network and a residual quantization network; The process involves calling the quantization network of the initial deep learning network at layer L to quantize the training vectors at layer L to obtain the encoded bitstream at layer L, including: The convolutional neural network of the Lth initial deep learning network is invoked to perform convolution processing on the Lth training vector to obtain the Lth convolution vector; wherein the Lth convolution vector has at least one Lth convolution sub-vector. The index corresponding to each Lth convolutional subvector is obtained from the code table of the residual quantization network, and the indexes corresponding to each Lth convolutional subvector are summarized to obtain the Lth encoded code stream; wherein, the code table is an index codeword table pre-set in the residual quantization network, the index codeword table records at least one layer of indexes and codewords corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used for comparison with the Nth convolutional subvector.

9. The model training method according to claim 6, characterized in that, The Lth encoded bitstream is decoded by calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the Lth layer initial deep learning network to obtain the Lth training output bitstream, including: The Lth encoded bitstream is dequantized using a preset code table to obtain the Lth code table vector. The decoder network of the initial deep learning network at layer L and the decoder network from the initial deep learning network at layer L-1 to the initial deep learning network at layer 1 are sequentially called to perform dimensionality upscaling on the Lth code table vector to obtain the Lth training output bitstream.

10. The model training method according to claim 9, characterized in that, The Lth encoded bitstream is dequantized using a preset code table to obtain the Lth code table vector, which includes: Obtain the codeword of each index in the code table in the Lth encoded bitstream; The Lth code table vector is obtained by summing the codewords corresponding to each index.

11. The model training method according to claim 6, characterized in that, Based on the vector loss, the training input bitstream, and the Lth training output bitstream, the model parameters of the network units in the Lth initial deep learning network are adjusted to transform the Lth initial deep learning network into the Lth mature deep learning network, including: The first loss function is called to obtain the first loss value based on the training input bitstream and the Lth training output bitstream; wherein, the first loss value represents the difference between the training input bitstream and the Lth training output bitstream; The second loss function is called to obtain the second loss value based on the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to the preset code table and the Lth convolution vector. The model parameters of the network units in the initial deep learning network of the Lth layer are adjusted by gradient descent based on the first loss value and the second loss value, so that the adjusted initial deep learning network of the Lth layer is transformed into the mature learning network of the Lth layer.

12. A hierarchical encoding and decoding device based on a multi-level deep learning network model, characterized in that, The multi-level deep learning network model includes M layers of mature deep learning networks. The M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1. The hierarchical encoding and decoding device includes: an encoding input module, an encoding processing module, and a decoding processing module; The encoding input module is used to receive the input bitstream and encoding bitrate requirement information, and to determine an N-layer mature deep learning network based on the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M. The encoding input module is connected to the encoding processing module. The encoding processing module is used to call the encoder network and the quantization network of the Nth layer mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream. The encoding processing module is connected to the decoding processing module. The decoding processing module is used to call the decoder network in the N-layer mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

13. A model training device for a multi-level deep learning network model, characterized in that, The multi-level deep learning network model includes an M-layer initial deep learning network; the model training device is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; where M is a natural number greater than 1. The model training device includes: a training encoding module, a training decoding module, a training adjustment module, and a training integration module; The training encoding module is used to acquire the training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder and quantization networks of the Lth layer initial deep learning network to encode the training input bitstream to obtain the Lth encoded bitstream; where L refers to the number of layers in the initial deep learning network used to encode and decode the input bitstream; the Lth layer initial deep learning network is the last initial deep learning network in the Lth layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M; The training encoding module is connected to the training decoding module. The training decoding module is used to call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream. The training adjustment module is connected to the training decoding module. The training decoding module is used to adjust the model parameters of the network units in the Lth layer initial deep learning network according to the vector loss, the training input bitstream, and the Lth training output bitstream, so that the Lth layer initial deep learning network is transformed into the Lth layer mature deep learning network. The vector loss is the difference between the Lth convolutional vector and the vector represented by the Lth encoded bitstream; the Lth convolutional vector is the input vector used by the Lth layer initial deep learning network to generate the Lth encoded bitstream; the network unit includes one or more of an encoder network, a quantization network, and a decoder network; the model parameters include the weights and / or biases of the network units. The training adjustment module is connected to the training integration module. The training integration module is used to completely transform the initial M-layer deep learning network into a multi-level deep learning network model containing M mature deep learning networks. This multi-level deep learning network model includes M mature deep learning networks. The multi-level deep learning network model includes M mature deep learning networks and is used to receive input bitstream and encoding rate requirement information. Based on the encoding rate requirement information, it determines an N-layer mature deep learning network. N is a natural number greater than or equal to 1 and less than or equal to M. The module calls the encoder network and the quantization network of the Nth mature deep learning network to encode the input bitstream to obtain the Nth encoded bitstream. Finally, it calls the decoder network of the Nth mature deep learning network to decode the Nth encoded bitstream to obtain the output bitstream.

14. A computer device, characterized in that, include: A processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 11.

Images