Hierarchical coding and decoding method, model training method, device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and provides a hierarchical coding and decoding method, a model training method, devices, equipment and a medium, and the method comprises the steps: receiving an input code stream and coding rate requirement information, and determining N layers of mature deep learning networks according to the coding rate requirement information; n is a natural number greater than or equal to 1 and less than or equal to M; calling an encoder network in the N layers of mature deep learning networks and a quantization network of the Nth layer of mature deep learning network, and encoding the input code stream to obtain an Nth encoded code stream; and calling a decoder network in the N layers of mature deep learning networks, and decoding the Nth encoding code stream to obtain an output code stream. The utilization rate and training efficiency of the deep learning network are improved, and consumption of storage resources and computing resources is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a hierarchical encoding and decoding method and a model training method, device, equipment and medium. Background Art

[0002] In typical encoding and decoding processes such as audio and video streams, it is usually necessary to prepare multiple bit rates according to different transmission / storage conditions. The bit rate refers to the compression bit rate used to compress the input stream. When conditions are good, a compression algorithm with a higher bit rate is used, and when conditions are bad, a compression algorithm with a lower bit rate is used.

[0003] The current compression algorithms based on deep learning all adopt the method of retraining at different bit rates, that is, designing and training a deep learning network according to one bit rate, and training several deep learning networks for several bit rates. However, the inventors found that this approach not only results in low utilization of each deep learning network due to the lack of correlation between the multiple deep learning networks, but also wastes storage and computing resources due to the need to repeatedly train multiple deep learning networks. Summary of the invention

[0004] The present application provides a hierarchical encoding and decoding method and a model training method, device, equipment and medium to solve the problem.

[0005] In a first aspect, the present application provides a hierarchical encoding and decoding method based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1;

[0006] The hierarchical coding and decoding method comprises:

[0007] Receiving an input bitstream and encoding bitrate requirement information, and determining an N-layer mature deep learning network according to the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M;

[0008] Calling the encoder network in the N-layer mature deep learning network and the quantization network in the N-th layer of the mature deep learning network to encode the input bitstream to obtain an N-th encoded bitstream;

[0009] Calling the decoder network in the N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream.

[0010] In the above scheme, calling the encoder network in the N-layer mature deep learning network and the quantization network in the N-th layer mature deep learning network to encode the input bitstream to obtain the N-th encoded bitstream includes:

[0011] Sequentially calling encoder networks from the first layer of mature deep learning networks to the Nth layer of mature deep learning networks, performing dimensionality reduction processing on the input code stream to obtain the Nth encoding vector; wherein the first layer of mature deep learning networks is the mature deep learning network that is at the first position among the M layers of mature deep learning networks; the Nth layer of mature deep learning networks is the mature deep learning network that is at the last position among the N layers of mature deep learning networks;

[0012] The quantization network of the N-th layer mature deep learning network is called to quantize the N-th encoding vector to obtain the N-th encoding bit stream.

[0013] In the above scheme, the quantization network includes: a convolutional neural network and a residual quantization network;

[0014] Calling the quantization network of the N-th layer of mature deep learning network to quantize the N-th encoding vector to obtain the N-th encoding bitstream, including:

[0015] Calling the convolutional neural network of the N-th layer of mature deep learning network to perform convolution processing on the N-th encoding vector to obtain an N-th convolution vector; wherein the N-th convolution vector has at least one layer of N-th convolution sub-vectors;

[0016] The index corresponding to each of the N-th convolution sub-vectors is obtained from the code table of the residual quantization network, and the index corresponding to each of the N-th convolution sub-vectors is summarized to obtain the N-th encoded code stream; the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of indexes and a codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used to compare with the N-th convolution sub-vector.

[0017] In the above scheme, calling the decoder network in the N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream includes:

[0018] Performing inverse quantization processing on the Nth encoded bit stream using a preset code table to obtain an Nth code table vector;

[0019] The decoder networks from the N-th layer of mature deep learning network to the first layer of mature deep learning network are called in sequence, and the N-th code table vector is processed by dimension upgrading to obtain an output code stream.

[0020] In the above scheme, the Nth coded bit stream is dequantized by using a preset code table to obtain an Nth code table vector, including:

[0021] Obtaining a codeword of each index in the code table in the Nth encoded code stream;

[0022] The codewords corresponding to each index are summarized to obtain the Nth code table vector.

[0023] In a second aspect, the present application provides a model training method for a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of initial deep learning networks; the model training method is used to train the M layers of initial deep learning networks into M layers of mature deep learning networks; the M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0024] The model training method comprises:

[0025] Obtain a training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, encode the training input bitstream to obtain the L-th encoded bitstream; wherein, L layers refer to the number of layers of the initial deep learning network used for encoding and decoding the input bitstream; the L-th layer initial deep learning network is the last initial deep learning network in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M;

[0026] Calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream;

[0027] According to the vector loss, the training input code stream and the Lth training output code stream, the model parameters of the network unit in the Lth layer initial deep learning network are adjusted to convert the Lth layer initial deep learning network into the Lth layer mature deep learning network; wherein the vector loss is the difference between the Lth convolution vector and the vector represented by the Lth encoding code stream; the Lth convolution vector is the input vector of the Lth layer initial deep learning network for generating the Lth encoding code stream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network unit;

[0028] The M-layer initial deep learning network is completely converted into a multi-level deep learning network model of an M-layer mature deep learning network, which is set as a multi-level deep learning network model including an M-layer mature deep learning network; the multi-level deep learning network model including an M-layer mature deep learning network is used to receive an input bit stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M; the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network are called to encode the input bit stream to obtain an N-th encoded bit stream; the decoder network in the N-layer mature deep learning network is called to decode the N-th encoded bit stream to obtain an output bit stream.

[0029] In the above scheme, the encoder network in the L-1 layer mature deep learning network, as well as the encoder network and quantization network of the L-th layer initial deep learning network are called to encode the training input bitstream to obtain the L-th encoded bitstream, including:

[0030] Sequentially calling the encoder networks from the first layer of mature deep learning networks to the L-1th layer of mature deep learning networks, and the encoder network of the Lth layer of initial deep learning networks, and performing dimensionality reduction processing on the training input code stream to obtain the Lth training vector; wherein the first layer of initial deep learning network is the initial deep learning network that is at the first position among the M layers of initial deep learning networks; the Lth layer of initial deep learning network is the initial deep learning network that is at the last position among the L layers of initial deep learning networks;

[0031] The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain an L-th encoding bit stream.

[0032] In the above scheme, the quantization network includes: a convolutional neural network and a residual quantization network;

[0033] Calling the quantization network of the L-th layer initial deep learning network to quantize the L-th training vector to obtain the L-th encoding bitstream, including:

[0034] Calling the convolutional neural network of the Lth initial deep learning network to perform convolution processing on the Lth training vector to obtain an Lth convolution vector; wherein the Lth convolution vector has at least one layer of Lth convolution sub-vectors;

[0035] Obtain an index corresponding to each of the L-th convolution sub-vectors from the code table of the residual quantization network, and summarize the index corresponding to each of the L-th convolution sub-vectors to obtain an L-th encoded code stream; wherein the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of indexes and a codeword corresponding to each index; the index is a quantization value used to describe the codeword; and the codeword is a vector used to compare with the N-th convolution sub-vector.

[0036] In the above scheme, calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream includes:

[0037] Performing inverse quantization processing on the Lth encoded bit stream using a preset code table to obtain an Lth code table vector;

[0038] The decoder network of the L-th layer initial deep learning network and the decoder network from the L-1-th layer initial deep learning network to the first layer initial deep learning network are called in sequence, and the L-th code table vector is dimensionally upgraded to obtain the L-th training output code stream.

[0039] In the above scheme, the Lth encoded bit stream is dequantized by using a preset code table to obtain an Lth code table vector, including:

[0040] Obtaining a codeword of each index in the code table in the Lth encoded code stream;

[0041] The codewords corresponding to each index are summarized to obtain the Lth code table vector.

[0042] In the above scheme, according to the vector loss, the training input code stream and the L-th training output code stream, the model parameters of the network unit in the L-th layer initial deep learning network are adjusted to convert the L-th layer initial deep learning network into the L-th layer mature deep learning network, including:

[0043] Calling a first loss function to obtain a first loss value according to the training input code stream and the Lth training output code stream; wherein the first loss value represents the difference between the training input code stream and the Lth training output code stream;

[0044] Calling a second loss function to obtain a second loss value according to the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to a preset code table and the Lth convolution vector;

[0045] The model parameters of the network units in the L-th layer initial deep learning network are adjusted according to the first loss value and the second loss value by the gradient descent method, so that the adjusted L-th layer initial deep learning network is converted into the L-th layer mature learning network.

[0046] In a third aspect, the present application provides a hierarchical encoding and decoding device based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1;

[0047] The hierarchical coding and decoding device comprises: a coding input module, a coding processing module and a decoding processing module;

[0048] The encoding input module is used to receive an input code stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M;

[0049] The encoding input module is connected to the encoding processing module, and the encoding processing module is used to call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input code stream to obtain the N-th encoded code stream;

[0050] The encoding processing module is connected to the decoding processing module, and the decoding processing module is used to call the decoder network in the N-layer mature deep learning network to decode the N-th encoded code stream to obtain an output code stream.

[0051] In a fourth aspect, the present application provides a model training device for a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of initial deep learning networks; the model training method is used to train the M layers of initial deep learning networks into M layers of mature deep learning networks; the M layers of mature deep learning networks are used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0052] The model training device comprises: a training encoding module, a training decoding module, a training adjustment module and a training integration module;

[0053] The training encoding module is used to obtain a training input code stream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, and encode the training input code stream to obtain the L-th encoded code stream; wherein the L layer refers to the number of layers of the initial deep learning network used for encoding and decoding the input code stream; the L-th layer initial deep learning network is the initial deep learning network at the last position in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M;

[0054] The training encoding module is connected to the training decoding module, and the training decoding module is used to call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded code stream to obtain the L-th training output code stream;

[0055] The training adjustment module is connected to the training decoding module, and the training decoding module is used to adjust the model parameters of the network unit in the L-th layer initial deep learning network according to the vector loss, the training input code stream and the L-th training output code stream, so that the L-th layer initial deep learning network is converted into an L-th layer mature deep learning network; wherein the vector loss is the difference between the L-th convolution vector and the vector represented by the L-th encoding code stream; the L-th convolution vector is the input vector of the L-th layer initial deep learning network for generating the L-th encoding code stream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network unit;

[0056] The training adjustment module is connected to the training integration module, and the training integration module is used to completely convert the M-layer initial deep learning network into a multi-level deep learning network model of M-layer mature deep learning network, which is set to a multi-level deep learning network model including M-layer mature deep learning network; the multi-level deep learning network model including M-layer mature deep learning network is used to receive input bit stream and encoding bit rate requirement information, and determine N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M; call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bit stream to obtain the N-th encoded bit stream; call the decoder network in the N-layer mature deep learning network to decode the N-th encoded bit stream to obtain the output bit stream.

[0057] In a fifth aspect, the present application provides a computer device, comprising: a processor and a memory communicatively connected to the processor;

[0058] The memory stores computer-executable instructions;

[0059] The processor executes the computer-executable instructions stored in the memory to implement the method as claimed in claim 1.

[0060] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above-mentioned method when executed by a processor.

[0061] In a seventh aspect, the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0062] The present application provides a hierarchical encoding and decoding method and model training method, apparatus, device and medium, which determine N layers of mature deep learning networks according to the encoding bit rate requirement information, call the encoder network in the N layers of the mature deep learning networks and the quantization network of the N-th layer of mature deep learning networks, encode the input code stream to obtain the N-th encoded code stream; call the decoder network in the N layers of the mature deep learning networks, decode the N-th encoded code stream to obtain the output code stream, so that for an input code stream of a certain bit rate, the encoder network and the decoder network in the multi-layer mature deep learning network can be called to encode and decode it, and the quantization network of the N-th layer of mature deep learning network corresponding to the bit rate is called to encode the input code stream, so that multiple mature deep learning networks are called to encode and decode the input code stream of a certain bit rate, thereby improving the utilization rate of each layer of deep learning networks.

[0063] At the same time, for the training of each layer of the initial deep learning network, the previously trained mature deep learning network can be used to provide encoder network and decoder network support for the initial deep learning network currently being trained, and then based on the Nth training output bit stream and training input bit stream obtained under this support, only the model parameters of the initial deep learning network currently being trained need to be adjusted to complete the training of the initial deep learning network, thereby improving the training efficiency of the deep learning network, avoiding the repeated training of multiple deep learning networks, and greatly reducing the consumption of storage resources and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0065] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0066] Figure 2A flowchart of Example 1 of a hierarchical encoding and decoding method provided in an embodiment of the present application;

[0067] Figure 3 In Embodiment 1 of a hierarchical encoding and decoding method provided in an embodiment of the present application, taking N as 3 as an example, a schematic block diagram of a structure in which an input bit stream is encoded and decoded to obtain an output bit stream;

[0068] Figure 4 A flowchart of Example 2 of a model training method provided in an embodiment of the present application;

[0069] Figure 5 In Example 2 of a model training method provided in an embodiment of the present application, a structural schematic diagram of training a first-layer initial deep learning network by taking training of the first layer as an example;

[0070] Figure 6 In Example 2 of a model training method provided in an embodiment of the present application, a structural schematic diagram of training a second-layer initial deep learning network by taking training of the second layer as an example;

[0071] Figure 7 In Example 2 of a model training method provided in an embodiment of the present application, a structural schematic block diagram of training a third-layer initial deep learning network by taking the training of the third layer as an example;

[0072] Figure 8 A schematic diagram of program modules of Embodiment 3 of a hierarchical encoding and decoding device provided by the present invention;

[0073] Fig. 9 A schematic diagram of a program module of Embodiment 4 of a model training device provided by the present invention;

[0074] Fig.10 The figure is a schematic diagram of the hardware structure of the computer device in the computer device of the present invention.

[0075] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0077] See also Figure 1 , the specific application scenarios of this application are:

[0078] A server 11 running a hierarchical encoding and decoding method and a model training method is installed with a multi-level deep learning network model 12, which includes an M-layer mature deep learning network 13. The M-layer mature deep learning network 13 is used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; the mature deep learning network 13 includes: an encoder network 131, a quantization network 132, and a decoder network 133; the quantization network 132 includes a convolutional neural network 1321 and a residual quantization network 1322.

[0079] The server 11 receives the input bitstream and the encoding bitrate requirement information, and determines an N-layer mature deep learning network according to the encoding bitrate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M;

[0080] The server 11 calls the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bitstream to obtain the N-th encoded bitstream;

[0081] The server 11 calls the decoder network in the N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream.

[0082] In the present application, the encoder network and the quantization network are used to compress data, that is, to encode the input code stream (for example, pictures, videos, audio and other signals or files) into a coded code stream (a data stream that has been compressed); the decoder network is used to decompress the coded code stream (data stream) into an output code stream (for example, pictures, videos, audio and other signals or files).

[0083] Between the quantization network and the decoder network, there is a communication connection for data transmission. The two ends of the communication connection are respectively connected to the first terminal and the second terminal. The first terminal can call the encoder network and the quantization network, and the second terminal can call the decoder network. The first terminal sends the encoded code stream to the second terminal through the communication connection to achieve compressed output of the compressed input code stream, and decompress the compressed input code stream to obtain the output code stream.

[0084] The technical solution of the present application and how the technical solution of the present application solves the problems of the prior art are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0085] Embodiment 1:

[0086] See also Figure 2 The present application provides a hierarchical encoding and decoding method based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1;

[0087] The hierarchical coding and decoding method includes:

[0088] S201: Receive input bit stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M.

[0089] In this step, the encoding bit rate requirement information records the compression bit rate required for data compression of the input code stream, wherein the compression bit rate corresponds to the bit rate of the N-th layer of the deep learning network in the M-layer mature deep learning network, and then by determining the N-layer mature deep learning network, the number of layers used to provide encoding and decoding processing servers for the input code stream is determined. The bit rate of the N-th layer of the deep learning network refers to the compression bit rate used by the N-th layer of the deep learning network for data compression.

[0090] S202: Calling the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bit stream to obtain the N-th encoded bit stream.

[0091] In this step, by using a multi-layer mature deep learning network to encode the input bit stream, the technical effect of processing input bit streams of various bit rates based on a multi-level network is achieved, and the utilization rate of the multi-level deep learning network model is improved.

[0092] In a preferred embodiment, calling the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bitstream to obtain the N-th encoded bitstream includes:

[0093] The encoder networks from the first layer of mature deep learning network to the Nth layer of mature deep learning network are called in sequence to perform dimension reduction processing on the input code stream to obtain the Nth encoding vector; wherein the first layer of mature deep learning network is the mature deep learning network at the first position among the M layers of mature deep learning networks; the Nth layer of mature deep learning network is the mature deep learning network at the last position among the N layers of mature deep learning networks;

[0094] The quantization network of the N-th layer mature deep learning network is called to quantize the N-th encoding vector to obtain the N-th encoding bitstream.

[0095] The input code stream is processed for dimensionality reduction through an N-layer encoder network, so that the N-th code vector can be adapted to the quantization network. The N-th code vector is quantized through the quantization network, ensuring the accuracy of quantization processing of input code streams with different bit rates.

[0096] Exemplarily, taking N as 3 as an example, it represents the need for an encoder network and a decoder network of three layers of mature deep learning networks, and a quantization network in the third layer of mature deep learning networks to process the input code stream.

[0097] The encoder network 1 of the first-layer mature deep learning network, the encoder network 2 of the second-layer mature deep learning network, and the encoder network 3 of the third-layer mature deep learning network are called in sequence to perform dimensionality reduction processing on the input code stream to obtain the corresponding third encoding vector; then the third encoding vector is quantized through the quantization network of the third-layer mature deep learning network to obtain the third encoding code stream.

[0098] Furthermore, the quantization network includes: a convolutional neural network and a residual quantization network;

[0099] Call the quantization network of the Nth layer of mature deep learning network to quantize the Nth encoding vector to obtain the Nth encoding bitstream, including:

[0100] Calling the convolutional neural network of the Nth layer of mature deep learning network to perform convolution processing on the Nth encoding vector to obtain the Nth convolution vector; wherein the Nth convolution vector has at least one layer of Nth convolution sub-vectors;

[0101] The index corresponding to each N-th convolution sub-vector is obtained from the code table of the residual quantization network, and the index corresponding to each N-th convolution sub-vector is summarized to obtain the N-th encoded code stream; the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of index and the codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a multidimensional vector used to compare with the N-th convolution sub-vector.

[0102] First, the Nth coding vector is convolved so that the length of the output convolution vector is the specified length. Then, the index of each convolution sub-vector is determined for the convolution vector of the specified length according to the code table in the residual quantization network to quantize the convolution vector. The resulting coded bit stream is the data based on the code table used to represent the convolution vector.

[0103] In the example, the dimension of the residual quantization code table of each layer is (1024, 64), that is, the code table consists of 1024 vectors of length 64.

[0104] For example, see Figure 3, taking N as 3 as an example, its representation requires an encoder network and a decoder network of three layers of mature deep learning networks, as well as a quantization network in the third layer of mature deep learning networks to process the input code stream;

[0105] Assuming that the input bitstream is 96,000 samples in length, the input bitstream will be represented as (1,96000). At the same time, assuming that the time domain dimension reduction multiples of the encoder network of the first layer of mature deep learning network, the second layer of mature deep learning network, the third layer of mature deep learning network and the fourth layer of mature deep learning network are 3, 4, 5, and 5 respectively. The input bitstream (1,96000) passes through the first layer of mature deep learning network, the second layer of mature deep learning network and the third layer of mature deep learning network. The encoder networks 1, 2, and 3 of the network are reduced in dimension to a multi-dimensional vector of (64, 32000), a multi-dimensional vector of (128, 8000), and a multi-dimensional vector of (256, 1600), respectively; at this time, the third encoded vector is (256, 1600), and the third encoded vector is transformed into a third convolution vector of (64, 1600) after convolution processing by the convolutional neural network CNN3 of the third layer of mature deep learning network. The third convolution vector represents 1600 multi-dimensional vectors of length 64.

[0106] From the code table of the residual quantization network 3 of the third layer of the mature deep learning network, the index corresponding to each third convolution sub-vector is obtained, wherein the index corresponding to each third convolution sub-vector refers to the index of the codeword in the code table that is closest to the third convolution sub-vector; therefore, a total of 1600 10-bit indexes are obtained, wherein each index in the code table consists of 10 bits (10 bits cover the index range of the code table). Then, a 1600x10-bit third encoding code stream is generated, which is the compressed code stream.

[0107] S203: Calling a decoder network in an N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream.

[0108] By determining N layers of mature deep learning networks according to the encoding bit rate requirement information, calling the encoder network in the N layers of mature deep learning networks and the quantization network of the Nth layer of mature deep learning networks, encoding the input bit stream to obtain the Nth encoded bit stream; calling the decoder network in the N layers of mature deep learning networks, decoding the Nth encoded bit stream to obtain the output bit stream, so that for an input bit stream of a certain bit rate, the encoder network and the decoder network in the multi-layer mature deep learning network can be called to encode and decode it, and the quantization network of the Nth layer of mature deep learning network corresponding to the bit rate is called to encode the input bit stream, so that calling multiple mature deep learning networks to encode and decode the input bit stream of a certain bit rate is realized, and the utilization rate of each layer of deep learning networks is improved.

[0109] In a preferred embodiment, calling a decoder network in an N-layer mature deep learning network to decode the Nth encoded bitstream to obtain an output bitstream includes:

[0110] Dequantizing the Nth coded bitstream using a preset code table to obtain an Nth code table vector;

[0111] The decoder networks from the N-th layer of mature deep learning network to the first layer of mature deep learning network are called in sequence, and the N-th code table vector is dimensionally upgraded to obtain the output code stream.

[0112] By calling the decoder network of N-layer mature deep learning networks to perform dimensionality upgrade processing on the N-th code table vector, the participation of each mature deep learning network is improved, thereby improving the utilization rate of the multi-level deep learning network model.

[0113] Exemplarily, based on the above example, the third encoded code stream is inversely quantized by the quantization network 3 of the third mature neural network to obtain a third code table vector, and the encoder network 3, 3 in the third layer of the mature neural network, the encoder network 2 in the second layer of the mature neural network and the encoder network 1 in the first layer of the mature neural network are called in sequence to perform dimensionality increase processing on the third code table vector to obtain an output code stream.

[0114] Furthermore, the Nth coded bitstream is dequantized by using a preset code table to obtain an Nth code table vector, including:

[0115] Obtain the codeword of each index in the code table in the Nth encoded code stream;

[0116] The codewords corresponding to each index are summarized to obtain the Nth code table vector.

[0117] For example, based on the above example, please refer to Figure 3 , from the code table of the residual quantization network 3, 1600 vectors (64, 1600) are obtained that are related to the third encoded code stream 1600*10bit, so as to perform inverse quantization processing on the third encoded code stream to obtain a third code table vector (64, 1600), and the third code table vector (64, 1600) is successively processed through the decoder network 3 of the third layer of mature deep learning network, the decoder network 2 of the second layer of mature deep learning network, and the decoder network 1 of the first layer of mature deep learning network, and the third code table vector (64, 1600) is successively upgraded to obtain a multi-dimensional vector of (128, 8000), a multi-dimensional vector of (64, 32000) and a multi-dimensional vector of (1, 96000), and the vector (1, 96000) is used as the output code stream.

[0118] Embodiment 2:

[0119] See also Figure 4The present application provides a model training method for a multi-level deep learning network model, wherein the multi-level deep learning network model includes an M-layer initial deep learning network; the model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0120] Model training methods include:

[0121] S401: Obtain a training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, encode the training input bitstream to obtain the L-th encoded bitstream; wherein, L layers refer to the number of layers of the initial deep learning network used to encode and decode the input bitstream; the L-th layer initial deep learning network is the last initial deep learning network in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M.

[0122] By encoding the training input code based on the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, the training results based on the L-1 layer mature deep learning network are achieved, the technical effect of training the L-th layer initial deep learning network is achieved, and the training efficiency is improved.

[0123] S402: Call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream.

[0124] By combining the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network, the training results based on the L-1 layer mature deep learning network are achieved, and the technical effect of training the L-th layer initial deep learning network is achieved, thereby improving the training efficiency.

[0125] In a preferred embodiment, calling the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, encoding the training input bitstream to obtain the L-th encoded bitstream, including:

[0126] Sequentially call the encoder networks from the first mature deep learning network to the L-1 mature deep learning network, and the encoder network of the L initial deep learning network, and perform dimension reduction processing on the training input code stream to obtain the L training vector; wherein the first initial deep learning network is the initial deep learning network at the first position in the M initial deep learning networks; the L initial deep learning network is the initial deep learning network at the last position in the L initial deep learning networks;

[0127] The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain the L-th encoding bit stream.

[0128] Furthermore, the quantization network includes: a convolutional neural network and a residual quantization network;

[0129] The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain the L-th encoding bit stream, including:

[0130] Calling the convolutional neural network of the Lth initial deep learning network to perform convolution processing on the Lth training vector to obtain an Lth convolution vector; wherein the Lth convolution vector has at least one layer of Lth convolution sub-vectors;

[0131] The index corresponding to each L-th convolution sub-vector is obtained from the code table of the residual quantization network, and the index corresponding to each L-th convolution sub-vector is summarized to obtain the L-th encoded code stream; wherein the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of index and the codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a multidimensional vector used to compare with the N-th convolution sub-vector.

[0132] Exemplarily, a mono audio with a sampling rate of 48kHz is used as the training input code stream, and the encoder network and the decoder network are respectively set as convolutional neural networks. Therefore, the encoder network 1, encoder network 2, encoder network 3, and encoder network 4 of the initial neural network from the first layer to the fourth layer are set as encoder network CNN1, encoder network CNN2, encoder network CNN3, and encoder network CNN4 in sequence; the decoder network 1, decoder network 2, decoder network 3, and decoder network 4 are set as decoder network CNN1, decoder network CNN2, decoder network CNN3, and decoder network CNN4 in sequence.

[0133] Among them, the time domain dimensionality reduction multiples of encoder networks 1, 2, 3, and 4 are 3, 4, 5, and 5 respectively; the time domain dimensionality increase multiples of the decoder network are the same as the corresponding encoder network. The quantization network 1 of the initial neural network from the first layer includes: convolutional neural network CNN1 and residual quantization network 1; the quantization network 2 of the initial neural network from the second layer includes: convolutional neural network CNN2 and residual quantization network 2; the quantization network 3 of the initial neural network from the third layer includes: convolutional neural network CNN3 and residual quantization network 3; the quantization network 4 of the initial neural network from the fourth layer includes: convolutional neural network CNN4 and residual quantization network 4.

[0134] The code table of each residual quantization network is (1024, 64), that is, it consists of 1024 multidimensional vectors of length 64; the function of the convolutional neural network CNN is to further transform the output of the encoder network to be equal to the multidimensional vector length of the code table (length 64).

[0135] The first to fourth encoding streams correspond to four code rates from high to low, namely: training input code streams of encoding rate requirement information-encoding rate 1, encoding rate requirement information-encoding rate 2, encoding rate requirement information-encoding rate 3, and encoding rate requirement information-encoding rate 4. The first to fourth encoding streams are index values ​​of the code table corresponding to the multidimensional vector output by the convolutional neural network CNN.

[0136] See also Figure 5 , taking the first coding rate as an example, it is used to train the highest code rate; the input signal is processed and truncated into (1,9600), that is, monophonic, and the number of sample points is 9600 segments, which are used as training input code streams; the training input code stream is reduced to a first training vector of (64,3200) by encoder network 1; the first training vector is transformed into (64,3200), that is, 3200 first convolution vectors of length 64 by convolutional neural network CNN1 network; the index of the codeword closest to each first convolution subvector in the first convolution vector is found from the code table in residual quantization 1, and 3200 indexes are found from 3200 multidimensional vectors of length 64 as the first coding code stream; each index in the first coding code stream consists of 10 bits (10 bits cover the index range of the code table); 3200 multidimensional vectors of length 64 are transformed into 3200 10-bit vectors = 32000 bits; and the first coding code stream of 3200 10-bit indexes is output.

[0137] See also Figure 6, taking the second encoding rate as an example, based on the first layer of mature deep learning network, the weights of encoder network 1 and decoder network 1 in the first layer of mature deep learning network are trained and solidified weights. By processing the input signal, truncating it into (1,9600), that is, mono, a segment with 9600 sample points as the training input code stream; the training input code stream is reduced in dimension by encoder network 1 in the first layer of mature deep learning network to obtain a multi-dimensional vector of (64,3200); the multi-dimensional vector of (64,3200) is reduced in dimension again by encoder network 2 in the second layer of initial deep learning network to a second training vector of (128,800); the second training vector is transformed into (64,800) by the convolutional neural network CNN2 network, That is, 800 second convolution vectors with a length of 64; the second convolution vector finds the index of the codeword closest to each second convolution sub-vector in the second convolution vector from the code table in the residual quantization 2, and 800 multidimensional vectors of length 64 find 800 indexes to obtain a second encoded code stream; each index in the second encoded code stream consists of 10 bits (10 bits cover the index range of the code table); 800 multidimensional vectors of length 64 are transformed into 800 10-bit vectors = 8000 bits; output a second encoded code stream with 800 10-bit indexes.

[0138] See also Figure 7, taking the third encoding rate as an example, based on the first and second layers of mature deep learning networks, the weights of encoder network 1 and decoder network 1 in the first layer of mature deep learning networks are weights that have been trained and solidified; the weights of encoder network 2 and decoder network 2 in the second layer of mature deep learning networks are weights that have been trained and solidified. The input signal is processed and truncated into (1,9600), that is, mono, with a sample point of 9600 as the training input code stream; the training input code stream is reduced to a multi-dimensional vector of (64,3200) by the encoder network 1 of the first layer of mature deep learning network; the multi-dimensional vector of (64,3200) is reduced to a multi-dimensional vector of (128,800) by the encoder network 2 of the second layer of mature deep learning network; the multi-dimensional vector of (128,800) is reduced to a third training vector of (256,160) by the encoder network 3 of the third layer of initial deep learning network; the third training vector After being transformed into (64, 160) by the convolutional neural network CNN3, that is, 160 third convolution vectors of length 64; the third convolution vector finds the index of the codeword closest to each third convolution sub-vector in the third convolution vector from the code table in the residual quantization 3, and 160 multidimensional vectors of length 64 find the third encoded code stream of 160 indexes; each index of the third encoded code stream consists of 10 bits (10 bits cover the index range of the code table); 160 multidimensional vectors of length 64 are transformed into 160 10-bit vectors = 1600 bits; and output a third encoded code stream of 160 10-bit indexes.

[0139] Taking the fourth coding rate as an example, according to the above principle, based on the first, second and third layers of mature deep learning networks, the fourth white horse code stream is obtained.

[0140] In a preferred embodiment, calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the L layer initial deep learning network to decode the Lth encoded bitstream to obtain the Lth training output bitstream includes:

[0141] Dequantizing the Lth coded bit stream using a preset code table to obtain an Lth code table vector;

[0142] The decoder network of the L-th layer initial deep learning network and the decoder network from the L-1-th layer initial deep learning network to the first layer initial deep learning network are called in sequence to perform dimension-upgrading processing on the L-th code table vector to obtain the L-th training output code stream.

[0143] Furthermore, the Lth coded bit stream is dequantized by using a preset code table to obtain an Lth code table vector, including:

[0144] Obtain the codeword of each index in the code table in the Lth encoded code stream;

[0145] The multidimensional vectors corresponding to each index are aggregated to obtain the Lth code table vector.

[0146] Exemplarily, if a first encoded code stream is obtained, a multidimensional vector corresponding to each index in the first encoded code stream is found from the code table of the residual quantization network 1, and the multidimensional vectors are aggregated to form a first code table vector; the first code table vector is output to the decoder network 1, the decoder network 1 performs dimensionality upscaling on the first code table vector, and outputs a first training output code stream with a mono sample point number of 9600 vectors.

[0147] If a second encoded code stream is obtained, a multidimensional vector corresponding to each index in the second encoded code stream is found from the code table of the residual quantization network 2, and the multidimensional vectors corresponding to each index in the second encoded code stream are summarized to form a second code table vector; the second code table vector is output to the decoder network 2, and the second code table vector is upgraded in dimension through the decoder network 2 and the decoder network 1 in turn, and a second training output code stream of a multidimensional vector with a mono sample point number of 9600 is output.

[0148] If a third encoded code stream is obtained, a multidimensional vector corresponding to each index in the third encoded code stream is found from the code table of the residual quantization network 3, and the multidimensional vectors corresponding to each index in the third encoded code stream are summarized to form a third code table vector; the third code table vector is output to the decoder network 3, and the third code table vector is upgraded in turn through the decoder network 3, the decoder network 2 and the decoder network 1, and a third training output code stream of a multidimensional vector with a mono sample point number of 9600 is output.

[0149] If the fourth encoded code stream is obtained, according to the above principle, the fourth training output code stream is obtained.

[0150] S403: According to the vector loss, the training input bitstream and the Lth training output bitstream, the model parameters of the network units in the Lth layer initial deep learning network are adjusted to convert the Lth layer initial deep learning network into the Lth layer mature deep learning network; wherein the vector loss is the difference between the Lth convolution vector and the multidimensional vector represented by the Lth encoding bitstream; the Lth convolution vector is the input vector of the Lth layer initial deep learning network for generating the Lth encoding bitstream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network units.

[0151] In an exemplary embodiment, it is assumed that there are n types of bit rate requirement information, the bit rate requirement information includes bit rate requirement 1, bit rate requirement 2, ..., encoding stream 1 corresponds to the highest bit rate, encoding stream 2 corresponds to the second highest bit rate, encoding stream 3 corresponds to the third highest bit rate, and so on, encoding rate n corresponds to the lowest bit rate. In the training network, encoder network 1, encoder network 2, encoder network 3, ... are a gradual dimensionality reduction process to achieve a bit rate from high to low, and decoder network n ... decoder network 3, decoder network 2, decoder network 1 are a gradual dimensionality increase process, corresponding one to one with the encoder network. During training, first train the encoder network 1+quantization network 1+decoder network 1 corresponding to the highest bit rate; then train the second highest bit rate, at this time use the previously trained encoder network 1+decoder network 1, and only train encoder network 2+quantization network 2+decoder network 2; train the third highest bit rate again, at this time use the previously trained encoder network 1+encoder network 2+decoder network 2+decoder network 1, and only train encoder network 3+quantization network 3+decoder network 3; and so on, for the lowest encoding bit rate n, only encoder network n+quantization network n+decoder network n are needed, and the rest of the networks use the trained networks.

[0152] Therefore, the training process of the present application is: training the first layer initial deep learning network; training the encoder network 1+quantization network 1+decoder network 1 in the first layer initial deep learning network, so that the first layer initial deep learning network is converted into the first layer mature deep learning network. The training input code stream is a pre-prepared audio or picture.

[0153] The second layer of initial deep learning network is trained; wherein, the training input code stream is encoded using the encoder network 1 of the first layer of mature deep learning network, the encoder network 2 of the second layer of initial deep learning network, and the quantization network 2 to obtain the second encoded code stream; the second encoded code stream is decoded using the decoder network 2 of the second layer of initial deep learning network and the decoder network 3 of the first layer of mature deep learning network to obtain the second training output code stream; according to the training input code stream and the second training output code stream, the encoder network 2+quantization network 2+decoder network 2 in the second layer of initial deep learning network is trained. The training input code stream is a pre-prepared audio or picture.

[0154] And so on, according to this principle, the third-layer initial deep learning network, the fourth-layer initial deep learning network, and so on are trained in sequence until the M-th-layer initial deep learning network.

[0155] In a preferred embodiment, according to the vector loss, the training input bitstream and the Lth training output bitstream, the model parameters of the network units in the Lth layer initial deep learning network are adjusted to convert the Lth layer initial deep learning network into the Lth layer mature deep learning network, including:

[0156] Calling a first loss function to obtain a first loss value according to the training input code stream and the Lth training output code stream; wherein the first loss value represents the difference between the training input code stream and the Lth training output code stream;

[0157] Calling a second loss function to obtain a second loss value according to the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to a preset code table and the Lth convolution vector;

[0158] The model parameters of the network units in the L-th layer initial deep learning network are adjusted according to the first loss value and the second loss value through the gradient descent method, so that the adjusted L-th layer initial deep learning network is converted into the L-th layer mature learning network.

[0159] Exemplarily, the first loss function is:

[0160]

[0161] Among them, x i is the element value of the i-th sample point in the training input code stream, y i is the element value of the i-th sample point in the L-th training output code stream, i is a natural number greater than or equal to 1 and less than or equal to M; n is the number of sample points in the training input code stream, n is also the number of sample points in the L-th training output code stream, and Loss1 is the first loss value.

[0162] The second loss function is:

[0163]

[0164] Among them, e i is the element value of the i-th sample point in the L-th convolution vector, g i It is the element value of the i-th sample point in the L-th code table vector, i is a natural number greater than or equal to 1 and less than or equal to M; n is the number of sample points in the L-th convolution vector, n is also the number of sample points in the L-th code table vector, and Loss2 is the second loss value.

[0165] Calculate the sum of the first loss value and the second loss value (Loss = Loss1 + Loss2), or perform weighted summation of the first loss value and the second loss value (Loss = a × Loss1 + b × Loss2, where a and b are preset weight parameters), or calculate the average of the first loss value and the second loss value (Loss = (Loss1 + Loss2) / 2) to obtain the reconstruction loss value Loss.

[0166] The model parameters of the network units in the L-th layer initial deep learning network are adjusted according to the reconstruction loss value through the gradient descent method, so that the adjusted L-th layer initial deep learning network is converted into the L-th layer mature learning network.

[0167] Among them, the objective function of the gradient descent method is This objective function adjusts the model parameters of the network unit, where θ1 represents the model parameters to be adjusted, Loss represents the reconstruction loss, is the partial differential symbol, and θ2 represents the adjusted model parameters.

[0168] Therefore, the model parameters θ1 of one or more of the encoder network, quantization network and decoder network in the L-th layer initial deep learning network are adjusted to θ2 through the gradient descent method to adjust the L-th layer initial deep learning network to transform it into the L-th layer mature learning network.

[0169] S404: Completely convert the M-layer initial deep learning network into a multi-level deep learning network model of M-layer mature deep learning network, and set it to a multi-level deep learning network model including M-layer mature deep learning network; the multi-level deep learning network model including M-layer mature deep learning network is used to receive the input bit stream and encoding bit rate requirement information, and determine the N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M; call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bit stream to obtain the N-th encoded bit stream; call the decoder network in the N-layer mature deep learning network to decode the N-th encoded bit stream to obtain the output bit stream.

[0170] In this embodiment, by training each layer of the initial deep learning network, the previously trained mature deep learning network can be used to provide encoder network and decoder network support for the initial deep learning network currently being trained, and then based on the Nth training output code stream and training input code stream obtained under this support, only the model parameters of the initial deep learning network currently being trained need to be adjusted to complete the training of the initial deep learning network, thereby improving the training efficiency of the deep learning network, avoiding the repeated training of multiple deep learning networks, and greatly reducing the consumption of storage resources and computing resources.

[0171] Embodiment 3:

[0172] See also Figure 8The present application provides a hierarchical encoding and decoding device 8 based on a multi-level deep learning network model, wherein the multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1;

[0173] The hierarchical coding and decoding device 8 comprises: a coding input module 81, a coding processing module 82 and a decoding processing module 83;

[0174] The encoding input module 81 is used to receive the input code stream and the encoding bit rate requirement information, and determine the N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M;

[0175] The encoding input module 81 is connected to the encoding processing module 82, and the encoding processing module 82 is used to call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input code stream to obtain the N-th encoded code stream;

[0176] The encoding processing module 82 is connected to the decoding processing module 83, and the decoding processing module 83 is used to call the decoder network in the N-layer mature deep learning network to decode the N-th encoded code stream to obtain an output code stream.

[0177] Optionally, the encoding processing module 82 is specifically used for:

[0178] The encoder networks from the first layer of mature deep learning network to the Nth layer of mature deep learning network are called in sequence to perform dimension reduction processing on the input code stream to obtain the Nth encoding vector; wherein the first layer of mature deep learning network is the mature deep learning network at the first position among the M layers of mature deep learning networks; the Nth layer of mature deep learning network is the mature deep learning network at the last position among the N layers of mature deep learning networks;

[0179] The quantization network of the N-th layer mature deep learning network is called to quantize the N-th encoding vector to obtain the N-th encoding bitstream.

[0180] Optionally, the quantized network includes: a convolutional neural network and a residual quantized network;

[0181] The encoding processing module 82 is specifically used for:

[0182] Call the quantization network of the Nth layer of mature deep learning network to quantize the Nth encoding vector to obtain the Nth encoding bitstream, including:

[0183] Calling the convolutional neural network of the Nth layer of mature deep learning network to perform convolution processing on the Nth encoding vector to obtain the Nth convolution vector; wherein the Nth convolution vector has at least one layer of Nth convolution sub-vectors;

[0184] The index corresponding to each N-th convolution sub-vector is obtained from the code table of the residual quantization network, and the index corresponding to each N-th convolution sub-vector is summarized to obtain the N-th encoded code stream; the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of index and the codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used to compare with the N-th convolution sub-vector.

[0185] Optionally, the decoding processing module 83 is specifically used for:

[0186] Dequantizing the Nth coded bitstream using a preset code table to obtain an Nth code table vector;

[0187] The decoder networks from the N-th layer of mature deep learning network to the first layer of mature deep learning network are called in sequence, and the N-th code table vector is dimensionally upgraded to obtain the output code stream.

[0188] Optionally, the decoding processing module 83 is specifically used for:

[0189] Obtain the codeword of each index in the code table in the Nth encoded code stream;

[0190] The codewords corresponding to each index are summarized to obtain the Nth code table vector.

[0191] Embodiment 4:

[0192] See also Fig. 9 The present application provides a model training device 9 for a multi-level deep learning network model, wherein the multi-level deep learning network model includes an M-layer initial deep learning network; a model training method is used to train the M-layer initial deep learning network into an M-layer mature deep learning network; the M-layer mature deep learning network is used to encode and decode input bitstreams of M bitrates to obtain output bitstreams of M bitrates; wherein M is a natural number greater than 1;

[0193] The model training device 9 includes: a training encoding module 91, a training decoding module 92, a training adjustment module 93 and a training integration module 94;

[0194] The training encoding module 91 is used to obtain a training input code stream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, and encode the training input code stream to obtain the L-th encoded code stream; wherein L layers refer to the number of layers of the initial deep learning network used for encoding and decoding the input code stream; the L-th layer initial deep learning network is the initial deep learning network at the last position in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M;

[0195] The training encoding module 91 is connected to the training decoding module 92, and the training decoding module 92 is used to call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L layer initial deep learning network to decode the Lth encoded code stream to obtain the Lth training output code stream;

[0196] The training adjustment module 92 is connected to the training decoding module 93, and the training decoding module 93 is used to adjust the model parameters of the network unit in the L-th layer initial deep learning network according to the vector loss, the training input code stream and the L-th training output code stream, so that the L-th layer initial deep learning network is converted into the L-th layer mature deep learning network; wherein the vector loss is the difference between the L-th convolution vector and the vector represented by the L-th encoding code stream; the L-th convolution vector is the input vector of the L-th layer initial deep learning network for generating the L-th encoding code stream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network unit;

[0197] The training adjustment module 93 is connected to the training integration module 94, and the training integration module 94 is used to completely convert the M-layer initial deep learning network into a multi-level deep learning network model of an M-layer mature deep learning network, which is set to a multi-level deep learning network model including an M-layer mature deep learning network; the multi-level deep learning network model including an M-layer mature deep learning network is used to receive an input bit stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement information; wherein N is a natural number greater than or equal to 1 and less than or equal to M; call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bit stream to obtain the N-th encoded bit stream; call the decoder network in the N-layer mature deep learning network to decode the N-th encoded bit stream to obtain the output bit stream.

[0198] Optionally, a training encoding module 91, a training decoding module 92, a training adjustment module 93 and a training integration module 94;

[0199] Optionally, the training encoding module 91 is specifically used for:

[0200] Sequentially call the encoder networks from the first mature deep learning network to the L-1 mature deep learning network, and the encoder network of the L initial deep learning network, and perform dimension reduction processing on the training input code stream to obtain the L training vector; wherein the first initial deep learning network is the initial deep learning network at the first position in the M initial deep learning networks; the L initial deep learning network is the initial deep learning network at the last position in the L initial deep learning networks;

[0201] The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain the L-th encoding bit stream.

[0202] Optionally, the quantized network includes: a convolutional neural network and a residual quantized network;

[0203] The training encoding module 91 is specifically used for:

[0204] The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain the L-th encoding bit stream, including:

[0205] Calling the convolutional neural network of the Lth initial deep learning network to perform convolution processing on the Lth training vector to obtain an Lth convolution vector; wherein the Lth convolution vector has at least one layer of Lth convolution sub-vectors;

[0206] The index corresponding to each L-th convolution sub-vector is obtained from the code table of the residual quantization network, and the index corresponding to each L-th convolution sub-vector is summarized to obtain the L-th encoded code stream; wherein the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of index and the codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used to compare with the N-th convolution sub-vector.

[0207] Optionally, the training decoding module 92 is specifically used for:

[0208] Dequantizing the Lth coded bit stream using a preset code table to obtain an Lth code table vector;

[0209] The decoder network of the L-th layer initial deep learning network and the decoder network from the L-1-th layer initial deep learning network to the first layer initial deep learning network are called in sequence to perform dimension-upgrading processing on the L-th code table vector to obtain the L-th training output code stream.

[0210] Optionally, the training decoding module 92 is specifically used for:

[0211] Obtain the codeword of each index in the code table in the Lth encoded code stream;

[0212] The codewords corresponding to each index are summarized to obtain the Lth code table vector.

[0213] Optionally, the training adjustment module 93 is specifically used for:

[0214] Calling a first loss function to obtain a first loss value according to the training input code stream and the Lth training output code stream; wherein the first loss value represents the difference between the training input code stream and the Lth training output code stream;

[0215] Calling a second loss function to obtain a second loss value according to the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to a preset code table and the Lth convolution vector;

[0216] The model parameters of the network units in the L-th layer initial deep learning network are adjusted according to the first loss value and the second loss value through the gradient descent method, so that the adjusted L-th layer initial deep learning network is converted into the L-th layer mature learning network.

[0217] Embodiment 5:

[0218] See also Fig.10 , in order to achieve the above-mentioned purpose, the present application also provides a computer device 10, comprising: a processor 102 and a memory 101 in communication with the processor 102; the memory stores computer-executable instructions;

[0219] The processor executes the computer execution instructions stored in the memory 101 to implement the above-mentioned multi-level encoding and decoding method and model training method, wherein the components of the multi-level encoding and decoding device and the model training device can be dispersed in different computer devices, and the computer device 10 can be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including independent servers, or server clusters composed of multiple application servers) that executes the program. The computer device of this embodiment includes at least but is not limited to: a memory 101 and a processor 102 that can be interconnected through a system bus, such as Fig.10 It should be pointed out that Fig.10Only a computer device with component - is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. In this embodiment, the memory 101 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 101 can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 101 can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 101 can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory 101 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the multi-stage encoding and decoding device and the model training device of the third embodiment. In addition, the memory 101 can also be used to temporarily store various types of data that have been output or are to be output. The processor 102 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 102 is generally used to control the overall operation of the computer device. In this embodiment, the processor 102 is used to run the program code stored in the memory 101 or process data, such as running the multi-stage encoding and decoding device and the model training device, to implement the multi-stage encoding and decoding method and the model training method of the above-mentioned embodiment.

[0220] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of each embodiment of the present application. It should be understood that the above-mentioned processor can be a central processing unit (Central Processing Unit, referred to as CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as a hardware processor to be executed, or the hardware and software modules in the processor are combined and executed. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0221] To achieve the above purpose, the present application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, App application store, etc., on which computer execution instructions are stored, and the corresponding functions are realized when the program is executed by the processor 102. The computer-readable storage medium of this embodiment is used to store computer execution instructions for implementing the multi-level encoding and decoding method and the model training method, and when executed by the processor 102, the multi-level encoding and decoding method and the model training method of the above embodiment are realized.

[0222] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0223] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0224] The present application provides a computer program product, including a computer program, which implements the above-mentioned multi-level encoding and decoding method and model training method when executed by a processor.

[0225] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0226] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0227] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A hierarchical encoding and decoding method based on a multi-level deep learning network model, characterized in that: The multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1; The hierarchical coding and decoding method comprises: Receiving an input bitstream and encoding bitrate requirement information, and determining an N-layer mature deep learning network according to the encoding bitrate requirement information; wherein N is a natural number greater than or equal to 1 and less than or equal to M; Calling the encoder network in the N-layer mature deep learning network and the quantization network in the N-th layer of the mature deep learning network to encode the input bitstream to obtain an N-th encoded bitstream; Calling the decoder network in the N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream.

2. The hierarchical coding and decoding method according to claim 1, characterized in that: Calling the encoder network in the N-layer mature deep learning network and the quantization network in the N-th layer of the mature deep learning network to encode the input bitstream to obtain the N-th encoded bitstream, including: Sequentially calling encoder networks from the first layer of mature deep learning networks to the Nth layer of mature deep learning networks, performing dimensionality reduction processing on the input code stream to obtain the Nth encoding vector; wherein the first layer of mature deep learning networks is the mature deep learning network that is at the first position among the M layers of mature deep learning networks; the Nth layer of mature deep learning networks is the mature deep learning network that is at the last position among the N layers of mature deep learning networks; The quantization network of the N-th layer mature deep learning network is called to quantize the N-th encoding vector to obtain the N-th encoding bit stream.

3. The hierarchical coding and decoding method according to claim 2, characterized in that: The quantization network includes: a convolutional neural network and a residual quantization network; Calling the quantization network of the N-th layer of mature deep learning network to quantize the N-th encoding vector to obtain the N-th encoding bitstream, including: Calling the convolutional neural network of the N-th layer of mature deep learning network to perform convolution processing on the N-th encoding vector to obtain an N-th convolution vector; wherein the N-th convolution vector has at least one layer of N-th convolution sub-vectors; The index corresponding to each of the N-th convolution sub-vectors is obtained from the code table of the residual quantization network, and the index corresponding to each of the N-th convolution sub-vectors is summarized to obtain the N-th encoded code stream; the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of indexes and a codeword corresponding to each index; the index is a quantization value used to describe the codeword; the codeword is a vector used to compare with the N-th convolution sub-vector.

4. The hierarchical coding and decoding method according to claim 1, characterized in that: Calling the decoder network in the N-layer mature deep learning network to decode the N-th encoded bitstream to obtain an output bitstream, including: Performing inverse quantization processing on the Nth encoded bit stream using a preset code table to obtain an Nth code table vector; The decoder networks from the N-th layer of mature deep learning network to the first layer of mature deep learning network are called in sequence, and the N-th code table vector is processed by dimension upgrading to obtain an output code stream.

5. The hierarchical coding and decoding method according to claim 4, characterized in that: The Nth coded bit stream is dequantized by using a preset code table to obtain an Nth code table vector, including: Obtaining a codeword of each index in the code table in the Nth encoded code stream; The codewords corresponding to each index are summarized to obtain the Nth code table vector.

6. A model training method for a multi-level deep learning network model, characterized in that: The multi-level deep learning network model includes M layers of initial deep learning networks; the model training method is used to train the M layers of initial deep learning networks into M layers of mature deep learning networks; the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1; The model training method comprises: Obtain a training input bitstream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, encode the training input bitstream to obtain the L-th encoded bitstream; wherein, L layers refer to the number of layers of the initial deep learning network used for encoding and decoding the input bitstream; the L-th layer initial deep learning network is the last initial deep learning network in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M; Calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream; According to the vector loss, the training input code stream and the Lth training output code stream, the model parameters of the network unit in the Lth layer initial deep learning network are adjusted to convert the Lth layer initial deep learning network into the Lth layer mature deep learning network; wherein the vector loss is the difference between the Lth convolution vector and the vector represented by the Lth encoding code stream; the Lth convolution vector is the input vector of the Lth layer initial deep learning network for generating the Lth encoding code stream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network unit; The M-layer initial deep learning network is completely converted into a multi-level deep learning network model of an M-layer mature deep learning network, which is set as a multi-level deep learning network model including an M-layer mature deep learning network; the multi-level deep learning network model including an M-layer mature deep learning network is used to receive an input bit stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement; N is a natural number greater than or equal to 1 and less than or equal to M; the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network are called to encode the input bit stream to obtain an N-th encoded bit stream; the decoder network in the N-layer mature deep learning network is called to decode the N-th encoded bit stream to obtain an output bit stream.

7. The model training method according to claim 6, characterized in that: Calling the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, encoding the training input bitstream to obtain the L-th encoded bitstream, including: Sequentially calling the encoder networks from the first layer of mature deep learning networks to the L-1th layer of mature deep learning networks, and the encoder network of the Lth layer of initial deep learning networks, and performing dimensionality reduction processing on the training input code stream to obtain the Lth training vector; wherein the first layer of initial deep learning network is the initial deep learning network that is at the first position among the M layers of initial deep learning networks; the Lth layer of initial deep learning network is the initial deep learning network that is at the last position among the L layers of initial deep learning networks; The quantization network of the L-th layer initial deep learning network is called to quantize the L-th training vector to obtain an L-th encoded bitstream.

8. The model training method according to claim 7, characterized in that: The quantization network includes: a convolutional neural network and a residual quantization network; Calling the quantization network of the L-th layer initial deep learning network to quantize the L-th training vector to obtain the L-th encoding bitstream, including: Calling the convolutional neural network of the Lth initial deep learning network to perform convolution processing on the Lth training vector to obtain an Lth convolution vector; wherein the Lth convolution vector has at least one layer of Lth convolution sub-vectors; Obtain an index corresponding to each of the L-th convolution sub-vectors from the code table of the residual quantization network, and summarize the index corresponding to each of the L-th convolution sub-vectors to obtain an L-th encoded code stream; wherein the code table is an index codeword table preset in the residual quantization network, and the index codeword table records at least one layer of indexes and a codeword corresponding to each index; the index is a quantization value used to describe the codeword; and the codeword is a vector used to compare with the N-th convolution sub-vector.

9. The model training method according to claim 6, characterized in that: Calling the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded bitstream to obtain the L-th training output bitstream, including: Performing inverse quantization processing on the Lth encoded bit stream using a preset code table to obtain an Lth code table vector; The decoder network of the L-th layer initial deep learning network and the decoder network from the L-1-th layer initial deep learning network to the first layer initial deep learning network are called in sequence, and the L-th code table vector is dimensionally upgraded to obtain the L-th training output code stream.

10. The model training method according to claim 9, characterized in that: The Lth coded bit stream is dequantized by using a preset code table to obtain an Lth code table vector, including: Obtaining a codeword of each index in the code table in the Lth encoded code stream; The codewords corresponding to each index are summarized to obtain the Lth code table vector.

11. The model training method according to claim 6, characterized in that: According to the vector loss, the training input bitstream and the Lth training output bitstream, adjusting the model parameters of the network units in the Lth layer initial deep learning network so that the Lth layer initial deep learning network is converted into an Lth layer mature deep learning network, including: Calling a first loss function to obtain a first loss value according to the training input code stream and the Lth training output code stream; wherein the first loss value represents the difference between the training input code stream and the Lth training output code stream; Calling a second loss function to obtain a second loss value according to the vector loss; wherein the second loss value represents the difference between the Lth convolution vector obtained by encoding and decoding the Lth convolution vector according to a preset code table and the Lth convolution vector; The model parameters of the network units in the L-th layer initial deep learning network are adjusted according to the first loss value and the second loss value by the gradient descent method, so that the adjusted L-th layer initial deep learning network is converted into the L-th layer mature learning network.

12. A hierarchical encoding and decoding device based on a multi-level deep learning network model, characterized in that: The multi-level deep learning network model includes M layers of mature deep learning networks, and the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1; The hierarchical coding and decoding device comprises: a coding input module, a coding processing module and a decoding processing module; The encoding input module is used to receive an input code stream and encoding bit rate requirement information, and determine an N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M; The encoding input module is connected to the encoding processing module, and the encoding processing module is used to call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input code stream to obtain the N-th encoded code stream; The encoding processing module is connected to the decoding processing module, and the decoding processing module is used to call the decoder network in the N-layer mature deep learning network to decode the N-th encoded code stream to obtain an output code stream.

13. A model training device for a multi-level deep learning network model, characterized in that: The multi-level deep learning network model includes M layers of initial deep learning networks; the model training method is used to train the M layers of initial deep learning networks into M layers of mature deep learning networks; the M layers of mature deep learning networks are used to encode and decode input bit streams of M bit rates to obtain output bit streams of M bit rates; wherein M is a natural number greater than 1; The model training device comprises: a training encoding module, a training decoding module, a training adjustment module and a training integration module; The training encoding module is used to obtain a training input code stream, call the encoder network in the L-1 layer mature deep learning network, and the encoder network and quantization network of the L-th layer initial deep learning network, and encode the training input code stream to obtain the L-th encoded code stream; wherein the L layer refers to the number of layers of the initial deep learning network used for encoding and decoding the input code stream; the L-th layer initial deep learning network is the initial deep learning network at the last position in the L-layer initial deep learning network; L is a natural number greater than or equal to 1 and less than or equal to M; The training encoding module is connected to the training decoding module, and the training decoding module is used to call the decoder network in the L-1 layer mature deep learning network and the decoder network in the L-th layer initial deep learning network to decode the L-th encoded code stream to obtain the L-th training output code stream; The training adjustment module is connected to the training decoding module, and the training decoding module is used to adjust the model parameters of the network unit in the L-th layer initial deep learning network according to the vector loss, the training input code stream and the L-th training output code stream, so that the L-th layer initial deep learning network is converted into an L-th layer mature deep learning network; wherein the vector loss is the difference between the L-th convolution vector and the vector represented by the L-th encoding code stream; the L-th convolution vector is the input vector of the L-th layer initial deep learning network for generating the L-th encoding code stream; the network unit includes one or more of an encoder network, a quantization network and a decoder network; the model parameters include weights and / or bias values ​​of the network unit; The training adjustment module is connected to the training integration module, and the training integration module is used to completely convert the M-layer initial deep learning network into a multi-level deep learning network model of M-layer mature deep learning network, which is set to a multi-level deep learning network model including M-layer mature deep learning network; the multi-level deep learning network model including M-layer mature deep learning network is used to receive input bit stream and encoding bit rate requirement information, and determine N-layer mature deep learning network according to the encoding bit rate requirement information; N is a natural number greater than or equal to 1 and less than or equal to M; call the encoder network in the N-layer mature deep learning network and the quantization network of the N-th layer mature deep learning network to encode the input bit stream to obtain the N-th encoded bit stream; call the decoder network in the N-layer mature deep learning network to decode the N-th encoded bit stream to obtain the output bit stream.

14. A computer device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images