An audio encoding and decoding method based on neural network model

Abstract

The present invention discloses an audio coding and decoding method based on a neural network model, comprising the following steps: Step 1, generating an audio signal recognized by a discriminator through a generator, driving the discriminator to continuously update its identification model; Step 2, distinguishing the audio signal of the generator from the real audio signal through the discriminator, and improving the generation ability of the generator. The audio coding and decoding method of the present invention has simple steps, can not only effectively reduce the amount of data transmitted through the network when using online music services, but also can reduce storage requirements or network bandwidth, can further compress the quantized representation of audio by 10% compared with the prior art, while limiting real-time operation on a single-core CPU, and using RVQ to improve coding efficiency.

Description

Technical Field

[0001] The present invention relates to the field of digital audio coding and decoding and transmission technology, and in particular to an audio coding and decoding method based on a neural network model. Background Art

[0002] With the continuous development of Internet technology, digital music, and online music service products, people's demand for high-quality music is also constantly upgrading, and concepts such as HiRes and lossless sound quality have also emerged. When providing online music services, it is essential to use audio compression technology.

[0003] In the field of audio compression, there are two compression methods, namely lossy compression and lossless compression. Common MP3, WMA, OGG are called lossy compression. As the name implies, lossy compression reduces the audio sampling frequency and bit rate, and the output audio file will be smaller than the original file. Another type of audio compression is called lossless compression, which is the theme content. Lossless compression can compress the volume of the audio file to a smaller size while preserving 100% of all the data of the original file, and after restoring the compressed audio file, it can achieve the same size and bit rate as the source file. Lossless compression formats include APE, FLAC, WavPack, LPAC, WMALossless, AppleLossless, La, OptimFROG, Shorten, and the common and mainstream lossless compression formats are only APE and FLAC.

[0004] In the prior art, when online music service products provide music services, it is inevitable to transmit music files to the user's terminal. Even though some compression technologies have been used to effectively reduce the space occupied by audio files, higher-quality audio files will still occupy more storage space; higher-quality audio files will have a greater chance of being selected by users, and will be transmitted more times on the network, which will more significantly consume the network bandwidth traffic of the online music service platform. Summary of the invention

[0005] The purpose of the present invention is to provide a technical solution for an audio encoding and decoding method based on a neural network model to address the deficiencies in the prior art. The audio encoding and decoding method has simple steps and can not only effectively reduce the amount of data transmitted through the network when using online music services, but also reduce storage requirements or network bandwidth. Compared with the prior art, the quantized representation of the audio can be further compressed by 10%, while limiting real-time operation on a single-core CPU and using RVQ to improve coding efficiency.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] An audio encoding and decoding method based on a neural network model, characterized by comprising the following steps:

[0008] Step 1: Generate an audio signal recognized by the discriminator through the generator, driving the discriminator to continuously update its identification model;

[0009] Step 2: Use the discriminator to distinguish the generator's audio signal from the real audio signal to improve the generator's generation ability.

[0010] The audio encoding and decoding method has simple steps and can not only effectively reduce the amount of data transmitted through the network when using online music services, but also reduce storage requirements or network bandwidth. Compared with the existing technology, the quantized representation of the audio can be further compressed by 10%, while limiting real-time operation on a single-core CPU and using RVQ to improve encoding efficiency.

[0011] Further, the step of generating the audio signal recognized by the discriminator through the generator in step 1 specifically includes the following steps:

[0012] Step 1.1 The audio signal is passed into the neural network of the encoder, and the time domain waveform of the audio signal is feature extracted to obtain the potential representation of the audio signal;

[0013] Step 1.2: The potential representation of the audio signal is passed into the quantizer, and the potential representation of the audio signal is processed using vector quantization to obtain a compressed representation;

[0014] Step 1.3 reconstructs the time domain waveform through the decoder using the compressed representation.

[0015] Furthermore, the time domain waveform in step Step 1.1 is an audio signal with a duration of d, which is represented by the sequence x∈[-1,1]Ca×T, where Ca is the number of audio channels and T is the number of audio samples at a given sampling rate sr, that is, T=d·sr.

[0016] Furthermore, the quantizer in step 1.2 uses residual vector quantization (RVQ) to quantize the potential features of the time domain waveform output by the encoder.

[0017] Further, residual vector quantization involves projecting the input vector onto the closest entry in a codebook of a given size.

[0018] Furthermore, the encoder is responsible for processing the input time domain waveform into a low sampling rate embedding vector; the quantizer is used to quantize a series of embedding vectors generated by the encoder; and the decoder is used to restore the embedding vectors generated by the quantizer to the original time domain waveform.

[0019] Furthermore, a quantization loss is added between the encoder output and the quantized value of the quantizer And the gradient of the quantized value is not calculated. For each residual step c∈{1,...C}, where C depends on the bandwidth target of the current batch, the quantization loss for:

[0020]

[0021] where zc is the current residual and qc(zc) is the index of the most similar vector in the corresponding codebook.

[0022] Furthermore, a reconstruction loss is added between the encoder input and the decoder output. The reconstruction loss consists of time domain loss and frequency domain loss. The reconstruction loss function is for:

[0023]

[0024] Where x is the audio signal of the time domain waveform of the encoder input audio, and x' is the audio signal of the time domain waveform of the decoder output audio.

[0025] Furthermore, the discriminator adopts a multi-resolution complex STFT discriminator.

[0026] Furthermore, the generator includes an adversarial loss function, which is used to define the probability that the virtual data generated by the generator is mistakenly judged as real data by the discriminator, and specifically includes the following structure:

[0027]

[0028] where k is the number of discriminators and Dk(x) represents the prediction of the kth discriminator for the true sample k at time t.

[0029] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0030] The audio encoding and decoding method of the present invention has simple steps, which can not only effectively reduce the amount of data transmitted through the network when using online music services, but also reduce storage requirements or network bandwidth. Compared with the prior art, the quantized representation of the audio can be further compressed by 10%, while limiting real-time operation on a single-core CPU and using RVQ to improve encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below in conjunction with the accompanying drawings:

[0032] Figure 1 A flowchart of an audio encoding and decoding method based on a neural network model of the present invention;

[0033] Figure 2 A specific flow chart of generating an audio signal recognized by a discriminator through a generator in the present invention;

[0034] Figure 3 This is a flowchart of the image-music multimodal description model training in the present invention;

[0035] Figure 4 This is a flowchart of the image-music multimodal description model in the present invention during operation. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] In the following description, specific embodiments of the present invention will be described with reference to steps and symbols of operations performed by one or more computers, unless otherwise stated. Therefore, it will be understood that these steps and operations, which are several times referred to as being performed by a computer, include manipulation by a computer processing unit of electronic signals representing data in a structured form. This manipulation transforms the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to those skilled in the art. The data structure in which the data is maintained is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present invention are described in the above text, which does not represent a limitation, and those skilled in the art will understand that the various steps and operations described below can also be implemented in hardware.

[0039] As used herein, the terms "component," "module," "system," "interface," "process," and the like are generally intended to refer to a computer-related entity: hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable application, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a controller and the controller can be components. One or more components can be within a process and / or thread of execution, and a component can be located on one computer and / or distributed between two or more computers.

[0040] Moreover, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture that uses standard programming and / or techniques to generate software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to include a computer program accessible from any computer-readable device, carrier, or medium. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0041] like Figures 1 to 4 As shown, an audio encoding and decoding method based on a neural network model of the present invention comprises the following steps:

[0042] Step 1: Generate an audio signal recognized by the discriminator through the generator, driving the discriminator to continuously update its identification model;

[0043] The generator generates an audio signal to be recognized by the discriminator, which specifically includes the following steps:

[0044] Step 1.1 The audio signal is transmitted to the neural network of the encoder, and the time domain waveform of the audio signal is feature extracted to obtain the potential representation of the audio signal; the encoder is a fully convolutional neural network structure.

[0045] The time domain waveform is an audio signal with a duration of d, represented by the sequence x∈[-1,1]Ca×T, where Ca is the number of audio channels and T is the number of audio samples at a given sampling rate sr, that is, T=d·sr.

[0046] The sequence enters a convolution kernel with a kernel size of 7 and a one-dimensional convolution layer with 32 channels for feature extraction. Then it enters the 4 convolution blocks that are connected closely behind. Each convolution block contains a residual processing unit and a downsampling layer. The residual unit contains two convolution kernels with a kernel size of 3 and a skip connection to downsample the data. Then the two-layer LSTM for sequence modeling is used for sequence modeling. Finally, it enters a one-dimensional convolution layer with a kernel size of 7 to obtain the potential representation of the time domain waveform.

[0047] Step 1.2: The potential representation of the audio signal is passed into the quantizer, and the potential representation of the audio signal is processed using vector quantization to obtain a compressed representation;

[0048] The quantizer uses residual vector quantization (RVQ) to quantize the underlying features of the time domain waveform output by the encoder. The main step of the quantizer is to reduce a vector with 128 dimensions to a single number (index in the codebook), thereby compressing the information.

[0049] Residual vector quantization consists in projecting the input vector onto the closest entry in a codebook of a given size.

[0050] In the ordinary quantization process, the quantizer needs to have a codebook table containing a certain number of learnable vectors. The input vector is compared with the vectors in the codebook and the index of the most similar vector is extracted. The quantizer needs to use a huge codebook to accurately represent the compressed data. RVQ calculates the residual after quantization through an improvement process, which quantizes the input vector x by applying M consecutive VQ modules to it. The first VQ module is applied to the input vector x using the first codebook CB1. Then, after finding the index of the most similar vector in the first codebook, the remainder R1 is calculated. After that, R1 is passed as input to the next VQ module using the second codebook CB2. This process will continue for M stages, in which the indices of the three most similar vectors from different codebooks will be found. Finally, the input vector x is quantized as the sum of the indices of the M most similar vectors.

[0051] This application uses up to 32 codebooks (16 for the 48kHz model), each codebook contains 1024 vectors, each codebook is 10 bits, and a potential representation of the time domain waveform [B, D, T] from the encoder is assumed. This process converts it into a set of discrete indexes [B, Nq, T], where Nq is the number of selected codebooks. For audio with a sampling rate of 24kHz, the maximum bit rate is 24kbps. In order to support multiple bit rates, the number of codebooks is set to {2, 4, 8, 16, 32} during training, and up to 32 codebooks are used, corresponding to 1.5kbps, 3kbps, 6kbps, 12kbps, and 24kbps, respectively. A transormer model is used to map the index obtained by RVQ to the probability distribution of the new hidden space, and entropy code the cumulative distribution function of the corresponding probability density function. The model consists of 5 layers, 8 attention heads, and 200 channels.

[0052] During training, a quantization loss is added between the encoder output and the quantized value of the quantizer. And the gradient of the quantized value is not calculated. For each residual step c∈{1,...C}, where C depends on the bandwidth target of the current batch, the quantization loss for:

[0053]

[0054] where zc is the current residual and qc(zc) is the index of the most similar vector in the corresponding codebook.

[0055] Step 1.3 reconstructs the time domain waveform through the decoder using the compressed representation.

[0056] The decoder follows a similar process to the encoder, but in reverse order. Instead of strided convolutions, transposed convolutions are used, and the stride order is the opposite of the encoder, outputting the final mono or stereo audio.

[0057] A reconstruction loss is added between the input of the encoder and the output of the decoder. The reconstruction loss consists of time domain loss and frequency domain loss. The reconstruction loss function for:

[0058]

[0059] Where x is the audio signal of the time domain waveform of the encoder input audio, and x' is the audio signal of the time domain waveform of the decoder output audio.

[0060] The entire system is trained end-to-end to minimize the reconstruction loss when reconstructing the time domain waveform. When training the neural network, a 5-second music clip is used as an example to train the model.

[0061] The encoder is responsible for processing the input time domain waveform into a low sampling rate embedding vector; the quantizer is used to quantize a series of embedding vectors generated by the encoder; and the decoder is used to restore the embedding vectors generated by the quantizer to the original time domain waveform.

[0062] The generator includes an adversarial loss function, which is used to define the probability that the virtual data generated by the generator will be mistakenly judged as real data by the discriminator. Specifically, it includes the following structure:

[0063]

[0064] where k is the number of discriminators and Dk(x) represents the prediction of the kth discriminator for the true sample k at time t.

[0065] That is, the discriminator misclassifies the real sample as the generated error, and max(0,1-Dk,t(x)) ensures that when the discriminator predicts the real sample, the loss is 0, otherwise the loss is the absolute value of the prediction error. The discriminator is trained to minimize the adversarial loss function hinge-loss.

[0066] Step 2: Use the discriminator to distinguish the generator's audio signal from the real audio signal to improve the generator's generation ability.

[0067] To ensure that the reconstructed audio sounds as good as the original, a discriminator is used to distinguish whether a piece of audio has been processed by a codec or is the original unprocessed real audio.

[0068] The discriminator uses a multi-resolution complex STFT discriminator. This is a technique that converts audio signals into information in two dimensions: time and frequency. Each STFT discriminator consists of a 2D convolutional layer with a kernel size of 3x8 and 32 channels, a series of 2D convolutional layers with dilation rates increasing to 1, 2, 4, and a stride of 2, and a final 2D convolutional layer with a kernel size of 3x3 and a stride of (1,1). In simple terms, the discriminator first takes the 24kHz audio and then uses the STFT technique to convert the audio into a 2D image with time and frequency information. In this process, 5 different scales are used (STFT window lengths are [2048, 1024, 512, 256, 128]) and the LeakyReLU function is used as the activation function. Finally, a fully connected layer (that is, a network layer that aggregates information) summarizes this data and generates a single value. The purpose of this value is to give a signal that the waveform being examined is encoded audio or real, unprocessed audio.

[0069] The above codec methods are used to reduce storage requirements or network bandwidth. Ideally, the audio codec should be transparent to the end user so that the decoded audio is perceptually indistinguishable from the original audio and the encoding / decoding process does not introduce perceptible delays.

[0070] The audio encoding and decoding method has simple steps and can not only effectively reduce the amount of data transmitted through the network when using online music services, but also reduce storage requirements or network bandwidth. Compared with the existing technology, the quantized representation of the audio can be further compressed by 10%, while limiting real-time operation on a single-core CPU and using RVQ to improve encoding efficiency.

[0071] The above are only specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent replacements or modifications made based on the present invention to achieve basically the same technical effects are all included in the protection scope of the present invention.

Claims

1. An audio encoding and decoding method based on a neural network model, characterized in that The steps include: Step 1: Generate an audio signal recognized by the discriminator through the generator, driving the discriminator to continuously update its identification model; Step 2: Use the discriminator to distinguish the generator's audio signal from the real audio signal to improve the generator's generation ability.

2. The audio encoding and decoding method based on a neural network model according to claim 1, characterized in that: The step 1 in which the generator generates the audio signal recognized by the discriminator specifically includes the following steps: Step 1.1 The audio signal is passed into the neural network of the encoder, and the time domain waveform of the audio signal is feature extracted to obtain the potential representation of the audio signal; Step 1.2: The potential representation of the audio signal is passed into the quantizer, and the potential representation of the audio signal is processed using vector quantization to obtain a compressed representation; Step 1.3 reconstructs the time domain waveform through the decoder using the compressed representation.

3. The audio encoding and decoding method based on a neural network model according to claim 2, characterized in that: The time domain waveform in step Step1.1 is an audio signal with a duration of d, which is represented by the sequence x∈[-1,1]Ca×T, where Ca is the number of audio channels and T is the number of audio samples at a given sampling rate sr, that is, T=d·sr.

4. The audio encoding and decoding method based on a neural network model according to claim 2, characterized in that: The quantizer in step Step 1.2 uses residual vector quantization RVQ to quantize the potential features of the time domain waveform output by the encoder.

5. The audio encoding and decoding method based on a neural network model according to claim 4, characterized in that: The residual vector quantization consists in projecting the input vector onto the closest entry in a codebook of a given size.

6. The audio encoding and decoding method based on a neural network model according to claim 2, characterized in that: The encoder is responsible for processing the input time domain waveform into a low sampling rate embedding vector; The quantizer is used to quantize a series of embedding vectors generated by the encoder; The decoder is used to restore the embedded vector generated by the quantizer into an original time domain waveform.

7. The audio encoding and decoding method based on a neural network model according to claim 6, characterized in that: A quantization loss lw is added between the output of the encoder and the quantized value of the quantizer, and the gradient of the quantized value is not calculated. For each residual step c∈{1,...C}, where C depends on the bandwidth target of the current batch, the quantization loss lw is: lw=∑c=1||zc-qc(zc)||; where zc is the current residual and qc(zc) is the index of the most similar vector in the corresponding codebook.

8. The audio encoding and decoding method based on a neural network model according to claim 6, characterized in that: A reconstruction loss is added between the input of the encoder and the output of the decoder. The reconstruction loss consists of a time domain loss and a frequency domain loss. The reconstruction loss function lt is: lt(x, x') = ||x - x'||; Where x is the audio signal of the time domain waveform of the encoder input audio, and x' is the audio signal of the time domain waveform of the decoder output audio.

9. The audio encoding and decoding method based on a neural network model according to claim 1, characterized in that: The discriminator adopts a multi-resolution complex STFT discriminator.

10. The audio encoding and decoding method based on a neural network model according to claim 1, characterized in that: The generator includes an adversarial loss function, which is used to define the probability that the virtual data generated by the generator is mistakenly judged as real data by the discriminator, and specifically includes the following structure: lg(x)=1 / k*∑k max(0, 1-Dk, t(x))); where k is the number of discriminators and Dk(x) represents the prediction of the kth discriminator for the true sample k at time t.

Images