Audio coding and decoding method and device based on neural network, equipment and storage medium

Abstract

The invention relates to the technical field of audio coding and decoding, and discloses an audio coding and decoding method and device based on a neural network, equipment and a storage medium. The method comprises the following steps of: firstly, gradually quantizing residual information of an audio signal by introducing an improved residual vector quantization method, and more finely retaining audio characteristics in a compression process; then, a loss function fusing reconstruction loss, perception loss, residual vector quantization loss and commitment loss is adopted, so that the reconstruction quality of the audio signal under different bit rates is remarkably improved, and efficient audio compression can be realized while high tone quality is ensured; and finally, an entropy coding module is introduced after quantization, so that the code rate of coding is further reduced, and the bandwidth requirement of audio real-time transmission is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of audio coding and decoding technology, and in particular to a neural network-based audio coding and decoding method, device, equipment and storage medium. Background Art

[0002] The more popular traditional codecs are more combined with traditional methods such as linear prediction coding, code-excited linear prediction and modified discrete cosine transform. These audio codecs occupy an important position in the field of audio compression coding, but there are still some shortcomings: traditional audio codecs rely more on artificial pre-designed signal processing methods such as linear prediction coding. These algorithms usually need to make a trade-off between audio quality and compression coding efficiency, and they usually perform well at medium and high bit rates, but are prone to unavoidable sound quality degradation at low bit rates, especially in complex audio environments. Traditional audio codecs are usually designed and optimized for specific audio content, but it is difficult to perform well in relatively broad and general audio content and application scenarios; in scenarios with complex background noise or echo, the performance of traditional audio codecs will be significantly reduced because they mainly rely on rule-driven audio signal processing bases and are difficult to maintain stable performance in complex environments; the design of traditional codecs is often static, which means that most of them need to operate in a designed environment and are difficult to flexibly adjust under different bit rates, sampling rates or delay requirements.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an audio coding and decoding method, device, equipment and storage medium based on a neural network, aiming to solve the problems of traditional coding and decoding methods such as degraded sound quality at low bit rates, poor adaptability and lack of flexibility.

[0005] The first aspect of the present invention provides an audio coding and decoding method based on a neural network, comprising: pre-building a codec network, the codec network comprising: an encoder network for receiving an original audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal A decoder network; pre-designing a fusion loss function of the codec network, the fusion loss function including reconstruction loss, perceptual loss, residual vectorization loss and commitment loss; using the original audio data as input to perform end-to-end training on the codec network; obtaining the model output through forward propagation; calculating the loss value of the fusion loss function; and using a back-propagation algorithm to propagate the loss value back to each parameter in the codec network; updating the parameters to reduce the loss value, and obtaining an audio codec model; inputting the audio to be processed into the audio codec model, and outputting a reconstructed audio signal.

[0006] Optionally, in a first implementation manner of the first aspect of the present invention, the encoder network includes a first convolutional layer with a channel number of C and a convolution kernel size of 7, B convolutional blocks, an LSTM for sequence modeling, and a second convolutional layer with a convolution kernel size of 7 and an output channel of D; wherein each convolutional block is composed of a residual unit, followed by a downsampling layer composed of convolutions with a stride of S, the kernel size K of the downsampling layer is twice the stride S, and the residual unit contains two convolutions with a kernel size of 3 and 1 jump connection.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the quantizer is a residual multi-stage vector quantizer, and the encoder network output audio latent representation z is compressed into a discrete quantization index, i.e., a codebook vector, in the residual multi-stage vector quantizer, and the quantization index represents a discrete value obtained after multi-level quantization, wherein the residual multi-stage vector quantizer is composed of Nq layers of vector quantization layers cascaded together; the unquantized input vector x is processed by the first vector quantizer codebook and the quantization residual is calculated to obtain the quantization result q1 of the first layer and the first residual r1; the residual r1 is quantized by the second vector quantizer codebook to obtain the quantization result q2 of the second layer and the second residual r2; quantization is performed iteratively until a predetermined number of quantization layers Nq is reached; the quantization results of each layer are combined to obtain the final quantization result. , the quantization codeword q is the result of the entire quantization process, representing the encoded signal, the final quantization feature .

[0008] Optionally, in a third implementation of the first aspect of the present invention, the codec network further includes an entropy encoder located after the quantizer, and the entropy encoder is used to arithmetically encode the probability distribution of the quantization index to generate a compressed code stream.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the entropy encoder is used to perform arithmetical encoding on the probability distribution of the quantization index to generate a compressed code stream, including the steps of: constructing a probability model for the quantized symbol sequence, assuming that the symbol sequence is S={s1,s2,…,sn}, and the probability of each symbol si is obtained from the statistical results of historical data, expressed as P(si), then the cumulative probability C(si) of each symbol is the sum of the probabilities of all symbols before symbol si, that is, ; At the beginning of arithmetic coding, an initial interval [0,1] is defined. For each symbol si in the symbol sequence, according to its probability P(si) and cumulative probability C(si), the current interval [low, high] is gradually reduced to a smaller interval [low', high'], as follows: ; ; When all symbols are processed, the midpoint of the interval is selected as the final encoding result. This value is represented by binary, which is the final compressed code stream.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the fusion loss function is: , where L_rec represents reconstruction loss, L_percept represents perception or discrimination loss, L_RVQ represents residual vector quantization loss, and L_commit represents commitment loss. The λ value corresponding to each loss represents the trade-off and balance between various indicators.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, during end-to-end training of the codec network, a balancer is introduced to dynamically adjust the gradient of each loss term in the fusion loss function to balance the contribution of each loss term to the model during the optimization process.

[0012] The second aspect of the present invention provides an audio codec device based on a neural network, comprising: a network construction module for pre-constructing a codec network, wherein the codec network comprises: an encoder network for receiving an original audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal A decoder network; a loss function design module, used to pre-design the fusion loss function of the codec network, the fusion loss function includes: reconstruction loss, perception loss, residual vectorization loss and commitment loss, the codec network is end-to-end trained using the original audio data as input, the model output is obtained by forward propagation, the loss value of the fusion loss function is calculated, and the loss value is fed back to each parameter in the codec network by a back propagation algorithm, the parameters are updated to reduce the loss value, and an audio codec model is obtained; an output module, used to input the audio to be processed into the audio codec model, and output a reconstructed audio signal.

[0013] The third aspect of the present invention provides a neural network-based audio codec device, comprising: a memory and at least one processor, the memory storing computer-readable instructions, the memory and the at least one processor being interconnected via lines; the at least one processor calling the computer-readable instructions in the memory so that the neural network-based audio codec device performs the various steps of the neural network-based audio codec method as described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the various steps of the neural network-based audio encoding and decoding method as described above.

[0015] Beneficial effects: The present invention aims at the technical problems of traditional audio codecs such as degraded sound quality, poor adaptability and lack of flexibility at low bit rates, and provides an audio coding and decoding method based on a neural network. First, by introducing an improved residual vector quantization method, the residual information of the audio signal is gradually quantized, and the audio features are more finely retained during the compression process; then, a loss function that combines reconstruction loss, perceptual loss, residual vector quantization loss and commitment loss is adopted, which significantly improves the reconstruction quality of the audio signal at different bit rates, and can achieve efficient audio compression while ensuring high sound quality; finally, by introducing an entropy coding module after quantization, the encoding bit rate is further reduced, which significantly reduces the bandwidth requirement for real-time audio transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a neural network-based audio encoding and decoding method provided in an embodiment of the present invention.

[0017] Figure 2 A schematic diagram of a neural network-based audio encoding and decoding method provided in an embodiment of the present invention.

[0018] Figure 3The following is a graph showing the MUSHRA score results for different models under different data sets and bit rates.

[0019] Figure 4 A schematic diagram of the structure of an audio encoding and decoding device based on a neural network provided by the present invention.

[0020] Figure 5 A schematic diagram of the structure of an audio codec device based on a neural network provided by the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present invention provide a method, apparatus, device and storage medium for audio coding and decoding based on a neural network. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] See also Figure 1 , Figure 1 A flowchart of an audio encoding and decoding method based on a neural network provided by the present invention is shown in the figure, which includes the following steps: S10, pre-constructing a codec network, the codec network comprising: an encoder network for receiving raw audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal The decoder network of S20, pre-designing a fusion loss function of the codec network, the fusion loss function including: reconstruction loss, perception loss, residual vectorization loss and commitment loss, using the original audio data as input to perform end-to-end training on the codec network, obtaining a model output through forward propagation, calculating a loss value of the fusion loss function, and using a back-propagation algorithm to transmit the loss value back to each parameter in the codec network, updating the parameters to reduce the loss value, and obtaining an audio codec model; S30: Input the audio to be processed into the audio coding and decoding model, and output a reconstructed audio signal.

[0023] Specifically, if Figure 2 As shown, the encoder network contains a one-dimensional convolution layer with C channels and a kernel size of 7, followed by B convolution blocks; each convolution block consists of a residual unit, followed by a downsampling layer consisting of convolutions with a stride of S, and the kernel size K of the downsampling layer is twice the stride S; the residual unit contains two convolutions with a kernel size of 3 and a skip connection; after each downsampling operation, the number of channels is doubled; the convolution block is followed by a two-layer LSTM (Long Short-Term Memory Network) for sequence modeling, and finally a convolution layer with a kernel size of 7 and D output channels. As an example, setting C=32, B=4, stride S = (2,4,5,8), the time length of the input audio signal will be reduced to 1 / 320 of the original (2x4x5x8). For the activation function, ELU (Exponential Linear Unit) was selected as the nonlinear activation function, with , where α=1. As an example, the input of the encoder network is an audio waveform, assuming its length is t. After sampling, this audio waveform signal can be represented as a set of discrete sequence signals , where c i is the number of channels of the input audio, It is the number of audio samples of duration t in this audio data at a given sampling rate fs.

[0024] In this embodiment, if Figure 2 As shown, the composition of the decoder network is a mirror image of the encoder network. The decoder network uses transposed convolution instead of stride convolution and finally outputs reverse mono or stereo audio.

[0025] The architecture of the codec network in this embodiment is streaming, which is for the consideration of real-time audio transmission. On the encoder network side, the padding operation will only be placed before the first time step to ensure that the convolution kernel can correctly process the boundary part of the audio signal while avoiding unnecessary delays in streaming processing. For each convolution layer, the padding amount is equal to the size of the convolution kernel minus the size of the stride; all padding is only applied to the beginning of the segment, and unfilled data is retained at the end of each segment. In each convolution block, the stride setting ensures that a certain number of encoding steps can be generated when processing an audio segment. As an example, for a 24kHz audio signal, the encoder outputs a feature representation of 75 potential steps per second. In order to maintain real-time performance, the encoder network will generate a set of latent representations after each segment is processed and pass them to the subsequent quantization module. Corresponding to the encoder network, the decoder network also only applies padding at the beginning of the segment. Each time the decoder network processes a segment, it saves the end of the previous segment (overlapping with the beginning of the current segment) and decodes it together with the newly received latent representation to reduce boundary effects and ensure the continuity of the audio signal. The time step of each encoder network output corresponds to 320 sampling points of the input (a multiple of downsampling). When the input audio sampling rate is 24kHz, the audio length corresponding to each time step is approximately 13.33 milliseconds.

[0026] In this embodiment, the quantizer is a residual multi-stage vector quantizer, and the encoder network output audio latent representation z is compressed into discrete quantization indexes, i.e., codebook vectors, in the residual multi-stage vector quantizer. The quantization index represents a discrete value obtained after multi-stage quantization, wherein the residual multi-stage vector quantizer is composed of Nq layers of vector quantization layers cascaded together. Specifically, after the encoder network outputs the latent representation, this embodiment uses an improved residual vector quantization method to quantize it. The goal of the quantizer is to compress the output of the encoder network to a target bit rate R (unit: bit / second). The codebook size of a traditional vector quantizer grows exponentially with the number of bits allocated to each audio frame, which not only lacks feasibility, but also makes it difficult to dynamically adjust the bit rate. In an ordinary vector quantization, the input vector x is directly mapped to the closest codebook vector q, with .

[0027] This embodiment uses a residual multi-stage vector quantizer, and the quantizer is cascaded in Nq layers. First, the unquantized input vector x is processed by the first vector quantizer codebook and the quantization residual is calculated to obtain the first layer quantization result q 1 and the first residual r 1 ,have: ; ; Next, the residual r 1 Quantization is performed using a second vector quantizer codebook: ; ; This iteration is carried out until the predetermined number of quantization layers Nq is reached; the quantization results of each layer are combined to obtain the final quantization result ; The quantization codeword q is the result of the entire quantization process and represents the encoded signal; the final quantization feature .

[0028] As an example, assume the target encoding bit rate R = 6000bps, the step size is 320, the audio sampling rate fs = 24000hz, and the encoder network outputs 75 latent representation frames per second, which means that 6000 bits per second will be evenly distributed to 75 frames, and each frame needs to be allocated r = 6000 / 75 = 80 bits. If a normal vector quantizer solution is used, a size of 2 is required. 80 This is not realistic. In the residual vector quantizer, the calculation will be evenly distributed to each quantizer. Take Nq=8, and the codebook size for each quantizer is . For each target bit rate, the budgeted number of bits per frame r is obtained. The coding efficiency and computational complexity can be balanced by controlling Nq. A larger number of residual vector quantizer layers may achieve higher coding efficiency and coding quality, but will significantly increase computational complexity, while a lower Nq may result in a larger quantization error and poorer coding quality. More importantly, through residual vector quantization, an adaptive bit rate is achieved in the network. A higher Nq can bring higher bit rate and high quality audio, while a lower Nq can also reduce the bit rate.

[0029] This embodiment introduces a multi-layer quantization structure based on residual vector quantization in the audio encoding and decoding process. This structure can gradually approach the target audio signal in multiple quantization steps, so that high sound quality can be maintained at a low bit rate. Traditional vector quantization methods can usually only quantize the signal once, which can easily lead to a decrease in sound quality. The multi-layer quantization structure can better capture the complex details of the audio signal and reduce the quantization error, so that high sound quality can be maintained at a low bit rate. By adjusting the number of quantization layers and the number of quantization bits in each layer, flexible adaptation to different audio content and application scenarios can be achieved to meet different encoding requirements.

[0030] In this embodiment, the codec network also includes an entropy encoder located after the quantizer, and the entropy encoder is used to perform arithmetic encoding on the probability distribution of the quantization index to generate a compressed code stream. Specifically, after the residual vector is encoded, the present embodiment also introduces entropy coding technology to further compress the audio data. In residual vector quantization, the audio signal is compressed into discrete quantization indices, namely codebook vectors. These quantization indices represent discrete values ​​obtained after multi-level quantization. For these discrete values, a probability distribution model of the quantization index is established to estimate the probability of each index appearing. The present invention uses arithmetic coding to generate corresponding codes based on the probability distribution of the quantization index. The efficiency of entropy coding depends on the probability distribution of the quantization index. As an example, first, a probability model is constructed for the quantized symbol sequence. Assume that the symbol sequence is S={s1,s2,…,sn}, and the probability of each symbol si is obtained from the statistical results of historical data, expressed as P(si). The cumulative probability C(si) of each symbol is the sum of the probabilities of all symbols before symbol si, that is ; At the beginning of arithmetic coding, an initial interval [0,1] is defined. For each symbol si in the symbol sequence, according to its probability P(si) and cumulative probability C(si), the current interval [low, high] is gradually reduced to a smaller interval [low', high'], as follows: ; ; When all symbols are processed, the encoder selects the midpoint of the interval as the final encoding result. This value is represented by binary and is the final compressed code stream.

[0031] This embodiment introduces entropy coding technology to the encoder output and residual vector quantization results, and realizes adaptive adjustment of the bit rate by further compressing the quantized coding symbols. The entropy encoder dynamically allocates the number of bits according to the statistical characteristics of the input data, thereby further reducing the required bit rate while maintaining the sound quality.

[0032] In this embodiment, a fusion loss function of the codec network is pre-designed, and the fusion loss function includes: reconstruction loss, perception loss, residual vectorization loss and commitment loss. The codec network is trained end-to-end using the original audio data as input, and the model output is obtained through forward propagation. The loss value of the fusion loss function is calculated, and the loss value is transmitted back to each parameter in the codec network using a back-propagation algorithm. The parameters are updated to reduce the loss value, and the audio codec model is obtained. Specifically, the fusion loss function: , where L_rec represents reconstruction loss, L_percept represents perceptual loss, L_RVQ represents residual vector quantization loss, and L_commit represents commitment loss. The λ value corresponding to each loss represents the trade-off and balance between various indicators.

[0033] First, the reconstruction loss term consists of a time domain term and a frequency domain term. In the time domain, it is expected to minimize the L1 distance between the target audio and the compressed audio, that is, In the frequency domain, by calculating the Mel spectrograms of the target signal and the reconstructed signal at different time scales and performing a weighted linear combination of their L1 and L2 distances, we get , where S i (x) represents the Mel spectrum of the i-th scale, which is obtained by window size , the step length is It is calculated from the standardized short-time Fourier transform of , with 64 buckets, s represents the number of scales used; e={5,…,11} represents the set of scales, and α represents the set of scalar coefficients that balance the L1 and L2 distance loss terms. Set α i = 1.

[0034] The goal of the reconstruction loss is to ensure that the generated estimated audio is as close to the original audio as possible in both time and frequency. The final reconstruction loss is the weighted sum of the two: .

[0035] Perceptual loss, also known as discriminative loss, is used to optimize generative models. The decoder network of the present invention generates waveforms from compressed audio encodings, which is the generator. Unlike the reconstruction loss, which directly compares the differences between sample points, the perceptual loss optimizes the model by measuring the similarity between the generated audio and the target audio in the high-level feature space, trying to make the generated output perceptually closer to the target.

[0036] The perceptual loss of this embodiment is realized through the discriminator, which includes a time domain discriminator and a frequency domain discriminator. The time domain discriminator receives the time domain audio signal, and then passes through four layers of one-dimensional convolutional layers. The convolution kernel size of each layer is k=5, the number of channels is (32, 64, 128, 256), the stride is s=2, and the activation function uses LeakyReLU; the local features of the signal are gradually extracted through multiple layers of convolutional layers; in order to aggregate the features in the time dimension, a global average pooling layer is applied after the convolution layer, and finally a fully connected layer maps the output of the global pooling to a scalar. The frequency domain discriminator is based on short-time Fourier transform. First, it accepts a time domain signal and converts it into a frequency domain representation through short-time Fourier transform, which is: , where N=512 is the window length of STFT, H=256 is the jump step of STFT window, w(n) represents the window function, and Hanning window is used as the window function here: ; The obtained STFT time-frequency graph will pass through three layers of two-dimensional convolutional layers, each with a convolution kernel size of (3×3) and 32, 64 and 128 channels respectively; the convolutional layer is followed by a global average pooling layer, and finally input into a fully connected layer to map to a scalar. Whether it is a time domain or frequency domain discriminator, a binary cross entropy loss function is used during training to measure the perceptual difference between the discriminator's estimated audio and the real audio. There are: ; , where D represents the output of the discriminator for the time domain and frequency domain respectively (i.e., authenticity). Ultimately, the perceptual loss is the weighted sum of the time domain discriminator loss and the frequency domain discriminator loss: .

[0037] RVQ loss is the loss of residual vector quantization. The input signal x is generated by the encoder network to generate the initial feature representation z. As shown above, the residual of the first step of quantization is r 1 =ZQ 1 (z), the residual after the second step quantization is r 2 =r 1 -Q 2 (r 1 ), and so on, until the Nqth quantizer. The RVQ loss function is used to guide the quantization process and minimize the residual sum of squares after each level of quantization. The loss of each level is: , the total RVQ loss is the sum of the losses of all quantization levels: .

[0038] The purpose of commitment loss and RVQ loss is to make the vector quantization process constrain the feature representation of the encoder output as much as possible. Commitment loss prevents the output of the encoder from deviating from the quantized representation. In the process of vector quantization, the feature vector z generated by the encoder will eventually be mapped to a discrete codebook vector. The commitment loss is calculated by the L2 norm between the coded representation and the quantized representation, which is: ,in, is the residual vector before quantization of the i-th layer, Represents the result vector after quantization of the i-th layer. The final loss function is a fusion of reconstruction loss, perceptual loss, RVQ loss and commitment loss.

[0039] In order to optimize the performance of the audio codec, this embodiment integrates four key loss functions, namely reconstruction loss, perception loss, residual vector quantization loss and commitment loss, and introduces a loss balancer to jointly optimize the quality of the audio codec and the perceived effect of reconstruction. By introducing the balancer, the balanced contribution of several loss functions in model training is ensured, thereby stabilizing the optimization process of the model. The balance and fusion of several loss functions have improved the model in all dimensions of audio encoding and reconstruction (time domain fidelity, frequency domain accuracy, perceived quality, and quantization efficiency, etc.). The dynamically adjusted weight mechanism improves the adaptability of the model under different audio content and scenarios, ensuring performance stability and sound quality improvement in low bitrates or complex audio scenarios.

[0040] When training a network, different loss functions may produce different gradients. The scales and magnitudes of these gradients may vary greatly, which can easily cause instability in the training process and affect the convergence speed and performance of the model. To address this problem, this embodiment introduces a balancer to dynamically adjust the gradient of each loss term to balance their contribution to the model during the optimization process.

[0041] Specifically, the core of the balancer is to normalize the gradient of each loss term according to its scale, so that the weights λ between different loss terms are no longer affected by their natural scale. First, define the gradient of each loss term with respect to the model output , then, calculate the exponential moving average of the L2 norm of each gradient, denoted as , β is the attenuation factor of the exponential moving average process, which is used to smooth the change of the gradient; in each back propagation process, a reference norm R is used to standardize the gradient to obtain the balanced gradient: ,in, is the weight of the ith loss, is the sum of all loss term weights; this balanced gradient ensures that the gradient of each loss function is scaled according to its relative importance, etc.

[0042] In this embodiment, the audio to be processed is input into the audio codec model, and the reconstructed audio signal is output. The audio codec model designed in this embodiment is intended to adapt to a variety of audio scenarios. In order to verify its performance under different audio contents, including speech, music and general audio scenarios, this embodiment selects three corresponding representative data sets: in terms of speech, the DNS Challenge4 data set is selected, covering a variety of speech scenarios and different speakers; in terms of music, the Jamendo data set is used; in terms of general audio, the AudioSet data set is used.

[0043] The audio codec of the present invention is compared with the current mainstream Opus and EVS codecs, and the bit rate range of the evaluation is {1.5, 3, 6, 12} kbps, covering low, medium and high bit rate scenarios. The results are shown in Table 1 and Figure 3 As shown, it can be found that at a lower bit rate, the neural network structure of the present invention shows stronger compression capability and perceptual quality compared to the traditional codecs of Opus and EVS; at the same time, the data in Table 1 also show that the present invention has stable performance in different scene data sets, and the MUSHRA score in music and general audio scenes is even close to the quality of the reference standard audio, which shows that the present invention has a wide range of applicability. The subjective audio quality evaluation method of MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) is adopted, and the specific evaluation results are shown in Table 1: Table 1: MUSHRA evaluation results of the present invention, Opus, and EVS audio codecs, with the audio sampling rate being 24kHz

[0044] from Figure 3 It can also be seen that after applying entropy coding, the bit rate of the coding of the present invention is further reduced.

[0045] In addition to MUSHRA as a subjective evaluation, the present invention also introduces objective evaluation indicators, including ViSQOL (Virtual Speech Quality Objective Listener) and SI-SNR (Scale-Invariant Signal-to-Noise Ratio). ViSQOL is an objective indicator for measuring the perceived quality of audio. It obtains the sound quality score by calculating the spectral similarity of the audio signal, simulating the perception process of human hearing, and comparing the similarity between signals. It first performs a short-time Fourier transform on the audio signal to generate a spectrum, then calculates the similarity between the reference audio and the distorted audio in each time-frequency segment, and finally obtains the global similarity by weighted summation of the local similarities, and maps it to the MOS-LQO (Mean Opinion Score - Listening Quality Objective) range to generate the ViSQOL score. First, the perceptual similarity of the reference signal and the distorted signal is calculated using the logarithmic Mel spectrum representation, as follows: , where Dmel(t) is the Mel spectrum distance at the tth frame, M is the number of Mel bands, here M=40, Mref(m,t) and Mdist(m,t) are the Mel spectrum representations of the reference signal and the distorted signal, respectively, and m is the Mel band index. After obtaining the similarity, the similarity of each frame is converted into a ViSQOL score q(t) through a mapping function. The ViSQOL score is usually between 0 and 100, indicating the perceived quality of the signal. The mapping function is , where α is the scaling factor, here set α = 1. The final ViSQOL score is the weighted average of the similarity of each frame .

[0046] SI-SNR is another indicator to measure signal similarity, which focuses more on the structural similarity of the signal. First, align the reference signal and the estimated signal, and we have ,in is the reconstructed signal, s is the reference signal, and <> represents the inner product. Then the error signal is calculated, Finally, we get .

[0047] The present invention was tested at a bit rate of 6kbps and compared with the traditional Opus and EVS codecs. The results are shown in Table 2. The present invention is significantly better than the traditional Opus and EVS codecs in terms of SI-SNR and ViSQOL indicators, indicating that it has a greater advantage in the fidelity of audio signals. In general, the present invention can provide audio quality that is better than traditional codecs under low bit rate conditions, which shows that the present invention can be applied to application scenarios with higher sound quality requirements while ensuring audio streaming transmission.

[0048] Table 2: Objective evaluation results of the present invention, Opus, and EVS audio codecs at a bit rate of 6 kbps

[0049] The above describes the audio codec method based on a neural network in an embodiment of the present invention. The following describes the audio codec device based on a neural network in an embodiment of the present invention. Figure 4 In one embodiment of the present invention, an audio codec device based on a neural network includes: The network construction module 10 is used to pre-construct a codec network, wherein the codec network includes: an encoder network for receiving raw audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal The decoder network of A loss function design module 20 is used to pre-design a fusion loss function of the codec network, wherein the fusion loss function includes: reconstruction loss, perception loss, residual vectorization loss and commitment loss, and to perform end-to-end training on the codec network using the original audio data as input, obtain the model output through forward propagation, calculate the loss value of the fusion loss function, and use the back-propagation algorithm to transmit the loss value back to each parameter in the codec network, update the parameters to reduce the loss value, and obtain the audio codec model; The output module 30 is used to input the audio to be processed into the audio coding model and output a reconstructed audio signal.

[0050] Based on the same idea as the method in the above-mentioned embodiment, the device provided in the present application can implement the method in the above-mentioned embodiment. For the convenience of explanation, the structural diagram of the device embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer modules than shown in the figure, or a combination of certain modules, or different module arrangements.

[0051] Figure 4 The neural network-based audio codec device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The neural network-based audio codec device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0052] Figure 5 1 is a schematic diagram of the structure of an audio codec device based on a neural network provided by an embodiment of the present invention. The audio codec device 100 based on a neural network may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 11 (for example, one or more processors) and a memory 12, and one or more storage media 13 (for example, one or more mass storage devices) storing application programs 133 or data 132. Among them, the memory 12 and the storage medium 13 can be short-term storage or permanent storage. The program stored in the storage medium 13 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the audio codec device 100 based on the neural network. Furthermore, the processor 11 can be configured to communicate with the storage medium 13 to execute a series of instruction operations in the storage medium 13 on the audio codec device 100 based on the neural network.

[0053] The neural network-based audio codec device 100 may further include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The device structure shown does not constitute a limitation on the neural network-based audio codec device 100 , and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the neural network-based audio encoding and decoding method.

[0055] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0056] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0057] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio encoding and decoding method based on a neural network, characterized in that: Includes steps: A codec network is pre-built, the codec network comprising: an encoder network for receiving raw audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal The decoder network of Pre-designing a fusion loss function of the codec network, the fusion loss function includes: reconstruction loss, perception loss, residual vectorization loss and commitment loss, using the original audio data as input to perform end-to-end training on the codec network, obtaining the model output through forward propagation, calculating the loss value of the fusion loss function, and using a back-propagation algorithm to propagate the loss value back to each parameter in the codec network, updating the parameters to reduce the loss value, and obtaining an audio codec model; The audio to be processed is input into the audio coding and decoding model, and a reconstructed audio signal is output.

2. The neural network-based audio encoding and decoding method according to claim 1, characterized in that: The encoder network includes a first convolutional layer with C channels and a convolution kernel size of 7, B convolutional blocks, an LSTM for sequence modeling, and a second convolutional layer with a convolution kernel size of 7 and an output channel of D; wherein each convolutional block consists of a residual unit, followed by a downsampling layer consisting of convolutions with a stride of S, the kernel size K of the downsampling layer is twice the stride S, and the residual unit contains two convolutions with a kernel size of 3 and 1 jump connection.

3. The neural network-based audio encoding and decoding method according to claim 1, characterized in that: The quantizer is a residual multi-stage vector quantizer, and the encoder network output audio latent representation z is compressed into a discrete quantization index, i.e., a codebook vector, in the residual multi-stage vector quantizer. The quantization index represents a discrete value obtained after multi-level quantization, wherein the residual multi-stage vector quantizer is composed of Nq layers of vector quantization layers cascaded together; the unquantized input vector x is processed by the first vector quantizer codebook and the quantization residual is calculated to obtain the quantization result q1 and the first residual r1 of the first layer; the residual r1 is quantized by the second vector quantizer codebook to obtain the quantization result q2 and the second residual r2 of the second layer; quantization is iterated until the predetermined number of quantization layers Nq is reached; the quantization results of each layer are combined to obtain the final quantization result , the quantization codeword q is the result of the entire quantization process, representing the encoded signal, the final quantization feature .

4. The neural network-based audio encoding and decoding method according to claim 3, characterized in that: The codec network further includes an entropy encoder located after the quantizer, and the entropy encoder is used to perform arithmetic encoding on the probability distribution of the quantization index to generate a compressed code stream.

5. The neural network-based audio encoding and decoding method according to claim 4, characterized in that: The entropy encoder is used to perform arithmetic encoding on the probability distribution of the quantization index to generate a compressed code stream, including the steps of: A probability model is constructed for the quantized symbol sequence. Assuming that the symbol sequence is S={s1,s2,…,sn}, the probability of each symbol si is obtained from the statistical results of historical data, expressed as P(si), then the cumulative probability C(si) of each symbol is the sum of the probabilities of all symbols before symbol si, that is, ； At the beginning of arithmetic coding, an initial interval [0,1] is defined. For each symbol si in the symbol sequence, according to its probability P(si) and cumulative probability C(si), the current interval [low, high] is gradually reduced to a smaller interval [low', high'], as follows: ； ； When all symbols are processed, the midpoint of the interval is selected as the final encoding result. This value is represented by binary, which is the final compressed code stream.

6. The neural network-based audio encoding and decoding method according to claim 1, characterized in that: The fusion loss function: , where L_rec represents reconstruction loss, L_percept represents perceptual loss, L_RVQ represents residual vector quantization loss, and L_commit represents commitment loss. The λ value corresponding to each loss represents the trade-off and balance between various indicators.

7. The neural network-based audio encoding and decoding method according to claim 1, characterized in that: During the end-to-end training of the encoder-decoder network, a balancer is introduced to dynamically adjust the gradient of each loss term in the fusion loss function, thereby balancing the contribution of each loss term to the model during the optimization process.

8. An audio codec device based on a neural network, characterized in that: include: A network construction module is used to pre-construct a codec network, wherein the codec network includes: an encoder network for receiving raw audio data input and outputting an audio latent representation z, a network for compressing the audio latent representation z and outputting a compressed latent representation z q quantizer, and a quantizer for converting the compressed latent representation z q Reconstruction into time domain signal The decoder network of A loss function design module is used to pre-design a fusion loss function of the codec network, wherein the fusion loss function includes: reconstruction loss, perception loss, residual vectorization loss and commitment loss, and to perform end-to-end training on the codec network using the original audio data as input, obtain the model output through forward propagation, calculate the loss value of the fusion loss function, and use the back-propagation algorithm to transmit the loss value back to each parameter in the codec network, update the parameters to reduce the loss value, and obtain the audio codec model; The output module is used to input the audio to be processed into the audio coding model and output a reconstructed audio signal.

9. An audio codec device based on a neural network, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the neural network-based audio encoding and decoding method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the neural network-based audio encoding and decoding method as described in any one of claims 1 to 7 are implemented.

Images