A decoder, a system, a method and a storage medium for audio discretization reconstruction

Abstract

The present invention provides a decoder for audio discretization reconstruction, comprising: a CNN layer, for inputting discretized features of audio, and performing preliminary feature restoration on the discretized features to generate restored features; a STFT prediction layer, for predicting Fourier spectrum coefficients of the restored features at multiple frequency domain scales; using the predicted Fourier spectrum coefficients, performing spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales; an ISTFT layer, for performing inverse short-time Fourier transform on the STFT spectrum graphs at multiple frequency domain scales, converting the STFT spectrum graphs back to the time domain, and generating multiple complete candidate audios; a fusion layer, for fusing multiple candidate audios to generate reconstructed audio. When the decoder of the present invention performs audio reconstruction, distortion and noise in the reconstructed audio can be avoided, high-fidelity audio can be reconstructed, and the quality of the reconstructed audio can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a decoder, a system and method for audio discretization reconstruction, and a storage medium. Background Art

[0002] Audio discretization reconstruction is to map the continuous audio feature sequence to a finite discrete code space, convert the continuous audio feature sequence into discrete features (discrete codes), characterize and reconstruct the continuous audio feature sequence, so as to achieve more efficient audio compression, generation and transmission.

[0003] Currently, audio discretization solves the combination of audio modality and large language models. Complex and lengthy audio cannot be directly input into the large model for learning. After audio discretization and compression, the audio can be well combined with the large model, making the end-to-end large model that can directly understand and generate audio truly implemented.

[0004] In the prior art, an audio discretization reconstruction system mainly includes an encoder, a quantizer, and a decoder.

[0005] The encoder compresses the high-dimensional raw audio into a low-dimensional feature space and extracts a continuous audio feature sequence, usually using a convolutional neural network (CNN) or a Transformer model. The encoder first extracts the feature information of the audio, such as time-frequency features. The extracted features are a continuous potential representation, i.e., a continuous audio feature sequence, before being transmitted to the quantizer.

[0006] The quantizer is the core part of the existing audio discretization and reconstruction system. It discretizes the continuous audio feature sequence output by the encoder, maps the continuous audio feature sequence to the discrete space by finding the discrete code vector closest to the feature (selected from the code table), and generates discretized features.

[0007] The decoder uses the discretized features obtained by the quantizer to reconstruct the audio.

[0008] The existing audio discretization and reconstruction system compresses, discretizes and reconstructs the audio through an encoder, quantizer and decoder. The decoder, like the encoder, uses a convolutional neural network (CNN) decoder, a deconvolution decoder or a decoder based on an autoregressive structure to take the discretized features as input and reproduce the original audio waveform or spectrum information.

[0009] The existing audio discretization reconstruction system can map the continuous audio feature sequence to a limited discrete code space, and characterize and reconstruct the continuous audio feature sequence. However, the current audio discretization reconstruction system still has the following shortcomings:

[0010] (1) The encoder and decoder basically follow a mirror-symmetric structure. However, the feature extraction and reconstruction of audio may have different emphases in parameters. The symmetric structure of the encoder and decoder cannot meet the needs of feature extraction and reconstruction at the same time, thus affecting the audio feature extraction of the encoder and the audio reconstruction of the decoder.

[0011] For example, the mirror-symmetric structure of the decoder and encoder. This standard structure generally uses a series of timing operations due to the need for upsampling, such as dilated convolution (to increase the temporal receptive field) and transformed convolution (also called deconvolution, used for upsampling), to increase the receptive field and gradually upsample from features to waveforms. Such a structure can easily produce aliasing effects, that is, insufficient sampling can be performed in the frequency domain, causing higher frequency components to be "folded" into a lower frequency range and confused with the actual low-frequency components, resulting in distortion and noise during audio reconstruction, affecting the quality of audio reconstruction.

[0012] (2) The existing decoder and encoder have a mirror-symmetric structure. Whether the decoder directly reconstructs the audio or restores the characteristic spectrogram (such as frequency spectrum or mel-spectrogram), the accuracy of audio reconstruction using discrete features is relatively limited. In order to meet the requirements of high compression rate, the quality of audio reconstruction is often limited by the excessively high downsampling rate of the encoder and the limited codebook parameters of the quantizer. When reconstructing audio, some details of the reconstructed audio are distorted, and high-fidelity audio cannot be generated. Summary of the invention

[0013] The present invention proposes a decoder, a system, a method and a storage medium for audio discretization reconstruction to solve the technical problem that in the existing audio discretization reconstruction, some details of the reconstructed audio are distorted and high-fidelity audio cannot be generated.

[0014] One aspect of the present invention is to provide a decoder for audio discretization reconstruction, the decoder comprising: a CNN layer, for inputting discretized features of audio, and performing preliminary feature restoration on the discretized features to generate restored features;

[0015] The STFT prediction layer is used to predict the Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients;

[0016] Using the predicted Fourier spectrum coefficients, performing spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales;

[0017] An ISTFT layer, for performing an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, converting the STFT spectrograms back to the time domain, and generating multiple complete candidate audios;

[0018] The fusion layer is used to fuse the multiple candidate audios to generate reconstructed audio.

[0019] In a preferred embodiment, the STFT prediction layer includes a plurality of stacked depthwise convolutional layers and a plurality of point-by-point convolutional layers.

[0020] In a preferred embodiment, the STFT prediction layer predicts Fourier spectrum coefficients for the restored features at a frequency domain scale; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients;

[0021] The predicted Fourier spectrum coefficients are used to perform spectrum restoration on the restored features at a frequency domain scale to generate an STFT spectrum diagram of the restored features at a frequency domain scale.

[0022] In a preferred embodiment, the fusion layer fuses the multiple candidate audios by using any one of weighted averaging, low-frequency retention and high-frequency enhancement rules, or fully connected layer prediction.

[0023] Another aspect of the present invention is to provide a system for audio discretization reconstruction, the system comprising an encoder, a quantizer, and a decoder provided by the present invention.

[0024] In a preferred embodiment, the encoder is used to obtain original audio;

[0025] Downsampling the original audio to extract a continuous audio feature sequence of the original audio;

[0026] The quantizer is used to discretize the extracted continuous audio feature sequence to generate discrete audio features.

[0027] Another aspect of the present invention is to provide a method for audio discretization reconstruction, the method comprising the following method steps:

[0028] S1. Get the original audio;

[0029] Downsampling the original audio to extract a continuous audio feature sequence of the original audio;

[0030] S2, discretizing the extracted continuous audio feature sequence to generate discrete audio features;

[0031] S3, performing preliminary feature restoration on the discretized features to generate restored features;

[0032] S4. predicting Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients;

[0033] Using the predicted Fourier spectrum coefficients, performing spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales;

[0034] S5, performing an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, converting the STFT spectrograms back to the time domain, and generating multiple complete candidate audios;

[0035] S6. Fusing the plurality of candidate audios to generate reconstructed audio, thereby completing audio discretization reconstruction.

[0036] In a preferred embodiment, in step S4, the Fourier spectrum coefficients are predicted for the restored features at a frequency domain scale; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients;

[0037] The predicted Fourier spectrum coefficients are used to perform spectrum restoration on the restored features at a frequency domain scale to generate an STFT spectrum diagram of the restored features at a frequency domain scale.

[0038] In a preferred embodiment, in step S6, the plurality of candidate audios are fused by using any one of weighted averaging, low frequency retention and high frequency enhancement rules, or fully connected layer prediction.

[0039] Another aspect of the present invention is to provide a computer storage medium, wherein the computer storage medium is used to store computer execution instructions, and the computer execution instructions are used to execute a method for audio discretization reconstruction provided by the present invention.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention proposes a decoder, a system, a method and a storage medium for audio discretization reconstruction. The decoder performs preliminary feature restoration on discretization features to generate restored features; predicts Fourier spectrum coefficients on the restored features at multiple frequency domain scales, uses the predicted Fourier spectrum coefficients to perform spectrum recovery on the restored features at multiple frequency domain scales, and generates STFT spectrum graphs of the restored features at multiple frequency domain scales. The STFT spectrum graphs at multiple frequency domain scales are inversely short-time Fourier transformed, and the STFT spectrum graphs are converted back to the time domain to generate multiple complete candidate audios. Finally, the multiple candidate audios are fused to generate reconstructed audio, which can achieve high-quality audio reconstruction under the premise of ensuring a high compression rate of the encoder.

[0042] The present invention proposes a decoder, a system, a method and a storage medium for audio discretization reconstruction. When the decoder performs audio reconstruction, distortion and noise in the reconstructed audio can be avoided, high-fidelity audio can be reconstructed, and the quality of the reconstructed audio can be improved.

[0043] The decoder, audio discretization reconstruction system, method and storage medium proposed in the present invention can significantly reduce the storage requirements of audio data while maintaining a high reconstruction quality, and are suitable for scenarios with limited communication and storage resources, such as voice calls, audio streaming media, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the prior art description. Obviously, the drawings described below are some implementations of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a structural block diagram of a system for audio discretization reconstruction of the present invention.

[0046] Figure 2 It is a structural block diagram of the decoder of the present invention.

[0047] Figure 3 The present invention is a flowchart of a method for audio discretization reconstruction. DETAILED DESCRIPTION

[0048] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0049] Combination Figure 1 and Figure 2According to an embodiment of the present invention, a system for audio discretization reconstruction includes an encoder (Encoder) 100, a quantizer (Quantizer) 200, and a decoder (Decoder) 300.

[0050] The encoder 100 is used to obtain original audio, downsample the original audio, and extract a continuous audio feature sequence of the original audio.

[0051] For example, an original audio with a duration of 1s and a sampling rate of 16,000 can be compressed from 16,000 sampling rate to 50 sampling rate by downsampling to 320 and extracting the continuous audio feature sequence of the original audio.

[0052] The encoder 100 of the present invention adopts an existing conventional discretization encoder-decoder encoder structure, for example, using an encoder (Encoder) in Encodec (discretization encoder-decoder).

[0053] The quantizer 200 is used to discretize the extracted continuous audio feature sequence to generate discrete audio features.

[0054] For example, the extracted continuous audio feature sequence is discretized by setting the codebook form and the number of codebooks for the quantizer 200 to generate discretized audio features. For example, the extracted continuous audio feature sequence is discretized using the Residual VQ form and 8 codebooks.

[0055] The decoder 300 is used to perform audio reconstruction on the discretized features of the audio generated by the quantizer 200 to generate reconstructed audio.

[0056] Specifically, Figure 2 As shown, the decoder 300 for audio discretization reconstruction of the present invention includes: a CNN layer (convolutional neural network layer) 301, which is used to input the discretization features of the audio, and perform preliminary feature restoration on the discretization features to generate restored features.

[0057] The CNN layer (convolutional neural network layer) 301 of the present invention adopts a 1D convolutional neural network layer to perform preliminary feature restoration on discretized features.

[0058] The STFT prediction layer (short-time Fourier transform prediction layer) 302 is used to predict Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients.

[0059] The predicted Fourier spectrum coefficients are used to perform spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales.

[0060] The STFT prediction layer 302 of the present invention includes multiple stacked depth convolution layers and multiple point-by-point convolution layers to achieve the simultaneous generation of local and global information of the STFT spectrum graph.

[0061] like Figure 2 As shown, in this embodiment, the STFT spectrum diagrams of the generated restored features at multiple frequency domain scales are: STFT spectrum diagram a, STFT spectrum diagram b, STFT spectrum diagram c, ...

[0062] The ISTFT layer (inverse short-time Fourier transform layer) 303 is used to perform an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, convert the STFT spectrograms back to the time domain, and generate multiple complete candidate audios.

[0063] like Figure 2 As shown, in this embodiment, exemplarily, the ISTFT layer 303 performs an inverse short-time Fourier transform on the STFT spectrum diagrams at multiple frequency domain scales: STFT spectrum diagram a, STFT spectrum diagram b, STFT spectrum diagram c, ..., and converts the STFT spectrum diagram back to the time domain, and generates multiple complete candidate audios: candidate audio a, candidate audio b, candidate audio c, ...

[0064] The fusion layer 304 is used to fuse multiple candidate audios to generate reconstructed audio.

[0065] like Figure 2 As shown, in this embodiment, fusion layer 304 exemplarily fuses multiple candidate audios: candidate audio a, candidate audio b, candidate audio c, ..., to generate reconstructed audio.

[0066] In some embodiments, the fusion layer 304 fuses multiple candidate audios by weighted averaging, low frequency preservation and high frequency enhancement rule, or full connection layer prediction. Preferably, the fusion layer 304 fuses multiple candidate audios by weighted averaging.

[0067] The decoder 300 of the present invention does not adopt mirror symmetry with the encoder 200. After the discretized features generated by the quantizer 200 enter the decoder 300, they are restored to restored features after preliminary features, and Fourier spectrum coefficients are predicted for the restored features at multiple frequency domain scales. The predicted Fourier spectrum coefficients are used to perform spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales. The STFT spectrum graphs at multiple frequency domain scales are subjected to inverse short-time Fourier transform, and the STFT spectrum graphs are converted back to the time domain to generate multiple complete candidate audios. Finally, the multiple candidate audios are fused to generate reconstructed audio, which can achieve high-quality audio reconstruction under the premise of ensuring the high compression rate of the encoder 100.

[0068] In some embodiments, considering the limited computing resources and real-time efficiency of some scenarios, the STFT prediction layer 302 predicts the Fourier spectrum coefficients of the restored features at a frequency domain scale, wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients. Using the predicted Fourier spectrum coefficients, the restored features are spectrally restored at a frequency domain scale to generate an STFT spectrum graph of the restored features at a frequency domain scale. The STFT spectrum graph at a frequency domain scale (single scale) may affect the details of the reconstructed audio, but can improve the efficiency of audio reconstruction.

[0069] It should be noted that the audio discretization reconstruction system provided by the present invention needs to be trained before audio reconstruction. Specifically, a large amount of original audio data is collected and preprocessed at a unified sampling rate, and then input into the encoder 100, and the reconstructed audio output by the decoder 300 is used as the training target to train the audio discretization reconstruction system of the present invention.

[0070] like Figure 3 As shown, according to an embodiment of the present invention, a method for audio discretization reconstruction is provided, comprising the following method steps:

[0071] Step S1, obtaining original audio.

[0072] The original audio is downsampled and a continuous audio feature sequence of the original audio is extracted.

[0073] Step S2: discretize the extracted continuous audio feature sequence to generate discrete audio features.

[0074] Step S3: Perform preliminary feature restoration on the discretized features to generate restored features.

[0075] Step S4: predicting Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients;

[0076] The predicted Fourier spectrum coefficients are used to perform spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales.

[0077] In some embodiments, for the restored feature, Fourier spectrum coefficients are predicted at a frequency domain scale, wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients. Using the predicted Fourier spectrum coefficients, spectrum restoration is performed on the restored feature at a frequency domain scale to generate an STFT spectrum graph of the restored feature at a frequency domain scale.

[0078] Step S5: performing an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, converting the STFT spectrograms back to the time domain, and generating multiple complete candidate audios.

[0079] Step S6: fuse multiple candidate audios to generate reconstructed audio, thereby completing audio discretization reconstruction.

[0080] In some embodiments, multiple candidate audios are fused by using any of weighted averaging, low-frequency retention and high-frequency enhancement rules, or fully connected layer prediction.

[0081] According to an embodiment of the present invention, a computer storage medium is provided for storing computer-executable instructions. The computer-executable instructions are used to execute a method for audio discretization reconstruction provided by the present invention.

[0082] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A decoder for audio discretization reconstruction, characterized in that: The decoder comprises: a CNN layer, which is used to input the discretized features of the audio, and perform preliminary feature restoration on the discretized features to generate restored features; An STFT prediction layer, including a plurality of stacked depth convolution layers and a plurality of point-by-point convolution layers, is used to predict Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients; Using the predicted Fourier spectrum coefficients, performing spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales; An ISTFT layer, for performing an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, converting the STFT spectrograms back to the time domain, and generating multiple complete candidate audios; The fusion layer is used to fuse the multiple candidate audios to generate reconstructed audio.

2. The decoder according to claim 1, characterized in that The STFT prediction layer predicts Fourier spectrum coefficients for the restored features at a frequency domain scale; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients; The predicted Fourier spectrum coefficients are used to perform spectrum restoration on the restored features at a frequency domain scale to generate an STFT spectrum diagram of the restored features at a frequency domain scale.

3. The decoder according to claim 1, characterized in that The fusion layer fuses the multiple candidate audios by using any one of weighted averaging, low-frequency retention and high-frequency enhancement rules, or full-connection layer prediction.

4. A system for audio discretization reconstruction, characterized in that: The system comprises an encoder, a quantizer, and a decoder as claimed in any one of claims 1 to 3.

5. The system according to claim 4, characterized in that The encoder is used to obtain original audio; Downsampling the original audio to extract a continuous audio feature sequence of the original audio; The quantizer is used to discretize the extracted continuous audio feature sequence to generate discrete audio features.

6. A method for audio discretization reconstruction, characterized in that: The method comprises the following steps: S1. Get the original audio; Downsampling the original audio to extract a continuous audio feature sequence of the original audio; S2, discretizing the extracted continuous audio feature sequence to generate discrete audio features; S3, performing preliminary feature restoration on the discretized features to generate restored features; S4. predicting Fourier spectrum coefficients at multiple frequency domain scales for the restored features; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients; Using the predicted Fourier spectrum coefficients, performing spectrum recovery on the restored features at multiple frequency domain scales to generate STFT spectrum graphs of the restored features at multiple frequency domain scales; S5, performing an inverse short-time Fourier transform on the STFT spectrograms at multiple frequency domain scales, converting the STFT spectrograms back to the time domain, and generating multiple complete candidate audios; S6. Fusing the plurality of candidate audios to generate reconstructed audio, thereby completing audio discretization reconstruction.

7. The method according to claim 6, characterized in that In step S4, the Fourier spectrum coefficients are predicted for the restored features at a frequency domain scale; wherein the predicted Fourier spectrum coefficients include amplitude spectrum coefficients and phase spectrum coefficients; The predicted Fourier spectrum coefficients are used to perform spectrum restoration on the restored features at a frequency domain scale to generate an STFT spectrum diagram of the restored features at a frequency domain scale.

8. The method according to claim 6, characterized in that In step S6, the plurality of candidate audios are fused by using any one of weighted averaging, low-frequency retention and high-frequency enhancement rules, or full-connected layer prediction.

9. A computer storage medium, characterized in that The computer storage medium is used to store computer-executable instructions, and the computer-executable instructions are used to execute the method of any one of claims 6 to 8.

Images