Audio reconstruction method, device, equipment and storage medium thereof
By extending the codebook space with a single quantizer and an improved decoder, combined with adversarial training, the problem of insufficient availability of existing audio reconstruction models is solved and high-quality audio reconstruction is achieved.
Patent Information
- Application Number
- CN202411756907.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2044-12-02
AI Technical Summary
The existing end-to-end discrete codec model has a large room for improvement in audio reconstruction, resulting in insufficient usability of the reconstructed audio.
A single quantizer and an improved decoder with an extended codebook space are used, combined with adversarial training, and encoding and decoding processing is performed through an audio encoder, a single quantizer and an improved decoder, and audio is reconstructed using single-sideband spectrum and inverse Fourier transform.
Improved the usability of audio reconstruction to ensure the realism and quality of reconstructed audio.
Smart Images
Figure CN119649849B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of R&D, design, and audio processing technology, and is applied to pre-processing scenarios of audio reconstruction and audio synthesis, and in particular to an audio reconstruction method, apparatus, device, and storage medium thereof. Background Art
[0002] In recent years, large-scale language models have achieved remarkable success in generative tasks involving multi-speaker speech synthesis, music generation, and audio generation. Furthermore, integrating speech modalities into large, multimodal unified models has also attracted widespread attention. These successes can be largely attributed to the use of discrete acoustic codec representations generated by neural codec models.
[0003] These discrete acoustic codec models bridge the gap between continuous speech and tag-based language models, discretizing high-rate audio signals into a limited set of tags, allowing the integration of audio data into large multimodal models. Most end-to-end discrete codec models typically adopt a three-stage structure consisting of an encoder, a residual vector quantization (RVQ) module, and a decoder. However, this three-stage structure still has significant room for improvement in audio reconstruction to achieve high usability of the reconstructed audio. Summary of the Invention
[0004] The purpose of the embodiments of the present application is to propose an audio reconstruction method, apparatus, device and storage medium thereof to improve audio reconstruction and achieve high availability of reconstructed audio.
[0005] In order to solve the above technical problems, the embodiments of the present application provide an audio reconstruction method, which adopts the following technical solutions:
[0006] An audio reconstruction method comprises the following steps:
[0007] Get the original audio stream;
[0008] Inputting the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream;
[0009] The latent feature representation is input into a single quantizer with an expanded codebook space size, and the latent feature representation is quantized to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is 2 10 Expand to 2 12 ;
[0010] Decoding the discretized audio feature representation using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module, and a convolutional block;
[0011] Converting the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum diagram;
[0012] An inverse Fourier transform is performed according to the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream.
[0013] Furthermore, the step of inputting the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream specifically includes:
[0014] Performing an initial convolution operation on the input original audio stream using the 1D convolution layer in the audio encoder to extract initial audio features from the original audio stream;
[0015] Performing downsampling and filtering processing on the initial audio features using a convolution block in the audio encoder, and obtaining key audio features corresponding to a target frequency according to a configuration of a sampling step size;
[0016] The key audio features are represented as potential features corresponding to the original audio stream.
[0017] Furthermore, before executing the step of obtaining the original audio stream, the method further includes:
[0018] Acquire an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object;
[0019] Processing the input audio stream sequentially through the audio encoder, the single quantizer, and the improved decoder to obtain reconstructed audio corresponding to the input audio stream;
[0020] The authenticity of the reconstructed audio is estimated using a preset discriminator until the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold, and the training of the audio encoder, the single quantizer and the improved decoder is completed.
[0021] Furthermore, the preset discriminator includes a multi-period discriminator, a multi-resolution discriminator, and a multi-scale discriminator. The step of using the preset discriminator to perform authenticity estimation on the reconstructed audio specifically includes:
[0022] Step 401: Using the multi-period discriminator, the multi-resolution discriminator, and the multi-scale discriminator, a quantization loss value, a Mel-spectrum reconstruction loss value, an adversarial loss value, and a feature matching loss value corresponding to the reconstructed audio during the reconstruction process are obtained.
[0023] In step 402, the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value are used as authenticity estimation factors, and combined with a preset comprehensive calculation method, the authenticity estimation is calculated.
[0024] Furthermore, before executing the step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as the authenticity estimation factors and combining them with a preset comprehensive calculation method to calculate the authenticity estimation, the method further includes:
[0025] Obtain loss requirement conditions pre-set for the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value, respectively, wherein the loss requirement condition includes that the loss value does not exceed a set loss threshold;
[0026] After step 401, the following steps are further included:
[0027] Comparing the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirements;
[0028] If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, then performing model parameter tuning processing on the audio encoder, the single quantizer, and the improved decoder until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions;
[0029] If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirements, step 402 is directly executed.
[0030] Furthermore, the step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as authenticity estimation factors, and combining a preset comprehensive calculation method to calculate the authenticity estimation, specifically includes:
[0031] Obtaining dynamic floating weights pre-set for the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value;
[0032] According to the dynamic floating weights, current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point are obtained respectively;
[0033] Based on the current weights corresponding to the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value, and feature matching loss value at the current training time point, a weighted sum algorithm is used to calculate the total loss value corresponding to the current training time point;
[0034] The total loss value is used to obtain the corresponding authenticity estimate at the current training time point, wherein the total loss value and the authenticity estimate are in inverse proportion.
[0035] Furthermore, after executing the step of obtaining the corresponding truth value at the current training time point using the total loss value, the method further includes:
[0036] Determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold;
[0037] If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, the training is continued, and the latest weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the latest training time point are obtained in an iterative calculation manner, and
[0038] Based on the latest weights, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding truthfulness estimate at the latest training time point is obtained through the total loss value;
[0039] The training of the audio encoder, the single quantizer and the improved decoder is completed until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold.
[0040] In order to solve the above technical problems, the embodiment of the present application also provides an audio reconstruction device, which adopts the following technical solutions:
[0041] An audio reconstruction device, comprising:
[0042] The original audio stream acquisition module is used to obtain the original audio stream;
[0043] An audio encoding processing module, configured to input the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream;
[0044] The quantization processing module is used to input the potential feature representation into a single quantizer with an expanded codebook space size, and quantize the potential feature representation to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is 2 10 Expand to 2 12 ;
[0045] A decoding processing and real-valued signal acquisition module, configured to decode the discretized audio feature representation using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module, and a convolutional block;
[0046] a signal spectrum conversion module, configured to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum;
[0047] The inverse Fourier transform module is used to perform inverse Fourier transform according to the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream.
[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0049] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned audio reconstruction method when executing the computer-readable instructions.
[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0051] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the audio reconstruction method described above.
[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0053] The audio reconstruction method described in the embodiment of the present application is as follows: obtaining the original audio stream; inputting it into a preset audio encoder to obtain the corresponding potential feature representation; inputting the potential feature representation into a single quantizer with an expanded codebook space size, quantizing the potential feature representation to obtain a discretized audio feature representation; using an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; using a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performing an inverse Fourier transform based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0056] Figure 2 is a flowchart of an embodiment of an audio reconstruction method according to the present application;
[0057] Figure 3 This is a flowchart of a specific embodiment of adversarial generative training in the audio reconstruction method described in this application;
[0058] Figure 4 is a flowchart of a specific embodiment of performing realism estimation on the reconstructed audio in the audio reconstruction method described in this application;
[0059] Figure 5 This is a flowchart of a specific embodiment of determining the model parameter tuning process in the audio reconstruction method described in this application;
[0060] Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 402 is shown;
[0061] Figure 7 is a flowchart of a specific embodiment of iteratively calculating the total loss value in the audio reconstruction method described in this application;
[0062] Figure 8is a structural diagram of an embodiment of an audio reconstruction device according to the present application;
[0063] Figure 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0065] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0067] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.
[0068] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0069] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.
[0070] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .
[0071] It should be noted that the audio reconstruction method provided in the embodiment of the present application is generally executed by a server, and accordingly, the audio reconstruction device is generally set in the server.
[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.
[0073] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio reconstruction method according to the present application. The audio reconstruction method comprises the following steps:
[0074] Step 201: Obtain the original audio stream.
[0075] In this embodiment, the original audio stream includes an audio stream to be reconstructed, for example, an audio stream of a real speaker in a speech synthesis scenario. By reconstructing the audio stream of the real speaker, the imitation speech of the target object can be subsequently synthesized.
[0076] Step 202: Input the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream.
[0077] In this embodiment, the audio encoder selects an encoder in an end-to-end transformer speech processing architecture model, for example, an encoder in SoundStream and an encoder in Encodec, where SoundStream and Encodec are both audio codecs of end-to-end architecture.
[0078] Specifically, the original audio stream is input into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream, that is, the audio signal is downsampled in the time domain by the audio encoder to obtain compressed audio frames.
[0079] Step 203: Input the latent feature representation into a single quantizer with an expanded codebook space size, and perform quantization processing on the latent feature representation to obtain a discretized audio feature representation.
[0080] The codebook space size of the single quantizer is 2 10 Expand to 2 12 .
[0081] In this embodiment, each compressed audio frame is quantized by the single quantizer to obtain a discrete audio feature representation. Compared with the existing processing method, the present application no longer uses a series of quantizers, but uses a single quantizer. In order to ensure that the codebook space of the single quantizer meets the quantization processing requirements and all speech representations are compressed into the codebook space of the single quantizer, the codebook space size of the single quantizer is first expanded, avoiding the use of a series of quantizers and performing a complex cross-channel dimensional hierarchical design on the quantizer.
[0082] In step 204, the discretized audio feature representation is decoded using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module, and a convolutional block.
[0083] In this embodiment, the standard decoder, which mirrors the audio encoder, is no longer used for decoding. This standard approach requires a series of dilated convolutions to expand the receptive field and transposed convolutions to sequentially upsample the feature sequence to a waveform, which is susceptible to aliasing artifacts. Here, the improved decoder is considered to be composed solely of one-dimensional convolutional layers, attention modules, and convolution blocks.
[0084] Specifically, during the decoding process, the discretized audio feature representation is input into the one-dimensional convolution layer, the attention module and the convolution block. The purpose of the convolution block is to embed the input discretized audio feature representation features into the hidden dimension. A series of convolution blocks can be used, and the kernel size of the series of convolution blocks gradually increases to form a deep convolution structure in which the convolution kernel size increases layer by layer. Layer normalization is used between the series of convolution blocks, and the GELU activation function is used for activation processing after the last convolution block of the deep convolution structure to obtain a real-valued signal.
[0085] Step 205: convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum diagram.
[0086] Step 206: Perform inverse Fourier transform according to the phase and amplitude values of the signal spectrum to complete the reconstruction of the original audio stream.
[0087] In this embodiment, the original audio stream is obtained; input into a preset audio encoder to obtain the corresponding potential feature representation; the potential feature representation is input into a single quantizer with an expanded codebook space size, and the potential feature representation is quantized to obtain a discretized audio feature representation; the discretized audio feature representation is decoded using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; the real-valued signal is converted using a single sideband spectrum to obtain a converted signal spectrum diagram; an inverse Fourier transform is performed based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding, the high availability of the reconstructed audio is guaranteed when the audio is actually reconstructed.
[0088] In this embodiment, the audio encoder is composed of a 1D convolutional layer and four convolutional blocks, wherein the 1D convolutional layer has 32 convolution channels and a convolution kernel size of 7×7, each convolution block is composed of a residual unit and a downsampling layer, the residual unit includes two convolutional layers with a convolution kernel size of 3×3 and a skip connection layer, the downsampling layer includes a convolutional layer with a stride of S, and the convolution kernel size of the convolution layer with a stride of S is twice the step size, and the configuration of the step size S includes two configuration modes, one configuration mode is (2, 4, 5, 8), and the other configuration mode is (4, 5, 5, 6).
[0089] Furthermore, the audio encoder includes a sequential processing structure after the convolutional blocks. For example, a two-layer LSTM sequential network is placed after each convolutional block for sequence modeling. This ensures timing control during audio reconstruction.
[0090] In this embodiment, the step of inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream specifically includes: using the 1D convolution layer in the audio encoder to perform an initial convolution operation on the input original audio stream to extract the initial audio features in the original audio stream; using the convolution block in the audio encoder to downsample and filter the initial audio features, and obtaining the key audio features corresponding to the target frequency based on the configuration of the sampling step size; and using the key audio features as the potential feature representation corresponding to the original audio stream.
[0091] Specifically, the 1D convolution layer in the audio encoder is used to perform an initial convolution operation on the input original audio stream to extract the initial audio features in the original audio stream, wherein the initial convolution operation represents sampling the input original audio stream 24,000 times per second, and the initial audio features are the audio features obtained by sampling 24,000 times per second; the convolution block in the audio encoder is used to perform downsampling and filtering on the initial audio features, and according to the configuration of the sampling step, the key audio features corresponding to the target frequency are obtained, wherein, when the sampling step is configured as When the sampling step is (2,4,5,8), that is, after the four convolution blocks are down-sampled and filtered, the key audio features with a target frequency of 75 times per second are obtained, and the filtering multiple is 2×4×5×8; furthermore, when the sampling step configuration is (4,5,5,6), that is, after the four convolution blocks are down-sampled and filtered, the key audio features with a target frequency of 40 times per second are obtained, and the filtering multiple is 4×5×5×6; accordingly, the key audio features with a target frequency of 75 times per second or the key audio features with a target frequency of 40 times per second are represented as the potential features corresponding to the original audio stream.
[0092] It should be understood that in this embodiment, the audio encoder, the single quantizer, and the improved decoder essentially constitute an audio sample generator. In order to ensure that the reconstructed audio of the entire audio sample generator is closer to the real speech, before the actual audio reconstruction is performed, an adversarial generation training method is used in advance to train the entire audio sample generator.
[0093] Continue to refer Figure 3 In some optional implementations, a step of performing adversarial generative training is further included before step 201. Figure 3 This is a flowchart of a specific embodiment of performing adversarial generative training in the audio reconstruction method described in this application, including the following steps:
[0094] Step 301: obtaining an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object;
[0095] Specifically, the real object includes a real person.
[0096] Step 302: Process the input audio stream sequentially through the audio encoder, the single quantizer, and the improved decoder to obtain reconstructed audio corresponding to the input audio stream;
[0097] Step 303 : Utilize a preset discriminator to perform a fidelity estimation on the reconstructed audio until the fidelity estimation of the reconstructed audio reaches a preset fidelity threshold, and the training of the audio encoder, the single quantizer, and the improved decoder is completed.
[0098] Specifically, the entire architecture of the audio encoder, the single quantizer, and the improved decoder is regarded as an audio sample generator, and a preset discriminator is used to identify the authenticity estimation of the reconstructed audio generated by the generator compared with the actual audio, thereby ensuring that the training of the audio encoder, the single quantizer, and the improved decoder is completed, so as to finally enable the entire audio sample generator to reconstruct the audio stream in the actual speech synthesis pre-task.
[0099] In this embodiment, the preset discriminators include a multi-period discriminator, a multi-resolution discriminator, and a multi-scale discriminator. By utilizing the multi-period discriminator, the multi-resolution discriminator, and the multi-scale discriminator to jointly perform authenticity authentication processing on the reconstructed audio, the above three types of discriminators are adopted to support multi-period, multi-resolution, and multi-scale authentication during audio reconstruction training, so as to ensure the high availability of the audio sample generator that is finally trained in adversarial generation.
[0100] Continue to refer Figure 4 , Figure 4 This is a flowchart of a specific embodiment of performing realism estimation on the reconstructed audio in the audio reconstruction method described in this application, comprising the following steps:
[0101] Step 401: Using the multi-period discriminator, the multi-resolution discriminator, and the multi-scale discriminator, a quantization loss value, a Mel-spectrum reconstruction loss value, an adversarial loss value, and a feature matching loss value corresponding to the reconstructed audio during the reconstruction process are obtained.
[0102] It should be understood that the multi-cycle discriminator, multi-resolution discriminator and multi-scale discriminator are three types of discriminators, which refer to the identification that supports multi-cycle, multi-resolution and multi-scale, and do not represent the specific number of discriminators. The specific number of discriminators can be freely set according to the amount of adversarial training data. The discriminator calculates the loss values corresponding to the generator in different processing stages, such as the feature matching loss value of the encoder stage, the quantization loss value of the quantization processing stage, the feature matching loss value of the decoder stage, the Mel spectrum reconstruction loss value of the signal spectrum graph generation stage and the final adversarial loss value. When the loss value is output, it is combined with the corresponding loss function and output layer output.
[0103] In step 402, the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value are used as authenticity estimation factors, and combined with a preset comprehensive calculation method, the authenticity estimation is calculated.
[0104] Specifically, when performing realism estimation, the hinge loss form is adopted, and the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value are introduced. The quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value are used as realism estimation factors to jointly calculate the realism estimation.
[0105] In this embodiment, the feature matching loss value includes the feature matching loss value of the encoder stage and the feature matching loss value of the decoder stage.
[0106] In this embodiment, when performing the authenticity valuation calculation, loss values of multiple dimensions are introduced as valuation factors, thereby ensuring the reliability and scientific nature of the final authenticity valuation.
[0107] In this embodiment, in some optional implementations, after step 401, the following further includes: obtaining loss requirement conditions pre-set for the quantization loss value, mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value, wherein the loss requirement conditions include that the loss value does not exceed the set loss threshold.
[0108] Continue to refer Figure 5 In some optional implementations, a step of determining the model parameter tuning process is included before step 402. Figure 5 This is a flowchart of a specific embodiment of determining the model parameter tuning process in the audio reconstruction method described in this application, including the following steps:
[0109] Step 501: compare the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirements;
[0110] Step 502: If any of the quantization loss value, the Mel-spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement, the audio encoder, the single quantizer, and the improved decoder are subjected to model parameter tuning until the quantization loss value, the Mel-spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement.
[0111] Step 503: If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirements, then directly execute step 402.
[0112] Specifically, by limiting the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value to meet the corresponding loss requirements, the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value can be used as valuation factors for authenticity valuation. If any loss value among the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value does not meet the corresponding loss requirements, it is necessary to perform model parameter tuning on the audio encoder, single quantizer and improved decoder, thereby ensuring the high availability of the audio sample generator that has completed the final adversarial generation training.
[0113] Continue to refer Figure 6 , Figure 6 yes Figure 4 The flowchart of a specific embodiment of step 402 includes the following steps:
[0114] Step 601: Obtain dynamic floating weights pre-set for the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value;
[0115] In this embodiment, dynamic floating weights are introduced. Specifically, the dynamic floating weights are related to the order and temporality of audio processing, such as the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value. The feature matching loss value includes the feature matching loss value of the encoder stage and the feature matching loss value of the decoder stage. The order in which these loss values are obtained is the feature matching loss value of the encoder stage, the quantization loss value, the feature matching loss value of the decoder stage, the Mel spectrum reconstruction loss value, and the adversarial loss value. Moreover, as the adversarial generation training time increases, the important loss values focused on in each time period will gradually change. In summary, as the adversarial generation training time increases, the loss value focus gradually shifts from focusing on the feature matching loss value of the encoder stage first, then focusing on the quantization loss value, then focusing on the feature matching loss value of the decoder stage, and finally focusing on the final adversarial loss value. Therefore, the dynamic floating weight is set to combine the order and temporality of audio processing, so as to dynamically adjust the calculation weight of each loss value, which is more reasonable and scientific.
[0116] Step 602: Obtain current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point, based on the dynamic floating weights;
[0117] Step 603: Calculate the total loss value corresponding to the current training time point using a weighted summation algorithm based on the current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point.
[0118] Step 604: Obtain a corresponding authenticity estimate at the current training time point using the total loss value, wherein the total loss value and the authenticity estimate are in inverse proportion.
[0119] Specifically, the larger the total loss value, the smaller the authenticity estimation, and the lower the confidence of the reconstructed audio; the smaller the total loss value, the larger the authenticity estimation, the higher the confidence of the reconstructed audio, and the higher the availability of the reconstructed audio.
[0120] Continue to refer Figure 7 In some optional implementations, after step 604, a step of iteratively calculating the total loss value is also included. Figure 7 This is a flowchart of a specific embodiment of iteratively calculating the total loss value in the audio reconstruction method described in this application, comprising the following steps:
[0121] Step 701, determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold;
[0122] Step 702: If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, continue training and iteratively calculate the latest weights corresponding to the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value, and feature matching loss value at the latest training time point, and
[0123] Step 703: Based on the latest weights, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding truthfulness estimate at the latest training time point is obtained using the total loss value;
[0124] In step 704 , the training of the audio encoder, the single quantizer, and the improved decoder is completed until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold.
[0125] Specifically, if the quantization loss, mel-spectrogram reconstruction loss, adversarial loss, and feature matching loss all meet the corresponding loss requirements, and if the realism estimate of the reconstructed audio does not reach a preset realism threshold, training is continued, and an iterative calculation is performed to obtain the latest realism estimate corresponding to the reconstructed audio at the latest time point, thereby ultimately obtaining a trained audio encoder, a single quantizer, and the improved decoder. This ensures the realism of the subsequent actual audio reconstruction.
[0126] The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, quantizes the potential feature representation, and obtains a discretized audio feature representation; uses an improved decoder that has undergone adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed during actual audio reconstruction.
[0127] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.
[0128] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0129] In an embodiment of the present application, the original audio stream is obtained; input into a preset audio encoder to obtain a corresponding potential feature representation; the potential feature representation is input into a single quantizer with an expanded codebook space size, and the potential feature representation is quantized to obtain a discretized audio feature representation; the discretized audio feature representation is decoded using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; the real-valued signal is converted using a single sideband spectrum to obtain a converted signal spectrum diagram; an inverse Fourier transform is performed based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.
[0130] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of an audio reconstruction device, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0131] like Figure 8 As shown, the audio reconstruction device 800 described in this embodiment includes: an original audio stream acquisition module 801, an audio encoding processing module 802, a quantization processing module 803, a decoding processing and real-valued signal acquisition module 804, a signal spectrum diagram conversion module 805 and an inverse Fourier transform module 806.
[0132] in:
[0133] The original audio stream acquisition module 801 is used to acquire the original audio stream;
[0134] The audio encoding processing module 802 is configured to input the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream;
[0135] The quantization processing module 803 is used to input the potential feature representation into a single quantizer with an expanded codebook space size, and quantize the potential feature representation to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is 2 10 Expand to 2 12 ;
[0136] A decoding processing and real-valued signal acquisition module 804 is configured to decode the discretized audio feature representation using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module, and a convolution block;
[0137] The signal spectrum conversion module 805 is used to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum;
[0138] The inverse Fourier transform module 806 is configured to perform an inverse Fourier transform based on the phase and amplitude values of the signal spectrum to complete reconstruction of the original audio stream.
[0139] The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, quantizes the potential feature representation, and obtains a discretized audio feature representation; uses an improved decoder that has undergone adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed during actual audio reconstruction.
[0140] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0141] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0142] To solve the above technical problems, the present application also provides a computer device. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.
[0143] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected via a system bus. Figure 9Only a computer device 9 having components such as a memory 9a, a processor 9b, and a network interface 9c is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. It should be understood by those skilled in the art that a computer device herein is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, and the like.
[0144] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.
[0145] The memory 9a includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 9a may be an internal storage unit of the computer device 9, such as the hard disk or internal memory of the computer device 9. In other embodiments, the memory 9a may also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 9a may also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for an audio reconstruction method. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.
[0146] In some embodiments, the processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to execute computer-readable instructions stored in the memory 9a or process data, such as computer-readable instructions for executing the audio reconstruction method.
[0147] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0148] The computer device proposed in this embodiment belongs to the field of R&D design and audio processing technology, and is applied to pre-processing scenarios of audio reconstruction and audio synthesis. This application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, and quantizes the potential feature representation to obtain a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.
[0149] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the audio reconstruction method as described above.
[0150] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to pre-processing scenarios of audio reconstruction and audio synthesis. The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, and quantizes the potential feature representation to obtain a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.
[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0152] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.
Claims
1. An audio reconstruction method, characterized in that: The steps include: Get the original audio stream; Inputting the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream; Inputting the latent feature representation into a single quantizer with an expanded codebook space size, and quantizing the latent feature representation to obtain a discretized audio feature representation; Decoding the discretized audio feature representation using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; Converting the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum diagram; Performing inverse Fourier transform based on the phase and amplitude values of the signal spectrum to complete the reconstruction of the original audio stream, Before performing the step of obtaining the original audio stream, the method further includes: Obtaining an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object; Processing the input audio stream sequentially through the audio encoder, the single quantizer, and the improved decoder to obtain reconstructed audio corresponding to the input audio stream; Using a preset discriminator to perform a fidelity estimation on the reconstructed audio until the fidelity estimation of the reconstructed audio reaches a preset fidelity threshold, whereby the training of the audio encoder, the single quantizer, and the improved decoder is completed, wherein the preset discriminator includes a multi-period discriminator, a multi-resolution discriminator, and a multi-scale discriminator; The step of using a preset discriminator to perform authenticity estimation on the reconstructed audio specifically includes: Step 401: Using the multi-period discriminator, the multi-resolution discriminator, and the multi-scale discriminator, a quantization loss value, a Mel-spectrum reconstruction loss value, an adversarial loss value, and a feature matching loss value corresponding to the reconstructed audio during the reconstruction process are obtained. Step 402: Calculate the authenticity estimation by using the quantization loss, Mel spectrum reconstruction loss, adversarial loss, and feature matching loss as authenticity estimation factors and combining them with a preset comprehensive calculation method. Specifically, after step 401, the following steps are further included: Comparing the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirements; If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, then performing model parameter tuning processing on the audio encoder, the single quantizer, and the improved decoder until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions; If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirements, step 402 is directly executed.
2. The audio reconstruction method according to claim 1, characterized in that The step of inputting the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream specifically includes: Performing an initial convolution operation on the input original audio stream using the 1D convolution layer in the audio encoder to extract initial audio features from the original audio stream; Performing downsampling and filtering processing on the initial audio features using a convolution block in the audio encoder, and obtaining key audio features corresponding to a target frequency according to a configuration of a sampling step size; The key audio features are represented as potential features corresponding to the original audio stream.
3. The audio reconstruction method according to claim 1, characterized in that Before executing the step of using the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as authenticity estimation factors and combining them with a preset comprehensive calculation method to calculate the authenticity estimation, the method further includes: Obtain loss requirement conditions pre-set for the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value, respectively, wherein the loss requirement condition includes that the loss value does not exceed a set loss threshold.
4. The audio reconstruction method according to claim 1, characterized in that The step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as the authenticity estimation factors and combining them with a preset comprehensive calculation method to calculate the authenticity estimation specifically includes: Obtaining dynamic floating weights pre-set for the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value; According to the dynamic floating weights, current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point are obtained respectively; Based on the current weights corresponding to the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value, and feature matching loss value at the current training time point, a weighted sum algorithm is used to calculate the total loss value corresponding to the current training time point; The total loss value is used to obtain the corresponding authenticity estimate at the current training time point, wherein the total loss value and the authenticity estimate are in inverse proportion.
5. The audio reconstruction method according to claim 4, characterized in that After executing the step of obtaining a corresponding truth value estimate at the current training time point using the total loss value, the method further includes: Determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold; If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, the training is continued, and the latest weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the latest training time point are obtained in an iterative calculation manner, and Based on the latest weights, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding truthfulness estimate at the latest training time point is obtained through the total loss value; The training of the audio encoder, the single quantizer and the improved decoder is completed until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold.
6. An audio reconstruction device, characterized in that: include: The original audio stream acquisition module is used to obtain the original audio stream; An audio encoding processing module, configured to input the original audio stream into a preset audio encoder to obtain a latent feature representation corresponding to the original audio stream; a quantization processing module, configured to input the latent feature representation into a single quantizer with an expanded codebook space size, and perform quantization processing on the latent feature representation to obtain a discretized audio feature representation; A decoding processing and real-valued signal acquisition module, configured to decode the discretized audio feature representation using an improved decoder that has undergone adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; a signal spectrum conversion module, configured to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum; An inverse Fourier transform module is used to perform an inverse Fourier transform based on the phase and amplitude values of the signal spectrum to complete the reconstruction of the original audio stream. Before obtaining the original audio stream, the original audio stream obtaining module further includes: Obtaining an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object; Processing the input audio stream sequentially through the audio encoder, the single quantizer, and the improved decoder to obtain reconstructed audio corresponding to the input audio stream; Using a preset discriminator to perform a fidelity estimation on the reconstructed audio until the fidelity estimation of the reconstructed audio reaches a preset fidelity threshold, whereby the training of the audio encoder, the single quantizer, and the improved decoder is completed, wherein the preset discriminator includes a multi-period discriminator, a multi-resolution discriminator, and a multi-scale discriminator; The step of using a preset discriminator to perform authenticity estimation on the reconstructed audio specifically includes: Step 401: Using the multi-period discriminator, the multi-resolution discriminator, and the multi-scale discriminator, a quantization loss value, a Mel-spectrum reconstruction loss value, an adversarial loss value, and a feature matching loss value corresponding to the reconstructed audio during the reconstruction process are obtained. Step 402: Calculate the authenticity estimation by using the quantization loss, Mel spectrum reconstruction loss, adversarial loss, and feature matching loss as authenticity estimation factors and combining them with a preset comprehensive calculation method. Specifically, after step 401, the following steps are further included: Comparing the quantization loss value, the mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirements; If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, then performing model parameter tuning processing on the audio encoder, the single quantizer, and the improved decoder until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions; If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirements, step 402 is directly executed.
7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio reconstruction method according to any one of claims 1 to 5 when executing the computer-readable instructions.
8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the audio reconstruction method according to any one of claims 1 to 5.