Audio signal processing method and device, computer equipment and storage medium

Abstract

The invention discloses an audio signal processing method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The audio signal processing method is applied to a pre-built vocoder system, and the vocoder system is built based on a variational auto-encoder and comprises a prior encoder, a posterior encoder and a decoder. First, a loss function is constructed, including reconstruction loss and difference loss between encoders. And then, inputting the original audio signal into a posterior encoder to generate Gaussian distribution, sampling the distribution to generate an intermediate hidden variable, and inputting the hidden variable into a decoder for audio reconstruction to generate a restored audio signal. A pre-trained vocoder system is obtained by calculating loss and carrying out optimization and iterative training. And finally, processing the to-be-restored frequency spectrum by using the pre-trained vocoder system to generate a restored audio signal. According to the method, the stability of the model is effectively improved, the training difficulty is reduced, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an audio signal processing method, device, computer equipment and storage medium. Background Art

[0002] In the field of audio signal processing, how to restore the phase-less amplitude spectrum to audio has always been a difficult problem. The system that performs this task is generally called a vocoder. Traditional vocoders are usually implemented based on the Griffin-Lim algorithm, which is one of the classic algorithms in this field. The process of this algorithm is to gradually obtain the restored phase spectrum through multiple STFT-ISTFT iterations of a random initial phase spectrum. The Griffin-Lim algorithm is simple and easy to implement, but the effect of generating audio is not satisfactory. In addition, the Griffin-Lim algorithm is only applicable to linear amplitude spectra, and cannot be applied to Mel spectra with Mel frequency scale.

[0003] In recent years, with the development of deep learning technology, a series of deep learning-based vocoder systems have emerged, among which the best-performing ones are based on Generative Adversarial Network (GAN). The GAN-based vocoder introduces a discriminator network and jointly trains it with the generator network for adversarial learning, thereby gradually generating more realistic audio in the learning process. However, GAN is difficult to train, and the training hyperparameters need to be carefully adjusted. Otherwise, the gradient will vanish or explode during the training process, affecting the final effect. In addition, the GAN-based vocoder has poor generalization ability, and cannot restore the audio spectrum outside the training sample very well, making it difficult to generate audio with a reasonable phase. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an audio signal processing method, apparatus, computer equipment and storage medium to solve the problems of difficult training and poor generalization of existing GAN vocoders.

[0005] In order to solve the above technical problems, the present application provides an audio signal processing method, which adopts the following technical solution:

[0006] An audio signal processing method is applied to a pre-built vocoder system, the vocoder system is built based on a variational autoencoder. The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The audio signal processing method includes:

[0007] Constructing a loss function of a vocoder system, wherein the loss function of the vocoder system includes a first loss term and a second loss term, the first loss term represents an audio signal reconstruction loss, and the second loss term represents a loss between a priori encoder and a posteriori encoder;

[0008] Acquire pre-collected training data, wherein the training data is pre-collected original audio signals;

[0009] Input the original audio signal into the posterior encoder to obtain the posterior Gaussian distribution corresponding to the original audio signal;

[0010] Sampling the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable;

[0011] Inputting the first intermediate latent variable into a decoder for audio decoding to obtain a first restored audio signal;

[0012] Calculating a vocoder loss using a loss function of a vocoder system based on the first restored audio signal and the original audio signal;

[0013] A back propagation algorithm is used to transfer the vocoder loss in the vocoder system, and the vocoder system is iteratively trained to obtain a pre-trained vocoder system;

[0014] An audio restoration instruction is received, a frequency spectrum to be restored is obtained, and the frequency spectrum to be restored is input into a pre-trained vocoder system to obtain a second restored audio signal.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides an audio signal processing device, which adopts the following technical solution:

[0016] An audio signal processing device, the audio signal processing device is used to run a pre-built vocoder system, the vocoder system is built based on a variational autoencoder. The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The audio signal processing device includes:

[0017] A loss function construction module, used to construct a loss function of a vocoder system, wherein the loss function of the vocoder system includes a first loss term and a second loss term, the first loss term represents an audio signal reconstruction loss, and the second loss term represents a loss between a priori encoder and a posteriori encoder;

[0018] A training data acquisition module, used to acquire pre-collected training data, wherein the training data is a pre-collected original audio signal;

[0019] The a posteriori encoding module is used to input the original audio signal into the a posteriori encoder to obtain the a posteriori Gaussian distribution corresponding to the original audio signal;

[0020] A latent variable sampling module, used for sampling the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable;

[0021] An audio decoding module, used for inputting the first intermediate latent variable into a decoder for audio decoding to obtain a first restored audio signal;

[0022] A loss calculation module, configured to calculate a vocoder loss by using a loss function of a vocoder system based on the first restored audio signal and the original audio signal;

[0023] an iterative training module, for transferring vocoder loss in the vocoder system by using a back propagation algorithm, and iteratively training the vocoder system to obtain a pre-trained vocoder system;

[0024] The audio restoration module is used to receive an audio restoration instruction, obtain a frequency spectrum to be restored, and input the frequency spectrum to be restored into a pre-trained vocoder system to obtain a second restored audio signal.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0026] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above-mentioned audio signal processing methods when executing the computer-readable instructions.

[0027] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0028] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of any one of the above-mentioned audio signal processing methods are implemented.

[0029] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0030] The present application discloses an audio signal processing method, device, computer equipment and storage medium, which belongs to the field of artificial intelligence technology. The present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of training difficulty and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the training difficulty is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through hidden variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0033] Figure 2 A framework diagram of a vocoder system based on a variational autoencoder according to the present application is shown;

[0034] Figure 3 A flowchart of an embodiment of an audio signal processing method according to the present application is shown;

[0035] Figure 4 A schematic structural diagram of an embodiment of an audio signal processing device according to the present application is shown;

[0036] Figure 5 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0038] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0040] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0041] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0042] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc.

[0043] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0044] It should be noted that the audio signal processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the audio signal processing device is generally arranged in the server / terminal device.

[0045] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0046] In this embodiment, the present application discloses an audio signal processing method, which is applied to a pre-built vocoder system. Figure 2 , Figure 2 The framework diagram of the vocoder system based on the variational autoencoder according to the present application is shown. The vocoder system is built based on the variational autoencoder (VAE). The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The variational autoencoder VAE is a generation model based on probability coding and has been widely used in unsupervised learning. VAE combines the ideas of autoencoder and probability coding, and realizes a more flexible and controllable sample generation process by modeling the potential representation. VAE assumes that the potential representation obeys a prior distribution (usually a Gaussian distribution), and maps the input data to the distribution parameters of the latent space (such as mean and variance) through the encoder. The decoder samples from the latent space and maps the sampling results back to the original input space to generate new samples.

[0047] VAE mainly consists of two parts: encoder and decoder. The encoder is responsible for mapping the input data to the distribution parameters of the latent space, and the decoder is responsible for sampling from the latent space and mapping the sampling results back to the original input space.

[0048] The training goal of VAE is to maximize the likelihood of a certain data distribution, while introducing the idea of ​​variational inference and optimizing the variational lower bound (ELBO). This is achieved by minimizing the reconstruction error (i.e., the difference between the sample generated by the decoder and the original input data) and the regularization of the latent space (usually using KL divergence to measure the difference between the distribution of the encoder output and the prior distribution).

[0049] The a priori encoder is used to generate the a priori distribution of latent variables. By analyzing the structure of the input signal, the distribution hypothesis of the latent variables is established to ensure that the generated latent variables have reasonable distribution characteristics, which helps to improve the generation effect of the decoder. The posterior encoder calculates the corresponding latent variable distribution (usually Gaussian distribution) based on the input audio signal. The posterior encoder generates a more accurate latent variable distribution by learning the characteristics of the original data, helping the model to better reconstruct the audio signal. The decoder is a module in the VAE that converts the latent variables back to the original audio signal. By inputting the latent variables, it decodes them and tries to restore the audio signal similar to the input, completing the reconstruction or generation task of the audio data.

[0050] exist Figure 2 In the figure, the left half represents the training process of the model, and the right half represents the inference process of the model. During the training process of the model, the spectrum to be restored is input into the prior encoder to obtain the mean μ′ and standard deviation σ′ of the prior Gaussian distribution; the original audio signal is input into the encoder to obtain the mean μ and standard deviation σ of the estimated posterior Gaussian distribution, and an intermediate latent variable Z is sampled from the estimated posterior Gaussian distribution, and Z is input into the decoder to obtain the restored audio signal;

[0051] During the inference process, the spectrum to be restored is input into the prior encoder to obtain the mean μ′ and standard deviation σ′ of the prior Gaussian distribution. An intermediate latent variable Z′ is sampled from the prior Gaussian distribution, and Z′ is input into the decoder to obtain the restored audio signal. It should be noted that, unlike the general VAE model, the prior distribution of the model is learnable and relies on the spectrum as an input condition; such a process enhances the overall flexibility of the model and its robustness to different types of audio.

[0052] Continue to refer Figure 3 , shows a flow chart of an embodiment of a method for processing an audio signal according to the present application. The audio signal processing method comprises the following steps:

[0053] S301, constructing a loss function of a vocoder system, wherein the loss function of the vocoder system includes a first loss term and a second loss term, the first loss term represents an audio signal reconstruction loss, and the second loss term represents a loss between a priori encoder and a posteriori encoder.

[0054] Specifically, in the vocoder system, the loss function is the key optimization target. In order to ensure that the vocoder can effectively reconstruct and generate audio signals, the loss function of the vocoder system is usually set to include two main parts, namely the first loss term and the second loss term. The first loss term is the audio signal reconstruction loss, which measures the difference between the restored audio signal generated by the decoder and the original input audio signal. The mean square error or other distance metric functions are often used to calculate the difference, with the aim of making the restored audio signal as close to the input signal as possible. The second loss term focuses on the difference between the prior encoder and the posterior encoder. The KL divergence (Kullback-Leibler Divergence) is usually used to calculate the difference between the distributions generated by the two. This loss term aims to constrain the latent space of the model so that the distribution of the latent variables conforms to the assumptions of the prior distribution, thereby enhancing the generation ability and generalization performance of the model. By optimizing these two loss terms, the vocoder system can accurately reconstruct the audio signal while generating a model with a reasonable latent variable distribution.

[0055] Furthermore, the loss function of the vocoder system is expressed as follows:

[0056] F=-log p θ (x|z)+KL(q φ (z|x)|p γ (z|c))

[0057] Where F is the loss function of the vocoder system, x is the audio sampling signal, c is the spectral feature, -log p θ (x|z) is the first loss term, which represents the difference between the sample generated by the decoder and the original input data, KL(q φ (z|x)|p γ (z|c)) is the second loss term, which is used to measure the difference between the distribution of the encoder output and the prior distribution, p θ (x|z) represents the probability of observation data x under a given latent variable z, q φ (z|x) represents the posterior probability distribution of the latent variable z given the observed data x, p γ (z|c) represents the prior probability distribution of the latent variable z, and θ, φ, and γ are the parameters of the decoder, posterior encoder, and prior encoder, respectively.

[0058] Furthermore, the second loss term is expressed as follows:

[0059]

[0060] Where μ and σ are the mean and standard deviation of the posterior Gaussian distribution, and μ′ and σ′ are the mean and standard deviation of the prior Gaussian distribution.

[0061] S302: Acquire pre-collected training data, where the training data is pre-collected original audio signals.

[0062] Specifically, the training of the vocoder system relies on a large amount of high-quality audio data. In order to train a robust vocoder model, audio signals need to be collected and organized in advance as training data. The original audio signals can be audio clips of different types and sources, such as music, speech, environmental sound, etc., covering different frequencies, tones, and complexities. When collecting this data, it is necessary to consider the format of the audio data, the sampling rate, and the diversity and representativeness of the data volume to ensure that the model can adapt to a variety of input situations.

[0063] In addition, the training data needs to be preprocessed, such as noise reduction, standardization or data enhancement, to improve the model's robustness to noisy and mutated audio signals. The quality and quantity of training data directly affect the performance of the vocoder system, because the model needs to learn features from these data, master the time-frequency characteristics and potential structural laws of audio signals, and ultimately improve its generalization ability to unseen audio.

[0064] S303: Input the original audio signal to a posteriori encoder to obtain a posteriori Gaussian distribution corresponding to the original audio signal.

[0065] Specifically, in this step, the vocoder system processes the original audio signal using the posterior encoder to convert the input high-dimensional audio signal into the corresponding posterior Gaussian distribution. The posterior encoder is responsible for mapping the complex audio signal into the latent space, and generates a potential representation of the audio signal by learning the time-frequency characteristics and structure of the audio signal. This representation is usually in the form of a Gaussian distribution, which defines the mean and variance of each audio clip in the latent space. The output of the posterior encoder is this posterior Gaussian distribution, which reflects the model's understanding of the input audio signal and allows the vocoder system to perform further operations on the audio signal in the latent space. The key to this step is to extract the core features of the audio signal through the posterior distribution, while introducing a certain amount of randomness, so that the model can generate new or reconstructed audio signals in the subsequent decoding process, thereby having generative ability and generalization performance.

[0066] S304: Sampling the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable.

[0067] Specifically, once the posterior Gaussian distribution corresponding to the original audio signal is obtained, it is necessary to sample from this distribution to generate the first intermediate latent variable. Sampling is the process of converting the features (mean and variance) in the Gaussian distribution into a specific latent variable value. This latent variable is a simplified representation of the audio signal in the latent space. The sampling operation introduces randomness, which is also an important feature of the VAE model, allowing the generator to generate slightly different audio signals through different sampling processes. This randomness enhances the generation ability of the model while retaining the main features of the original audio signal. The sampling result of the latent variable has a lower dimension in the latent space and retains the core structural information of the input audio signal, which enables the decoding process to effectively reconstruct the audio. This latent variable can not only be used to restore the input audio signal, but also to generate new audio signals.

[0068] S305: Input the first intermediate latent variable into a decoder for audio decoding to obtain a first restored audio signal.

[0069] Specifically, after obtaining the intermediate latent variables, the next step of the system is to convert these latent variables back to audio signals through the decoder. As an important part of the vocoder system, the decoder's task is to map the low-dimensional representation (hidden variables) in the latent space back to the high-dimensional audio signal space. Specifically, the decoder generates corresponding audio features based on the latent variables, and gradually restores the time domain or frequency domain features similar to the original audio signal through the learned parameters. The decoding process requires the model to effectively capture the key information in the latent variables to ensure that the restored audio signal has a high auditory similarity to the original signal. The first restored audio signal generated is not completely accurate to the original audio, but by optimizing the loss function, it can be made as close to the original audio signal as possible. The performance of the decoder directly affects the reconstruction quality of the audio signal. Therefore, during the training process, the decoder needs to be fully optimized to ensure that it has a strong reconstruction capability and generation effect.

[0070] S306: Calculate a vocoder loss by using a loss function of a vocoder system based on the first restored audio signal and the original audio signal.

[0071] Specifically, after generating the first restored audio signal, the next step is to calculate the loss of the model through the loss function of the vocoder system. The loss function is used to measure the difference between the restored audio signal output by the model and the original audio signal input. The first loss (reconstruction loss) is used to evaluate the similarity between the restored audio signal and the original audio signal. The mean square error (MSE) or other error metrics are usually used to measure the difference between the two, with the goal of making the restored audio as consistent as possible with the original audio. The second loss measures the difference between the distributions generated by the prior encoder and the posterior encoder, usually calculated using the KL divergence. By combining these two parts of the loss, the system can simultaneously optimize the reconstruction quality of the audio signal and the rationality of the latent variable distribution. The calculation results of the loss function will be used to guide the parameter update of the model to gradually improve the performance of the vocoder system in audio signal processing, enabling it to reconstruct and generate audio signals more accurately.

[0072] S307, using a back propagation algorithm to propagate the vocoder loss in the vocoder system, and iteratively training the vocoder system to obtain a pre-trained vocoder system.

[0073] Specifically, in order to optimize the vocoder system, the model needs to be trained using the back-propagation algorithm. In this step, based on the loss calculated in the previous step, the loss is propagated back to each layer of the model through the back-propagation algorithm, and the parameters in the model are gradually updated. The back-propagation algorithm calculates the gradient of the loss function relative to each parameter through the chain rule, indicating how to adjust the weights of the model to minimize the loss. Each component of the vocoder system (a priori encoder, a posteriori encoder, and decoder) will be optimized in this process. The iterative training and updating process will be repeated many times. By continuously adjusting the model parameters, the model can gradually improve the reconstruction accuracy and generalization ability of the audio signal. After sufficient iterative training, the vocoder system can learn efficient feature representations from the training data, thereby achieving high-quality audio signal processing and reconstruction, and finally obtaining a pre-trained, stable vocoder system.

[0074] S308, receiving an audio restoration instruction, obtaining a frequency spectrum to be restored, and inputting the frequency spectrum to be restored into a pre-trained vocoder system to obtain a second restored audio signal.

[0075] Specifically, after completing the training of the vocoder system, the system can accept audio restoration instructions initiated by the user. At this time, the user can input the audio spectrum to be restored as the input signal of the model. The pre-trained vocoder system has learned the ability to process audio signals through a large amount of training data, so it can efficiently convert the input spectrum information into the corresponding restored audio signal. The input spectrum to be restored will first pass through the encoder module of the vocoder system to extract the corresponding latent variables, and then generate the final audio signal through the decoder. Since the system has been fully trained, the generated second restored audio signal should have a high similarity and quality with the real audio. The core of this process is to use the generation ability of the vocoder system to reasonably reconstruct the input spectrum to achieve the purpose of efficient audio restoration.

[0076] In the above embodiments, the present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of difficulty in training and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the difficulty of training is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through hidden variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects.

[0077] Furthermore, after the step of inputting the original audio signal into the a posteriori encoder to obtain the a posteriori Gaussian distribution corresponding to the original audio signal, the method further includes:

[0078] Performing Fourier transform on the original audio signal to obtain the original audio spectrum corresponding to the original audio signal;

[0079] The original audio spectrum is input into the prior encoder to obtain the prior Gaussian distribution corresponding to the original audio spectrum.

[0080] In this embodiment, after the original audio signal is input into the posterior encoder to obtain its corresponding posterior Gaussian distribution, it is also necessary to perform Fourier transform on the original audio signal to obtain its spectrum representation. Fourier transform is a common signal processing technology that converts the audio signal in the time domain into spectrum information in the frequency domain, thereby revealing the components and characteristics of the audio signal at different frequencies. For audio processing, the spectrum representation can more intuitively reflect the frequency structure of the signal, which helps the model to more effectively extract and learn the features in the audio signal. The spectrum of the original audio signal is input into the prior encoder, and the prior encoder generates the corresponding prior Gaussian distribution based on the spectrum data. Similar to the posterior encoder, the task of the prior encoder is to learn the latent space representation based on the spectrum information, which appears in the form of a Gaussian distribution. As a priori assumption of the latent variable, the prior Gaussian distribution provides a reference distribution to help the model guide the learning of the posterior distribution when processing the audio signal, ensuring that the latent variables generated by the model not only conform to the characteristics of the audio signal, but also maintain the generation ability and generalization. The a priori encoder and the posterior encoder work together to help the model understand the structure of the audio signal more deeply by jointly modeling the time domain and frequency domain features of the audio signal.

[0081] Through the above steps, the vocoder system can establish a more accurate mapping between the time domain and frequency domain features of the audio signal, improving the model's ability to process audio signals. Combined with the Gaussian distribution generated by the prior encoder and the posterior encoder, the vocoder system can better balance the reconstruction accuracy and generation quality, thereby achieving more efficient audio signal reconstruction and generation. At the same time, this dual encoding structure enhances the generalization ability of the model, making it more stable when processing complex audio inputs.

[0082] Furthermore, the step of performing Fourier transform on the original audio signal to obtain an original audio spectrum corresponding to the original audio signal specifically includes:

[0083] Performing a preprocessing operation on the original audio signal, wherein the preprocessing operation includes a denoising operation, a framing operation, and a windowing operation;

[0084] Perform short-time Fourier transform on the preprocessed original audio signal to obtain the initial audio spectrum;

[0085] Perform Mel filtering on the initial audio spectrum to obtain a Mel spectrum;

[0086] Perform logarithm operation on the Mel spectrum, and perform discrete cosine transform on the Mel spectrum after logarithm operation to obtain a Mel spectrum graph;

[0087] Get the original audio spectrum corresponding to the original audio signal from the Mel spectrogram.

[0088] In the present embodiment, first, the original audio signal is preprocessed. Preprocessing includes denoising, framing and windowing operations. The denoising operation is intended to eliminate background noise and irrelevant interference signals in the audio to ensure that the extracted spectrum is clearer. The framing operation divides the continuous audio signal into several short time periods. The windowing operation applies a windowing function before each frame segment to reduce the impact of edge effects on the spectrum to ensure the smoothness and accuracy of the spectrum.

[0089] Next, a short-time Fourier transform (STFT) is performed on the preprocessed audio signal to convert the audio signal of each time frame into the frequency domain to generate an initial audio spectrum. STFT processes the audio in segments, so that the changing characteristics of the audio signal in time and frequency can be captured. Then, the initial audio spectrum is Mel-filtered, that is, the spectrum is resampled into a Mel spectrum that is more in line with the auditory characteristics of the human ear through a set of Mel filters that simulate the auditory perception of the human ear. The Mel spectrum is designed to capture the key frequency characteristics of speech and audio signals, so that the spectral representation of the audio signal is more in line with the human auditory system.

[0090] After obtaining the Mel spectrum, it is transformed nonlinearly by taking the logarithm operation, and the amplitude spectrum is converted into a logarithmic spectrum to further simulate the human ear's perception of sound intensity. Subsequently, the Mel spectrum after taking the logarithm is subjected to discrete cosine transform (DCT) to further compress the spectrum information and extract the low-dimensional features of the audio signal. This operation helps to remove redundant information in the spectrum, retain the most representative audio features, and finally obtain the Mel spectrogram. By extracting the spectrum corresponding to the original audio signal from the Mel spectrogram, the frequency feature representation of the audio signal is obtained. This spectrum not only contains the frequency information of the audio, but also retains its key time domain features, which can better reconstruct or generate audio signals.

[0091] Through the above steps, the accuracy and expressiveness of audio spectrum extraction are improved. Using Mel filtering, logarithmic transformation and discrete cosine transformation, the vocoder system can more efficiently extract key features in the audio signal, improving the model's ability to understand and generalize audio signals. The final Mel spectrogram provides a more stable and accurate input for audio signal reconstruction and generation.

[0092] Further, based on the first restored audio signal and the original audio signal, the step of calculating the vocoder loss by using the loss function of the vocoder system specifically includes:

[0093] Calculate the error between the first restored audio signal and the original audio signal based on the first loss term to obtain a first loss;

[0094] Performing Fourier transform on the first restored audio signal to obtain a restored audio spectrum corresponding to the first restored audio signal;

[0095] Calculate the error between the restored audio spectrum and the original audio spectrum based on the second loss term to obtain a second loss;

[0096] The sum of the first loss and the second loss is calculated to obtain the vocoder loss.

[0097] In this embodiment, the error between the first restored audio signal and the original audio signal is first calculated based on the first loss term to obtain the first loss. The first loss is usually used to measure the reconstruction error in the time domain and the difference between the restored audio and the original audio. The mean square error or other difference measurement methods are usually used. Next, the first restored audio signal is Fourier transformed to obtain its frequency domain information and obtain the corresponding restored audio spectrum. In this way, the performance of the restored signal in the frequency space can be analyzed and compared with the original audio spectrum. Subsequently, the error between the restored audio spectrum and the original audio spectrum is calculated based on the second loss term to obtain the second loss. The Mel spectrum error or other frequency domain difference measurement is usually used. This step ensures that the model can not only reconstruct the time structure of the audio, but also match the original audio in frequency distribution. Finally, the system weightedly sums the first loss and the second loss to obtain the vocoder loss, ensuring that the model is optimized in both time and frequency dimensions.

[0098] Through the above steps, the system can optimize the reconstruction quality of audio in both the time domain and the frequency domain, thereby improving the overall performance of audio restoration.

[0099] The step of calculating the error between the first restored audio signal and the original audio signal based on the first loss term to obtain the first loss specifically includes:

[0100] According to a preset audio signal sampling rule, the first restored audio signal and the original audio signal are sampled respectively to obtain an audio sampling signal;

[0101] According to a preset spectrum feature sampling rule, the original audio spectrum and the restored audio spectrum are sampled respectively to obtain audio spectrum features, wherein the audio signal sampling rule and the spectrum feature sampling rule match each other;

[0102] Obtain decoder parameters, calculate an error between a first restored audio signal and an original audio signal based on the decoder parameters, the audio sampling signal, and the audio spectrum characteristics, and obtain a first loss.

[0103] In this embodiment, by sampling the audio signal and sampling the spectrum characteristics, the audio signal can be comprehensively analyzed from the perspectives of time domain and frequency domain. First, according to the preset audio signal sampling rules, the first restored audio signal and the original audio signal are sampled respectively, which can effectively capture the difference between the restored signal and the original signal in the time domain, and ensure that important audio information is not lost during the decoding process. Secondly, according to the preset spectrum feature sampling rules, the original audio spectrum and the restored audio spectrum are sampled, which can further capture the subtle differences between the signals from the perspective of frequency domain, especially for the frequency components that are sensitive to the human ear, and can better reflect the changes in sound quality. Subsequently, by introducing the decoder parameters, the audio sampling signal is combined with the spectrum characteristics, the error between the restored audio and the original audio is calculated, and based on this, the loss function, i.e., the first loss term, is obtained. This loss term can comprehensively consider the errors in the time domain and the frequency domain, provide a basis for the optimization of the audio restoration algorithm, and improve the restoration accuracy and quality of the audio.

[0104] Through the above steps, the distortion phenomenon in the audio restoration process can be effectively reduced, the restoration degree of the audio signal can be improved, and then the sound quality can be improved, making the restored audio more realistic.

[0105] The step of calculating the error between the restored audio spectrum and the original audio spectrum based on the second loss term to obtain the second loss specifically includes:

[0106] Get the mean and standard deviation of the posterior Gaussian distribution, and get the mean and standard deviation of the prior Gaussian distribution;

[0107] Obtain a posteriori encoder parameters and a priori encoder parameters;

[0108] Based on the posterior encoder parameters, the prior encoder parameters, the mean and standard deviation of the posterior Gaussian distribution, and the mean and standard deviation of the prior Gaussian distribution, calculate the KL divergence between the posterior Gaussian distribution and the prior Gaussian distribution;

[0109] The second loss is obtained by taking the KL divergence as the error between the restored audio spectrum and the original audio spectrum.

[0110] In this embodiment, first, the mean and standard deviation of the posterior prior Gaussian distribution and the prior Gaussian distribution are obtained, which can be used to characterize the probability distribution of the restored audio and the original audio in different states. Then, based on the posterior encoder parameters and the prior encoder parameters, these distribution information are used to calculate the KL divergence between the posterior Gaussian distribution and the prior Gaussian distribution. KL divergence is an indicator to measure the difference between two probability distributions, which can reflect the difference in spectral characteristics between the restored audio and the original audio. By using KL divergence as a metric for spectral error, the second loss function obtained can reflect the amount of information lost in the spectral restoration process. By comprehensively considering the frequency domain characteristics of the audio signal and the corresponding probability distribution differences, further optimization of the restored audio accuracy is achieved. In addition, the introduction of the posterior encoder and the prior encoder parameters also provides more abundant information and constraints for the optimization of the model, which helps to improve the robustness and effect of audio restoration.

[0111] Through the above steps, the error of the audio spectrum can be effectively measured by KL divergence, which can enhance the accuracy of audio spectrum restoration, thereby better preserving the detailed characteristics of the audio during the restoration process.

[0112] Further, the step of obtaining a spectrum to be restored and inputting the spectrum to be restored into a pre-trained vocoder system to obtain a second restored audio signal specifically includes:

[0113] Input the spectrum to be restored into the prior encoder to obtain the prior Gaussian distribution corresponding to the spectrum to be restored;

[0114] Sampling the prior Gaussian distribution corresponding to the spectrum to be restored to obtain the second intermediate latent variable;

[0115] The second intermediate latent variable is input into the decoder for audio decoding to obtain a second restored audio signal.

[0116] In this embodiment, the spectrum to be restored is first input into the prior encoder to obtain its corresponding prior Gaussian distribution. The task of the prior encoder is to map the spectrum features to the latent space and generate an ideal Gaussian distribution by learning the statistical characteristics of the spectrum. This Gaussian distribution not only reflects the characteristics of the input spectrum, but also provides an important potential representation for the decoding process. Next, sampling is performed from the obtained prior Gaussian distribution to generate a second intermediate latent variable. The generation of this latent variable has a certain degree of randomness, allowing the model to explore different possibilities in the latent space, so that the decoder can generate a variety of audio signals. Finally, the second intermediate latent variable is input into the decoder for audio decoding. The decoder converts the latent variable back to the audio signal to generate a second restored audio signal. In this process, the decoder effectively reconstructs audio similar to the original signal by using the information in the latent variable, ensuring that the high-quality restoration of the audio signal is achieved while maintaining the signal characteristics.

[0117] Through the above steps, the system can effectively convert spectral information into audio signals, ensuring that the quality of the restored audio is consistent with the original signal, thereby improving the accuracy and diversity of audio reconstruction.

[0118] In the above embodiment, the present application discloses an audio signal processing method, which belongs to the field of artificial intelligence technology. The present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of training difficulty and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the training difficulty is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through hidden variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects.

[0119] In this embodiment, the audio signal processing method is executed on the electronic device (eg Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0120] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned spectrum information to be restored, the above-mentioned spectrum information to be restored can also be stored in a node of a blockchain.

[0121] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0122] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0123] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0125] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0126] Further references Figure 4 , as a response to the above Figure 3 The present application provides an embodiment of an audio signal processing device, and the device embodiment is Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0127] like Figure 4As shown, the audio signal processing device 400 described in this embodiment is used to run a pre-built vocoder system, and the vocoder system is built based on a variational autoencoder. The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The audio signal processing device 400 includes:

[0128] A loss function construction module 401 is used to construct a loss function of a vocoder system, wherein the loss function of the vocoder system includes a first loss term and a second loss term, the first loss term represents the audio signal reconstruction loss, and the second loss term represents the loss between the a priori encoder and the a posteriori encoder;

[0129] The training data acquisition module 402 is used to acquire pre-collected training data, wherein the training data is pre-collected original audio signals;

[0130] The a posteriori encoding module 403 is used to input the original audio signal into the a posteriori encoder to obtain a posteriori Gaussian distribution corresponding to the original audio signal;

[0131] A latent variable sampling module 404 is used to sample the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable;

[0132] An audio decoding module 405, configured to input the first intermediate latent variable into a decoder for audio decoding to obtain a first restored audio signal;

[0133] A loss calculation module 406, configured to calculate a vocoder loss by using a loss function of a vocoder system based on the first restored audio signal and the original audio signal;

[0134] An iterative training module 407, for transferring vocoder loss in the vocoder system using a back propagation algorithm, and iteratively training the vocoder system to obtain a pre-trained vocoder system;

[0135] The audio restoration module 408 is used to receive an audio restoration instruction, obtain a frequency spectrum to be restored, and input the frequency spectrum to be restored into a pre-trained vocoder system to obtain a second restored audio signal.

[0136] In the above embodiment, the present application discloses an audio signal processing device, which belongs to the field of artificial intelligence technology. The present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of training difficulty and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the training difficulty is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through latent variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects.

[0137] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0138] The computer device 5 includes a memory 51, a processor 52, and a network interface 53 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 5 with a memory 51, a processor 52, and a network interface 53, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0139] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0140] The memory 51 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 51 can be an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 51 can also be an external storage device of the computer device 5, such as a plug-in hard disk equipped on the computer device 5, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 51 can also include both the internal storage unit of the computer device 5 and its external storage device. In this embodiment, the memory 51 is generally used to store the operating system and various application software installed on the computer device 5, such as computer-readable instructions of the audio signal processing method, etc. In addition, the memory 51 can also be used to temporarily store various types of data that have been output or are to be output.

[0141] The processor 52 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 52 is generally used to control the overall operation of the computer device 5. In this embodiment, the processor 52 is used to run computer-readable instructions stored in the memory 51 or process data, such as computer-readable instructions for running the audio signal processing method.

[0142] The network interface 53 may include a wireless network interface or a wired network interface. The network interface 53 is generally used to establish a communication connection between the computer device 5 and other electronic devices.

[0143] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of training difficulty and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the training difficulty is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through hidden variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects.

[0144] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the audio signal processing method as described above.

[0145] In the above embodiments, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. The present application is based on a vocoder system constructed based on a variational autoencoder VAE, which improves the existing GAN vocoder in terms of training difficulties and poor generalization. By encoding the audio signal into a Gaussian distribution and introducing a loss term between the prior encoder and the posterior encoder, the stability of the model is effectively improved and the training difficulty is reduced. At the same time, the back-propagation algorithm is used to iteratively optimize the loss function, which ensures that the vocoder system can learn the audio signal features more efficiently, avoids the common mode collapse problem in the GAN vocoder, and enhances the generalization ability of the model through hidden variable sampling, so that it can better handle different audio inputs and provide more accurate audio reconstruction effects.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0148] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An audio signal processing method, characterized in that: The audio signal processing method is applied to a pre-built vocoder system, which is built based on a variational autoencoder. The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The audio signal processing method includes: Constructing a loss function of the vocoder system, wherein the loss function of the vocoder system comprises a first loss term and a second loss term, wherein the first loss term represents an audio signal reconstruction loss, and the second loss term represents a loss between the a priori encoder and the a posteriori encoder; Acquire pre-collected training data, wherein the training data is pre-collected original audio signals; Inputting the original audio signal into the posterior encoder to obtain a posterior Gaussian distribution corresponding to the original audio signal; Sampling the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable; Inputting the first intermediate latent variable into the decoder for audio decoding to obtain a first restored audio signal; Calculating a vocoder loss using a loss function of the vocoder system based on the first restored audio signal and the original audio signal; Propagate the vocoder loss in the vocoder system by using a back propagation algorithm, and iteratively train the vocoder system to obtain a pre-trained vocoder system; An audio restoration instruction is received, a frequency spectrum to be restored is acquired, and the frequency spectrum to be restored is input into the pre-trained vocoder system to obtain a second restored audio signal.

2. The method for processing a frequency signal according to claim 1, wherein: After the step of inputting the original audio signal into the a posteriori encoder to obtain the a posteriori Gaussian distribution corresponding to the original audio signal, the method further includes: Performing Fourier transform on the original audio signal to obtain an original audio spectrum corresponding to the original audio signal; The original audio spectrum is input into the a priori encoder to obtain a priori Gaussian distribution corresponding to the original audio spectrum.

3. The method for processing a frequency signal according to claim 2, wherein: The step of performing Fourier transform on the original audio signal to obtain an original audio spectrum corresponding to the original audio signal specifically includes: Performing a preprocessing operation on the original audio signal, wherein the preprocessing operation includes a denoising operation, a framing operation, and a windowing operation; Performing short-time Fourier transform on the preprocessed original audio signal to obtain an initial audio spectrum; Performing Mel filtering on the initial audio spectrum to obtain a Mel spectrum; Performing a logarithm operation on the Mel spectrum, and performing a discrete cosine transform on the Mel spectrum after taking the logarithm, to obtain a Mel spectrum graph; An original audio spectrum corresponding to the original audio signal is obtained from the Mel-spectrogram.

4. The method for processing a frequency signal according to claim 2, wherein: The step of calculating the vocoder loss by using the loss function of the vocoder system based on the first restored audio signal and the original audio signal specifically comprises: Calculate the error between the first restored audio signal and the original audio signal based on the first loss term to obtain a first loss; Performing Fourier transform on the first restored audio signal to obtain a restored audio spectrum corresponding to the first restored audio signal; Calculate the error between the restored audio spectrum and the original audio spectrum based on the second loss term to obtain a second loss; The sum of the first loss and the second loss is calculated to obtain the vocoder loss.

5. The method for processing a frequency signal according to claim 1, wherein: The step of calculating the error between the first restored audio signal and the original audio signal based on the first loss term to obtain the first loss specifically includes: According to a preset audio signal sampling rule, the first restored audio signal and the original audio signal are sampled respectively to obtain audio sampling signals; According to a preset spectrum feature sampling rule, the original audio spectrum and the restored audio spectrum are sampled respectively to obtain audio spectrum features, wherein the audio signal sampling rule matches the spectrum feature sampling rule; A decoder parameter is obtained, and an error between the first restored audio signal and the original audio signal is calculated based on the decoder parameter, the audio sampling signal, and the audio spectrum feature to obtain the first loss.

6. The method for processing a frequency signal according to claim 5, wherein: The step of calculating the error between the restored audio spectrum and the original audio spectrum based on the second loss term to obtain the second loss specifically includes: Obtaining the mean and standard deviation of the posterior Gaussian distribution, and obtaining the mean and standard deviation of the prior Gaussian distribution; Obtain a posteriori encoder parameters and a priori encoder parameters; Calculate the KL divergence between the posterior Gaussian distribution and the prior Gaussian distribution based on the posterior encoder parameters, the prior encoder parameters, the mean and standard deviation of the posterior Gaussian distribution, and the mean and standard deviation of the prior Gaussian distribution; The KL divergence is taken as the error between the restored audio spectrum and the original audio spectrum to obtain the second loss.

7. The method for processing a frequency signal according to claim 1, wherein: The step of obtaining the spectrum to be restored, inputting the spectrum to be restored into the pre-trained vocoder system, and obtaining the second restored audio signal specifically includes: Inputting the spectrum to be restored into the a priori encoder to obtain a priori Gaussian distribution corresponding to the spectrum to be restored; Sampling the prior Gaussian distribution corresponding to the spectrum to be restored to obtain a second intermediate latent variable; The second intermediate latent variable is input into the decoder for audio decoding to obtain a second restored audio signal.

8. An audio signal processing device, characterized in that: The audio signal processing device is used to run a pre-built vocoder system, which is built based on a variational autoencoder. The vocoder system includes a priori encoder, a posteriori encoder and a decoder. The audio signal processing device includes: A loss function construction module, used to construct a loss function of the vocoder system, wherein the loss function of the vocoder system includes a first loss term and a second loss term, the first loss term represents the audio signal reconstruction loss, and the second loss term represents the loss between the a priori encoder and the a posteriori encoder; A training data acquisition module, used to acquire pre-collected training data, wherein the training data is a pre-collected original audio signal; a posterior encoding module, used for inputting the original audio signal into the posterior encoder to obtain a posterior Gaussian distribution corresponding to the original audio signal; A latent variable sampling module, used for sampling the posterior Gaussian distribution corresponding to the original audio signal to obtain a first intermediate latent variable; an audio decoding module, configured to input the first intermediate latent variable into the decoder for audio decoding to obtain a first restored audio signal; a loss calculation module, configured to calculate a vocoder loss by using a loss function of the vocoder system based on the first restored audio signal and the original audio signal; an iterative training module, for transferring the vocoder loss in the vocoder system by using a back propagation algorithm, and iteratively training the vocoder system to obtain a pre-trained vocoder system; The audio restoration module is used to receive an audio restoration instruction, obtain a frequency spectrum to be restored, and input the frequency spectrum to be restored into the pre-trained vocoder system to obtain a second restored audio signal.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio signal processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio signal processing method according to any one of claims 1 to 7 are implemented.

Images