A high-fidelity audio generation method, apparatus and storage medium

Abstract

The application discloses a high-fidelity audio generation method, device and storage medium, and the method comprises the following steps: performing down-sampling interpolation or low-pass filtering on a high-fidelity audio dataset to generate a spectrum-missing audio dataset; inputting the spectrum-missing audio dataset into a teacher model for prediction to generate a predicted high-fidelity dataset; performing audio feature extraction on the spectrum-missing audio dataset; training a student model by taking the audio features as input, taking the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels, and obtaining a high-fidelity audio generation model; inputting the audio features of the collected spectrum-missing audio into the high-fidelity audio generation model for prediction to obtain high-fidelity audio. The application adopts a knowledge distillation method to train a low-latency and small-computing student model, so that the student model can be deployed on a hardware end, and the technical problem that an existing large model has high latency, large computing amount and is difficult to be deployed on a hardware end is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signal processing, and in particular to a high-fidelity audio generation method and device and storage medium. BACKGROUND

[0002] In many application scenarios, high-fidelity audio can improve the listening experience of users in conference systems. However, the high-frequency signals of audio recorded by a microphone device in a special scene such as a far field are missing, resulting in a lack of high-frequency information of human voice signals in many cases and a dull listening experience.

[0003] Traditional high-fidelity audio generation methods include adding harmonics and spectral interpolation, but the performance of these two methods often fails to meet the requirements and additional noise is easily introduced. In addition, the use of spectral equalization can fine-tune the phenomenon of uneven spectral components, but the effect is not good for a serious spectral loss. In recent years, deep learning algorithms have become a hot direction for high-fidelity audio generation, but many time-domain convolution models are large and have poor real-time performance, making it difficult to deploy on hardware and unsuitable for real-time scenarios such as conferences. SUMMARY

[0004] The present application provides a high-fidelity audio generation method, device and storage medium to improve the technical problem that existing time-domain convolution large models have high latency, large computation, and are difficult to deploy on hardware.

[0005] Therefore, the first aspect of the present application provides a high-fidelity audio generation method, comprising:

[0006] Down-sampling interpolation or low-pass filtering is performed on a high-fidelity audio dataset to generate a spectral loss audio dataset;

[0007] The spectral loss audio dataset is input into a teacher model for prediction to generate a predicted high-fidelity dataset;

[0008] Audio features are extracted from the spectral loss audio dataset;

[0009] A student model is trained with the audio features as input, real labels extracted from the high-fidelity audio dataset, and soft labels extracted from the predicted high-fidelity dataset as training labels to obtain a high-fidelity audio generation model; the student model is a lightweight network;

[0010] Audio features of collected spectral loss audio are extracted and input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0011] Optionally, the audio feature extraction from the spectral loss audio dataset comprises:

[0012] triangular filtering is performed on the frequency point energy of each frame of signal in the spectrum missing audio data set to obtain the frequency band energy of each frame of signal;

[0013] discrete cosine transform is performed on the frequency band energy of each frame of signal to obtain the first-order difference signal and the second-order difference signal after transformation;

[0014] The audio features are generated based on the frequency band energy, total energy and first-order difference signal and second-order difference signal after transformation of the frequency band energy of each frame of signal. Optionally, the student model is trained with the audio features extracted from the spectrum missing audio data set as input, the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set as training labels to obtain a high-fidelity audio generation model, including:

[0015] The audio features of each audio sample extracted from the spectrum missing audio data set are input into the student model for prediction, and the loss value is calculated according to the audio prediction result of each audio sample and the real label and soft label of each audio sample;

[0016] The network parameters of the student model are updated through the loss value until the student model converges to obtain a trained high-fidelity audio generation model.

[0017] Optionally, the student model takes minimizing the band gain as the training target, wherein the calculation formula of the band gain is:

[0018]

[0019] In the formula, mask is the band gain; E a or c is the frequency band energy of each frame of signal in the high-fidelity audio data set or the predicted high-fidelity data set; b is the frequency band energy of each frame of signal in the spectrum missing audio data set; mask_scale is a scaling value.

[0020] Optionally, before the audio feature extraction on the spectrum missing audio data set, it further includes:

[0021] The high-frequency spectrum signal is interpolated using the low-frequency spectrum signal of each frame of signal in the spectrum missing audio data set, and the interpolated high-frequency spectrum signal is superimposed with the high-frequency spectrum signal before interpolation to obtain a completed spectrum missing audio data set;

[0022] Alternatively, the high-frequency spectrum signal of each frame of signal in the spectrum missing audio data set is filled by adding white noise to obtain a completed spectrum missing audio data set.

[0023] Optionally, the loss value is calculated according to the audio prediction result of each audio sample and the real label and soft label of each audio sample, including:

[0024] The first loss value is calculated according to the audio prediction result of each audio sample and the true label of each audio sample.

[0025] The second loss value is calculated according to the audio prediction result of each audio sample and the soft label of each audio sample.

[0026] The first loss value and the second loss value are weighted and summed to obtain a final loss value.

[0027] Optionally, the audio feature of the collected spectrum missing audio is input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio, comprising:

[0028] The audio feature of the collected spectrum missing audio is input into the high-fidelity audio generation model for prediction to obtain a frequency band gain prediction value;

[0029] The frequency band gain prediction value is multiplied by the scaling value to obtain the gain of each frequency band;

[0030] The gain of each frequency band is inversely triangularly transformed to obtain the gain of each frequency point;

[0031] The gain of each frequency point is applied to the frequency point of the spectrum missing audio, and inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

[0032] Optionally, the student model comprises two fully connected layers and three gated recurrent layers;

[0033] The first fully connected layer is used for processing the input feature;

[0034] The first gated recurrent layer is used for processing the output of the first fully connected layer;

[0035] The second gated recurrent layer is used for processing the first spliced feature obtained by splicing the input feature, the output of the first fully connected layer and the output of the first gated recurrent layer;

[0036] The third gated recurrent layer is used for processing the second spliced feature obtained by splicing the input feature, the output of the first gated recurrent layer and the output of the second gated recurrent layer;

[0037] The second fully connected layer is used for processing the output of the third gated recurrent layer to output a frequency band gain prediction value.

[0038] The second aspect of the present application provides a high-fidelity audio generation device, comprising:

[0039] The preprocessing unit is used for down-sampling interpolation or low-pass filtering on the high-fidelity audio dataset to generate a spectrum missing audio dataset;

[0040] a first prediction unit configured to input the spectrum missing audio data set into a teacher model for prediction to generate a predicted high-fidelity data set;

[0041] a feature extraction unit configured to perform audio feature extraction on the spectrum missing audio data set;

[0042] a training unit configured to train a student model with the audio features as input, with a real label extracted from the high-fidelity audio data set and a soft label extracted from the predicted high-fidelity data set as training labels, to obtain a high-fidelity audio generation model; the student model is a lightweight network;

[0043] a second prediction unit configured to input audio features of the collected spectrum missing audio into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0044] The third aspect of the present application provides a computer readable storage medium for storing program code, which, when executed by a processor, implements the high-fidelity audio generation method of any one of the first aspect.

[0045] From the above technical solutions, the present application has the following advantages:

[0046] The high-fidelity audio generation method provided by the present application trains a student model supervised by a teacher model, so that the student model learns the data probability distribution characteristics of the teacher model, and prevents overfitting and other problems in the training of the student model, thereby migrating the knowledge of the complex teacher model to a lightweight student model with high real-time performance and small calculation amount, which is suitable for scenarios such as conferences that require high real-time performance. That is, the present application uses knowledge distillation to train a low-latency, low-computational lightweight student model based on a cloud large model, which can be deployed on a hardware end, thereby improving the technical problem of high latency, large calculation amount and difficulty in deployment on a hardware end of the existing time domain convolution large model. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0048] Figure 1 A flowchart of a high-fidelity audio generation method provided by an embodiment of the present application;

[0049] Figure 2Another flowchart of a high-fidelity audio generation method provided by an embodiment of the present application is shown in FIG. 10.

[0050] Figure 3 A structural diagram of a student model provided by an embodiment of the present application is shown in FIG. 11.

[0051] Figure 4 A structural diagram of a high-fidelity audio generation device provided by an embodiment of the present application is shown in FIG. 12. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] For the convenience of understanding, please refer to Figure 1 and Figure 2 The present application provides a high-fidelity audio generation method, which comprises the following steps:

[0054] Step 110: down-sampling interpolation or low-pass filtering is performed on the high-fidelity audio dataset to generate a spectrum-missing audio dataset.

[0055] The high-fidelity audio dataset (dataset a) can be obtained from an existing speech dataset, or can be obtained by field collection, and then real labels for student model training are extracted from the high-fidelity audio dataset.

[0056] After obtaining the high-fidelity audio dataset, down-sampling interpolation is performed on each audio sample in the high-fidelity audio dataset, or a low-pass filter is used to filter each audio sample to generate a spectrum-missing audio dataset (dataset b) with missing high-frequency signals.

[0057] Step 120: the spectrum-missing audio dataset is input into a teacher model for prediction to generate a predicted high-fidelity dataset.

[0058] The spectrum-missing audio dataset is input into the trained teacher model for prediction to obtain a predicted high-fidelity dataset (dataset c). An open-source large model can be used as the teacher model, or a model can be trained as the teacher model. Then soft labels for student model training are extracted from the predicted high-fidelity dataset.

[0059] Step 130: audio feature extraction is performed on the spectrum-missing audio dataset.

[0060] The frequency point energy of each frame of signal in the spectrum missing audio data set is triangular filtered to obtain the frequency band energy of each frame of signal; the frequency band energy of each frame of signal is discrete cosine transformed to obtain the first-order difference signal and the second-order difference signal after transformation; and the audio features are generated based on the frequency band energy, the total energy of each frame of signal, and the first-order difference signal and the second-order difference signal after transformation of the frequency band energy.

[0061] In order to improve the real-time performance of the student model, nbins frequency points of each frame of signal are compressed into nbands frequency bands, the frequency band compression can reduce the size of the student model, and reduce the calculation amount in real-time inference. The frequency point energy to the frequency band energy can be compressed through a triangular filter. The extracted audio features of the spectrum missing audio data set (i.e. the input features of the student model) are nbands frequency band energy values (F 频带 ) of the current frame of signal taking log, and the total energy (F 总 ) of the frame of signal taking log and the first-order and second-order difference signals (F 一阶 , F 二阶 ) after discrete cosine transformation of the frequency band energy values, i.e. input features=logF 频带 +logF 总 +F 一阶 +F 二阶 , the input feature dimension is n_bands+1+2*ceps.

[0062] Step 140, training the student model by taking the audio features as input, and taking the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set as training labels, to obtain a high-fidelity audio generation model.

[0063] In the embodiments of the present application, the student model is a lightweight network, specifically a lightweight recurrent neural network. Figure 3 An example network structure of a student model given in the embodiments of the present application includes two fully connected layers and three gated recurrent layers; the first fully connected layer (Dense1) is used to process the input features; the first gated recurrent layer (Gru1) is used to process the output of the first fully connected layer; the second gated recurrent layer (Gru2) is used to process the first spliced features after splicing the input features, the output of the first fully connected layer and the output of the first gated recurrent layer; the third gated recurrent layer (Gru3) is used to process the second spliced features after splicing the input features, the output of the first gated recurrent layer and the output of the second gated recurrent layer; and the second fully connected layer (Dense2) is used to process the output of the third gated recurrent layer to output the frequency band gain prediction value. Figure 3The tanh and sigmoid after each layer represent the activation function of the corresponding layer. The output of the audio features extracted from the spectrum missing audio data set after the first fully connected layer is sent to the first gated recurrent layer and the second gated recurrent layer. The input of the second gated recurrent layer is the splicing of the output of the first gated recurrent layer, the output of the first fully connected layer and the audio features; the input of the third gated recurrent layer is the splicing of the output of the first and second gated recurrent layers and the audio features. The residual input mode of the student model can promote the training of the model to a deeper layer, and prevent problems such as gradient disappearance or explosion. Finally, the output of the third gated recurrent layer is subjected to the second fully connected layer to obtain the final mask (band gain).

[0064] In the embodiments of the present application, two kinds of labels will be used during the training of the student model. Especially when the amount of real recorded high-fidelity audio data is insufficient, a large amount of high-fidelity audio can be generated by the teacher model to obtain the labels for the training of the student model, so as to improve the training effect of the student model. Specifically, the extracted audio features of each audio sample in the spectrum missing audio data set are input into the student model for prediction, and the loss value is calculated according to the audio prediction result of each audio sample and the real label and soft label of each audio sample; the network parameters of the student model are updated through the loss value until the student model converges, and a trained high-fidelity audio generation model is obtained. The calculation formula of the training label is:

[0065]

[0066] In the formula, mask is the band gain, that is, the calculated label value; E a or c is the band energy of each frame signal in the high-fidelity audio data set or the predicted high-fidelity data set; E b is the band energy of each frame signal in the spectrum missing audio data set; mask_scale is a scaling value. When extracting the real label from the high-fidelity audio data set, the ratio of the band energy of each frame signal in the high-fidelity audio data set to the band energy of the corresponding frame signal in the spectrum missing audio data set is calculated; when extracting the soft label from the predicted high-fidelity audio data set, the ratio of the band energy of each frame signal in the predicted high-fidelity audio data set to the band energy of the corresponding frame signal in the spectrum missing audio data set is calculated. The present application considers that the ratio of the band energy of the same frame signal may be greater than 1, so a scaling value mask_scale is set to scale it to 0-1, and the specific value of mask_scale is determined according to the audio training data statistics.

[0067] Since the audio samples in the data set b may have spectrum missing, the band energy may be equal to 0, that is, E bFor zero or very small, which will result in the need for a large mask_scale to scale it to 0-1. So for this case, the spectral information needs to be completed first, and the interpolation method is preferred in the embodiments of the present application to complete the spectral information, that is, the low spectral signal of each frame signal in the spectral missing audio data set is used to interpolate the high spectral signal, and the interpolated high spectral signal is superimposed with the high spectral signal before interpolation to obtain the completed audio sample. For example, for an audio sample with a sampling rate of 16k, the completed spectrum is between 4-8k, and the spectrum signal of 0-4k is used to interpolate the 4-8k spectrum signal, and then the interpolated 4-8k spectrum signal is superimposed with the 4-8k spectrum signal before interpolation. The superimposed spectral signal is the spectral information of the input feature. In this way, the case that the calculated label_mask is much larger than the set maximum scaling value max_scale due to the missing of the high frequency signal of the original audio is prevented. In addition to using the interpolation method to pre-supplement the spectrum, a white noise adding method can also be used to fill the lower frequency points of the spectrum signal.

[0068] When calculating the loss value, a first loss value (real_loss) is calculated according to the audio prediction result (predicted mask value) of each audio sample and the true label (calculated mask value) of each audio sample; a second loss value (soft_loss) is calculated according to the audio prediction result of each audio sample and the soft label of each audio sample; the first loss value and the second loss value are weighted and summed to obtain the final loss value (loss), that is:

[0069]

[0070] The two loss function proportions are controlled by adjusting the coefficient , wherein between 0-1. The calculation of real_loss and soft_loss can use L1, L2, etc. loss function, and if it is a conference voice generation scene, a more complex loss function such as speech intelligibility can also be added. After calculating the total loss value loss, the network parameters of the student model are updated in reverse through the loss value loss until the student model converges (such as reaching the maximum iteration number or the training error being lower than the preset error threshold), and a trained student model is obtained. The trained student model is used as a high-fidelity audio generation model.

[0071] It can be understood that after the student model training is completed, a real-time high-fidelity audio generation test can be performed. For example, there is an audio test sample with a known spectrum missing. First, the high-frequency signal of the audio test sample is interpolated using the low-frequency signal of the audio test sample, and the high-frequency signals before and after interpolation are superimposed to obtain a completed audio test sample. Then, the audio features corresponding to the audio test sample are extracted and input into the trained student model to obtain the mask value predicted by the network. The mask value is multiplied by the mask_scale to obtain the gain of each frequency band. The gain of each frequency point is calculated by inverse triangular transformation. Then, the gain of each frequency point is applied to each frame of signal frequency point. After inverse Fourier transform and delay superposition operation, the high-fidelity audio time domain signal predicted by the model is obtained.

[0072] The student model learns the data probability distribution characteristics of the teacher model through the supervision of the teacher model, and also prevents overfitting and other problems in training, thereby migrating the knowledge of the high-complexity teacher model to the student model with higher real-time performance.

[0073] Step 150, extracting the audio features of the collected spectrum missing audio and inputting them into the high-fidelity audio generation model for prediction to obtain the high-fidelity audio.

[0074] Before extracting the audio features of the collected spectrum missing audio, the high-frequency signal of the collected spectrum missing audio is completed (by interpolation or adding white noise, etc.), to obtain the completed spectrum missing audio. The audio features of the completed spectrum missing audio are extracted and input into the high-fidelity audio generation model for prediction to obtain the frequency band gain prediction value. The frequency band gain prediction value is multiplied by the scaling value to obtain the gain of each frequency band. The gain of each frequency band is inverse triangular transformed to obtain the gain of each frequency point. The gain of each frequency point is applied to the frequency point of the spectrum missing audio, and inverse Fourier transform and delay superposition are performed to obtain the high-fidelity audio.

[0075] The high-fidelity audio generation method provided in the application trains a student model under the supervision of a teacher model, so that the student model learns the data probability distribution characteristics of the teacher model, and also prevents overfitting and other problems in student model training, thereby migrating the knowledge of the high-complexity teacher model to the lightweight student model with high real-time performance and small calculation amount. The student model is suitable for scenarios with high real-time requirements such as conferences, and can be deployed on a hardware end, thereby improving the technical problems of high latency, large calculation amount, and difficulty in deploying the existing time domain convolution large model on a hardware end. Further, the application greatly reduces the calculation complexity of real-time operation of the model and the resource consumption of model deployment through frequency band compression and model structure optimization, thereby further improving the real-time performance of the student model.

[0076] Please refer to Figure 4 The high-fidelity audio generation device provided in the embodiment of the application has the characteristics that it comprises:

[0077] a pre-processing unit, configured to perform down-sampling interpolation or low-pass filtering on the high-fidelity audio dataset to generate a spectrum-missing audio dataset;

[0078] a first prediction unit, configured to input the spectrum-missing audio dataset into a teacher model to generate a predicted high-fidelity dataset;

[0079] a feature extraction unit, configured to perform audio feature extraction on the spectrum-missing audio dataset;

[0080] a training unit, configured to train a student model by taking the audio features as input, and taking the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels, to obtain a high-fidelity audio generation model; the student model is a lightweight network;

[0081] a second prediction unit, configured to input the audio features of the collected spectrum-missing audio into the high-fidelity audio generation model to generate high-fidelity audio.

[0082] As a further improvement, the feature extraction unit is specifically configured to:

[0083] perform triangular filtering on the frequency point energy of each frame of signal in the spectrum-missing audio dataset to obtain the frequency band energy of each frame of signal;

[0084] perform discrete cosine transform on the frequency band energy of each frame of signal to obtain a first-order difference signal and a second-order difference signal after transformation;

[0085] generate the audio features based on the frequency band energy, the total energy, and the first-order difference signal and the second-order difference signal after transformation of the frequency band energy of each frame of signal.

[0086] As a further improvement, the training unit is specifically configured to:

[0087] input the extracted audio features of each audio sample in the spectrum-missing audio dataset into the student model to generate a prediction result of each audio sample, and calculate a loss value based on the prediction result of each audio sample, and the real labels and the soft labels of each audio sample;

[0088] update the network parameters of the student model by the loss value until the student model converges, to obtain the trained high-fidelity audio generation model.

[0089] wherein, the calculation formula of the training label is:

[0090]

[0091] In the formula, mask is the band gain; E a or c is the frequency band energy of each frame of signal in the high-fidelity audio dataset or the predicted high-fidelity dataset; Eb band energy of each frame signal in the spectrum missing audio data set; and mask_scale is a scaling value.

[0092] As a further improvement, the second prediction unit is specifically configured to:

[0093] extracting the audio feature of the collected spectrum missing audio as an input into the high-fidelity audio generation model for prediction to obtain a band gain prediction value;

[0094] multiplying the band gain prediction value by the scaling value to obtain the gain of each band;

[0095] performing inverse triangle transformation on the gain of each band to obtain the gain of each frequency point;

[0096] applying the gain of each frequency point to the frequency points of the spectrum missing audio, and performing inverse Fourier transform and delay superposition to obtain the high-fidelity audio.

[0097] As a further improvement, the device further comprises a preprocessing unit.

[0098] The preprocessing unit is configured to interpolate the high-frequency spectrum signal using the low-frequency spectrum signal of each frame signal in the spectrum missing audio data set, and superimpose the interpolated high-frequency spectrum signal and the high-frequency spectrum signal before interpolation; or, fill the high-frequency spectrum signal in each frame signal by adding white noise.

[0099] The present application trains the student model supervised by the teacher model, so that the student model learns the data probability distribution characteristics of the teacher model, and also prevents overfitting and other problems in the training of the student model, thereby migrating the knowledge of the high-complexity teacher model to the lightweight student model with high real-time performance and small calculation amount. The student model is suitable for scenarios with high real-time requirements such as conferences, and can be deployed on the hardware side, thereby improving the technical problems of high latency, large calculation amount, and difficulty in deploying the existing time-domain convolution large model on the hardware side. Further, the present application greatly reduces the calculation complexity of real-time operation of the model and the resource consumption of model deployment through band compression and model structure optimization, and further improves the real-time performance of the student model.

[0100] The present application also provides an electronic device, characterized in that the device comprises a processor and a memory.

[0101] The memory is configured to store program code and transmit the program code to the processor.

[0102] The processor is configured to execute the high-fidelity audio generation method in the foregoing method embodiments according to the instructions in the program code.

[0103] The embodiment of the present application further provides a computer readable storage medium, characterized by storing program codes, which are executed by a processor to implement the high-fidelity audio generation method in the foregoing method embodiment.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiment, which will not be repeated here.

[0105] The terms "first", "second", "third", "fourth" and the like (if any) in the description and the above drawings of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0106] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b and c can be single or multiple.

[0107] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0109] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0110] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for executing all or part of the steps of the method described in each embodiment of the present application by a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various program code storage media.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating high-fidelity audio, characterized in that, include: Perform downsampling interpolation or low-pass filtering on high-fidelity audio datasets to generate audio datasets with missing spectra. The missing spectrum audio dataset is input into the teacher model for prediction, generating a high-fidelity prediction dataset. Audio feature extraction is performed on the aforementioned audio dataset with missing spectrum. Using the audio features as input, and the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels, a student model is trained to obtain a high-fidelity audio generation model. The student model is a lightweight network; The audio features of the collected spectrum missing audio are extracted and input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

2. The high-fidelity audio generation method according to claim 1, characterized in that, The audio feature extraction from the spectrum-deficient audio dataset includes: The frequency energy of each frame signal in the spectrum-deficient audio dataset is triangularly filtered to obtain the frequency band energy of each frame signal. Discrete cosine transform is performed on the frequency band energy of each frame signal to obtain the transformed first-order differential signal and second-order differential signal; Audio features are generated based on the frequency band energy, total energy, and first-order and second-order differential signals after frequency band energy transformation of each frame signal.

3. The high-fidelity audio generation method according to claim 1, characterized in that, The process of training a student model using the audio features as input and the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels to obtain a high-fidelity audio generation model includes: The audio features of each audio sample in the extracted spectrum missing audio dataset are input into the student model for prediction. The loss value is calculated based on the audio prediction results of each audio sample and the true label and soft label of each audio sample. The network parameters of the student model are updated using the loss value until the student model converges, resulting in a trained high-fidelity audio generation model.

4. The high-fidelity audio generation method according to claim 3, characterized in that, The formula for calculating the training labels is: In the formula, mask is the bandwidth gain; E a or c For high-fidelity audio datasets or to predict the frequency band energy of each frame signal in a high-fidelity dataset; E b `mask_scale` represents the frequency band energy of each frame signal in the spectrum-missing audio dataset; `mask_scale` is the scaling value.

5. The high-fidelity audio generation method according to claim 4, characterized in that, Prior to extracting audio features from the spectrum-deficient audio dataset, the process also includes: The high-spectrum signal is interpolated using the low-spectrum signal of each frame signal in the spectrum-missing audio dataset. The interpolated high-spectrum signal is then superimposed with the original high-spectrum signal to obtain the completed spectrum-missing audio dataset. Alternatively, white noise can be added to fill the high-spectral signals of each frame in the spectrum-deficient audio dataset to obtain the completed spectrum-deficient audio dataset.

6. The high-fidelity audio generation method according to claim 3, characterized in that, The calculation of the loss value based on the audio prediction results of each audio sample and the true and soft labels of each audio sample includes: The first loss value is calculated based on the audio prediction results of each audio sample and the true label of each audio sample; The second loss value is calculated based on the audio prediction results of each audio sample and the soft label of each audio sample; The first loss value and the second loss value are weighted and summed to obtain the final loss value.

7. The high-fidelity audio generation method according to claim 4, characterized in that, The audio features of the extracted spectrum-deficient audio are input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio, including: The audio features of the collected spectrum missing audio are extracted and input into the high-fidelity audio generation model for prediction to obtain the frequency band gain prediction value; The gain of each frequency band is obtained by multiplying the predicted frequency band gain by the scaling value. Perform an inverse triangular transformation on the gain of each frequency band to obtain the gain at each frequency point; The gain of each frequency point is applied to the frequency point of the audio spectrum missing, and inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

8. The high-fidelity audio generation method according to any one of claims 1-7, characterized in that, The student model includes two fully connected layers and three gated recurrent layers; The first fully connected layer is used to process the input features; The first gated loop layer is used to process the output of the first fully connected layer; The second gated recurrent layer is used to process the first concatenated feature obtained by concatenating the input features, the output of the first fully connected layer, and the output of the first gated recurrent layer. The third gated recurrent layer is used to process the second concatenated feature, which is the concatenated feature of the input feature, the output of the first gated recurrent layer, and the output of the second gated recurrent layer. The second fully connected layer is used to process the output of the third gated recurrent layer, outputting the predicted bandwidth gain value.

9. A high-fidelity audio generation device, characterized in that, include: The preprocessing unit is used to downsample, interpolate, or low-pass filter the high-fidelity audio dataset to generate a spectrum-missing audio dataset. The first prediction unit is used to input the spectrum-missing audio dataset into the teacher model for prediction and generate a high-fidelity prediction dataset. The feature extraction unit is used to extract audio features from the spectrum-deficient audio dataset. The training unit is used to train a student model with the audio features as input and the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels, so as to obtain a high-fidelity audio generation model. The student model is a lightweight network; The second prediction unit is used to extract the audio features of the collected spectrum missing audio and input them into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the high-fidelity audio generation method according to any one of claims 1-8.

Images