High-fidelity audio generation method and device and storage medium

Abstract

The invention discloses a high-fidelity audio generation method and device and a storage medium, and the method comprises the steps: carrying out the down-sampling interpolation or low-pass filtering of a high-fidelity audio data set, and generating a frequency spectrum missing audio data set; inputting the frequency spectrum missing audio data set into a teacher model for prediction, and generating a predicted high-fidelity data set; performing audio feature extraction on the frequency spectrum missing audio data set; training a student model by taking the audio features as input and taking a real label extracted from the high-fidelity audio data set and a soft label extracted from the predicted high-fidelity data set as training labels to obtain a high-fidelity audio generation model; and extracting the audio features of the collected frequency spectrum missing audio, and inputting the audio features into the high-fidelity audio generation model for prediction to obtain the high-fidelity audio. According to the method, the student model with low time delay and small calculation amount is trained in a knowledge distillation mode, so that deployment at a hardware end is realized, and the technical problems that an existing large model is high in time delay, large in calculation amount and difficult to deploy at the hardware end are solved.

Description

Technical Field

[0001] The present application relates to the technical field of audio signal processing, and in particular to a high-fidelity audio generation method, device and storage medium. Background Art

[0002] In many application scenarios, such as conference system high-fidelity audio, users' listening experience can be enhanced. However, the microphone device will miss the high-frequency signal of the audio recorded in special scenarios such as far field, resulting in the lack of high-frequency information of human voice signals in many occasions, and the sound is dull.

[0003] Traditional methods for generating high-fidelity audio include adding harmonics and spectral interpolation, but the performance of these two methods often fails to meet the requirements and can easily introduce additional noise. In addition, spectral equalization can be used to fine-tune the phenomenon of uneven spectral components, but it is not effective for cases with severe spectrum loss. In recent years, deep learning algorithms have become a hot topic for generating high-fidelity audio. However, many time-domain convolution models are large, have poor real-time performance, and are difficult to deploy on the hardware side. They are not suitable for scenarios with high real-time requirements, such as conferences. Summary of the invention

[0004] The present application provides a high-fidelity audio generation method, device and storage medium, which are used to improve the technical problems of existing large time-domain convolution models, such as high latency, large amount of calculation, and difficulty in hardware deployment.

[0005] In view of this, the first aspect of the present application provides a high-fidelity audio generation method, comprising:

[0006] Downsample, interpolate or low-pass filter the high-fidelity audio dataset to generate a spectrum-missing audio dataset;

[0007] Inputting the spectrum-missing audio dataset into the teacher model for prediction to generate a predicted high-fidelity dataset;

[0008] Performing audio feature extraction on the spectrum-missing audio data set;

[0009] Taking the audio features as input, and using the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set as training labels to train a student model, thereby obtaining a high-fidelity audio generation model; the student model is a lightweight network;

[0010] The audio features of the collected spectrally missing audio are extracted and input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0011] Optionally, extracting audio features from the spectrum-missing audio dataset includes:

[0012] Performing triangular filtering on the frequency point energy of each frame signal in the spectrum-missing audio data set to obtain the frequency band energy of each frame signal;

[0013] Performing discrete cosine transform on the frequency band energy of each frame signal to obtain a transformed first-order difference signal and a second-order difference signal;

[0014] The audio features are generated based on the frequency band energy, total energy and first-order difference signal and second-order difference signal after the frequency band energy transformation of each frame signal. Optionally, the audio features are used as input, and the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set are used as training labels to train the student model to obtain a high-fidelity audio generation model, including:

[0015] The audio features of each audio sample in the spectrum-missing audio data set are input into the student model for prediction, and the loss value is calculated according to the audio prediction result of each audio sample and the real label and soft label of each audio sample;

[0016] The network parameters of the student model are updated by the loss value until the student model converges to obtain a trained high-fidelity audio generation model.

[0017] Optionally, the student model takes minimizing the frequency band gain as a training objective, wherein the frequency band gain is calculated as follows:

[0018]

[0019] Where, mask is the frequency band gain; E a or c is a high-fidelity audio dataset or predicts the frequency band energy of each frame signal in a high-fidelity dataset; E b It is the frequency band energy of each frame signal in the spectrum-missing audio dataset; mask_scale is the scaling value.

[0020] Optionally, the extracting audio features from the spectrum-missing audio dataset also includes:

[0021] The high-frequency spectrum signal is interpolated by using the low-frequency spectrum signal of each frame signal in the spectrum-missing audio data set, and the high-frequency spectrum signal after interpolation is superimposed with the high-frequency spectrum signal before interpolation to obtain a completed spectrum-missing audio data set;

[0022] Alternatively, a high frequency spectrum signal of each frame signal in the spectrum-missing audio data set is filled by adding white noise to obtain a completed spectrum-missing audio data set.

[0023] Optionally, the calculating the loss value according to the audio prediction result of each audio sample and the real label and soft label of each audio sample includes:

[0024] Calculate a first loss value according to the audio prediction result of each audio sample and the true label of each audio sample;

[0025] Calculating a second loss value according to the audio prediction result of each audio sample and the soft label of each audio sample;

[0026] A weighted sum is performed on the first loss value and the second loss value to obtain a final loss value.

[0027] Optionally, the extracting and collecting audio features of the spectrum-missing audio and inputting them into the high-fidelity audio generation model for prediction to obtain high-fidelity audio includes:

[0028] Extracting audio features of the collected spectral-missing audio and inputting them into the high-fidelity audio generation model for prediction to obtain a frequency band gain prediction value;

[0029] Multiplying the predicted value of the frequency band gain by the scaling value to obtain the gain of each frequency band;

[0030] Perform an inverse triangular transformation on the gain of each frequency band to obtain the gain of each frequency point;

[0031] The gain of each frequency point is applied to the frequency point of the spectrum-missing audio, and inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

[0032] Optionally, the student model includes two fully connected layers and three gated recurrent layers;

[0033] The first fully connected layer is used to process the input features;

[0034] The first gated recurrent layer is used to process the output of the first fully connected layer;

[0035] The second gated recurrent layer is used to process a first concatenated feature after concatenating the input feature, the output of the first fully connected layer, and the output of the first gated recurrent layer;

[0036] The third gated recurrent layer is used to process a second concatenated feature after concatenating the input feature, the output of the first gated recurrent layer, and the output of the second gated recurrent layer;

[0037] The second fully connected layer is used to process the output of the third gated recurrent layer and output the frequency band gain prediction value.

[0038] A second aspect of the present application provides a high-fidelity audio generation device, comprising:

[0039] A preprocessing unit, used to perform downsampling, interpolation or low-pass filtering on the high-fidelity audio data set to generate a spectrum-missing audio data set;

[0040] A first prediction unit, configured to input the spectrum-missing audio data set into a teacher model for prediction, and generate a predicted high-fidelity data set;

[0041] A feature extraction unit, used to extract audio features from the spectrum-missing audio data set;

[0042] A training unit, configured to train a student model using the audio features as input, and using real labels extracted from the high-fidelity audio data set and soft labels extracted from the predicted high-fidelity data set as training labels to obtain a high-fidelity audio generation model; the student model is a lightweight network;

[0043] The second prediction unit is used to extract the audio features of the collected spectrum-missing audio and input them into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0044] A third aspect of the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, it implements any high-fidelity audio generation method described in the first aspect.

[0045] It can be seen from the above technical solutions that this application has the following advantages:

[0046] The high-fidelity audio generation method provided in the present application trains the student model through supervision of the teacher model, so that the student model learns the data probability distribution characteristics of the teacher model, and also prevents problems such as overfitting in the student model training, thereby migrating the knowledge of the highly complex teacher model to a lightweight student model with high real-time performance and low computational complexity. The student model is suitable for scenarios with high real-time requirements such as meetings. That is, the present application adopts the knowledge distillation method to train a lightweight student model with low latency and low computational complexity based on a large cloud model, which can be deployed on the hardware side, improving the technical problems of the existing large time domain convolution model with high latency, large computational complexity, and difficulty in deployment on the hardware side. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic flow chart of a high-fidelity audio generation method provided in an embodiment of the present application;

[0049] Figure 2Another flowchart of a high-fidelity audio generation method provided in an embodiment of the present application;

[0050] Figure 3 A structural schematic diagram of a student model provided in an embodiment of the present application;

[0051] Figure 4 A schematic structural diagram of a high-fidelity audio generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] For easier understanding, please refer to Figure 1 and Figure 2 , the embodiment of the present application provides a high-fidelity audio generation method, comprising:

[0054] Step 110: downsample, interpolate or low-pass filter the high-fidelity audio data set to generate a spectrum-missing audio data set.

[0055] A high-fidelity audio dataset (dataset a) can be obtained from an existing speech dataset, or it can be obtained through on-site collection, and then the true labels for student model training can be extracted from the high-fidelity audio dataset.

[0056] After obtaining the high-fidelity audio data set, each audio sample in the high-fidelity audio data set is downsampled and interpolated or filtered through a low-pass filter to generate a spectrum-missing audio data set (data set b) that lacks high-frequency signals.

[0057] Step 120: Input the spectrum-missing audio dataset into the teacher model for prediction to generate a predicted high-fidelity dataset.

[0058] Input the spectrum-missing audio dataset into the trained teacher model for prediction to obtain a predicted high-fidelity dataset (dataset c). You can use an open source large model as the teacher model, or you can train a model as the teacher model. Then extract the soft labels for student model training from the predicted high-fidelity dataset.

[0059] Step 130: extract audio features from the spectrum-missing audio dataset.

[0060] The frequency point energy of each frame signal in the spectrum-missing audio data set is triangular filtered to obtain the frequency band energy of each frame signal; the frequency band energy of each frame signal is discrete cosine transformed to obtain the transformed first-order difference signal and second-order difference signal; audio features are generated based on the frequency band energy of each frame signal, the total energy and the first-order difference signal and second-order difference signal after the frequency band energy transformation.

[0061] In order to improve the real-time performance of the student model, the nbins frequency points of each frame signal are compressed into nbands frequency bands. Band compression can reduce the size of the student model and reduce the amount of calculation during real-time inference. The frequency point energy to band energy can be compressed through a triangular filter. The audio features of the extracted spectrum-missing audio dataset (i.e., the input features of the student model) are the nbands frequency band energy values ​​(F 频带 ) and the total energy of the frame signal (F 总 ) Take the log and the frequency band energy value to perform discrete cosine transform on the first-order and second-order differential signals (F 一阶 、F 二阶 ), that is, input features = logF 频带 +logF 总 +F 一阶 +F 二阶 , the input feature dimension is n_bands+1+2*ceps.

[0062] Step 140: Using the audio features as input, and using the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels to train the student model, thereby obtaining a high-fidelity audio generation model.

[0063] In an embodiment of the present application, the student model is a lightweight network, specifically a lightweight recurrent neural network. Figure 3 An example network structure of a student model given in an embodiment of the present application, the student model includes two fully connected layers and three gated recurrent layers; the first fully connected layer (Dense1) is used to process input features; the first gated recurrent layer (Gru1) is used to process the output of the first fully connected layer; the second gated recurrent layer (Gru2) is used to process the first concatenated features after feature concatenation of the input features, the output of the first fully connected layer and the output of the first gated recurrent layer; the third gated recurrent layer (Gru3) is used to process the second concatenated features after feature concatenation of the input features, the output of the first gated recurrent layer and the output of the second gated recurrent layer; the second fully connected layer (Dense2) is used to process the output of the third gated recurrent layer and output a frequency band gain prediction value. Figure 3The tanh and sigmoid after each layer represent the activation function of the corresponding layer. The output of the audio features extracted from the spectrum-missing audio dataset after passing through the first fully connected layer is sent to the first gated recurrent layer and the second gated recurrent layer. The input of the second gated recurrent layer is the output of the first gated recurrent layer, the output of the first fully connected layer, and the audio features; the input of the third gated recurrent layer is the output of the first and second gated recurrent layers and the audio features. The residual input method of the student model can promote deeper training of the model and prevent problems such as gradient disappearance or explosion. Finally, the output of the third gated recurrent layer passes through the second fully connected layer to obtain the final mask (band gain).

[0064] In an embodiment of the present application, two labels will be used in combination during student model training. In particular, when the amount of high-fidelity audio data recorded actually is insufficient, a large amount of high-fidelity audio can be generated through the teacher model to obtain labels for student model training, so as to improve the training effect of the student model. Specifically, the audio features of each audio sample in the extracted spectrum-missing audio data set are input into the student model for prediction, and the loss value is calculated based on the audio prediction results of each audio sample and the real label and soft label of each audio sample; the network parameters of the student model are updated by the loss value until the student model converges to obtain a trained high-fidelity audio generation model. Among them, the calculation formula for the training label is:

[0065]

[0066] In the formula, mask is the band gain, that is, the calculated label value; E a or c is a high-fidelity audio dataset or predicts the frequency band energy of each frame signal in a high-fidelity dataset; E b It is the frequency band energy of each frame signal in the spectrum-missing audio dataset; mask_scale is the scaling value. When extracting the true label from the high-fidelity audio dataset, the ratio of the frequency band energy of each frame signal in the high-fidelity audio dataset to the frequency band energy of the corresponding frame signal in the spectrum-missing audio dataset is used for calculation; when extracting the soft label from the predicted high-fidelity audio dataset, the ratio of the frequency band energy of each frame signal in the predicted high-fidelity audio dataset to the frequency band energy of the corresponding frame signal in the spectrum-missing audio dataset is used for calculation. The present application takes into account that the ratio of the frequency band energy of the same frame signal may be greater than 1, so a scaling value mask_scale is set to scale it to between 0-1. The specific value of mask_scale is determined based on the statistics of the audio training data.

[0067] Since the audio samples in dataset b may have missing spectrum, the frequency band energy may be equal to 0, that is, E bIs zero or very small, which will result in a large mask_scale being required to scale it to between 0-1. Therefore, for this situation, it is necessary to complete the spectrum information first. In the embodiment of the present application, it is preferred to use interpolation to complete the spectrum information, that is, the low-frequency spectrum signal of each frame signal in the spectrum-missing audio data set is used to interpolate the high-frequency spectrum signal, and the high-frequency spectrum signal after interpolation is superimposed with the high-frequency spectrum signal before interpolation to obtain the completed audio sample. For example, for an audio sample with a sampling rate of 16k, the completed spectrum is between 4-8k, and the frequency band may be missing. The 4-8k spectrum signal is interpolated using the 0-4k spectrum signal, and then the interpolated 4-8k spectrum signal and the 4-8k spectrum signal before interpolation are superimposed, and the superimposed spectrum signal is the spectrum information of the input feature. This prevents the situation where the calculated label_mask is much larger than the set maximum scaling value max_scale due to the lack of high-frequency signals in the original audio. In addition to using the interpolation method to pre-supplement the spectrum, it is also possible to use the method of adding white noise to first fill in the lower frequency points of the spectrum signal.

[0068] When calculating the loss value, the first loss value (real_loss) is calculated according to the audio prediction result of each audio sample (the predicted mask value) and the true label of each audio sample (the calculated mask value); the second loss value (soft_loss) is calculated according to the audio prediction result of each audio sample and the soft label of each audio sample; the first loss value and the second loss value are weighted and summed to obtain the final loss value (loss), that is:

[0069]

[0070] By adjusting the coefficient To control the ratio of the two loss functions, Between 0 and 1. The calculation of real_loss and soft_loss can use loss functions such as L1 and L2. If it is a conference speech generation scenario, more complex loss functions such as speech intelligibility can also be added. After calculating the total loss value loss, the network parameters of the student model are reversely updated by the loss value loss until the student model converges (such as reaching the maximum number of iterations or the training error is lower than the preset error threshold, etc.), and the trained student model is obtained. The trained student model is used as the high-fidelity audio generation model.

[0071] It is understandable that after the student model training is completed, real-time high-fidelity audio generation test can be performed. For example, there is an audio test sample with a known missing spectrum. First, the low-frequency signal of the audio test sample is used to interpolate the high-frequency signal, and the high-frequency signals before and after the interpolation are superimposed to obtain the completed audio test sample. Then, the audio features corresponding to the audio test sample are extracted and input into the trained student model to obtain the mask value predicted by the network. The mask value is multiplied by mask_scale to obtain the gain of each frequency band. The gain of each frequency point is calculated through the inverse triangular transformation, and then applied to the frequency point of each frame signal. After the inverse Fourier transform and delay superposition operation, the high-fidelity audio time domain signal predicted by the model is obtained.

[0072] Through the supervision of the teacher model, the student model learns the data probability distribution characteristics of the teacher model and prevents problems such as overfitting during training, thereby transferring the knowledge of the highly complex teacher model to the more real-time student model.

[0073] Step 150: extract the audio features of the collected spectrum-missing audio and input them into a high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0074] Before extracting the audio features of the collected audio with missing spectrum, the high-frequency signal of the collected audio with missing spectrum is first complemented (by interpolation or adding white noise, etc.) to obtain the complemented audio with missing spectrum; the audio features of the extracted and complemented audio with missing spectrum are input into the high-fidelity audio generation model for prediction to obtain the predicted value of frequency band gain; the predicted value of frequency band gain is multiplied by the scaling value to obtain the gain of each frequency band; the gain of each frequency band is inversely triangularly transformed to obtain the gain of each frequency point; the gain of each frequency point is applied to the frequency point of the audio with missing spectrum, and an inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

[0075] The high-fidelity audio generation method provided in the present application trains the student model through the supervision of the teacher model, so that the student model learns the data probability distribution characteristics of the teacher model, and also prevents problems such as overfitting in the student model training, thereby migrating the knowledge of the highly complex teacher model to a lightweight student model with high real-time performance and low computational complexity. The student model is suitable for scenarios with high real-time requirements such as meetings, and can be deployed on the hardware side, improving the technical problems of the existing large time domain convolution model with high latency, large computational complexity, and difficulty in hardware deployment. Furthermore, the present application greatly reduces the computational complexity of the model's real-time operation and the resource consumption of the model deployment through methods such as frequency band compression and model structure optimization, further improving the real-time performance of the student model.

[0076] Please refer to Figure 4 , a high-fidelity audio generation device provided in an embodiment of the present application is characterized by comprising:

[0077] A preprocessing unit, used to perform downsampling, interpolation or low-pass filtering on the high-fidelity audio data set to generate a spectrum-missing audio data set;

[0078] A first prediction unit is used to input the spectrum-missing audio dataset into the teacher model for prediction, and generate a predicted high-fidelity dataset;

[0079] A feature extraction unit, used for extracting audio features from a spectrum-missing audio data set;

[0080] A training unit is used to train a student model using audio features as input, and using real labels extracted from a high-fidelity audio dataset and soft labels extracted from a predicted high-fidelity dataset as training labels to obtain a high-fidelity audio generation model; the student model is a lightweight network;

[0081] The second prediction unit is used to extract the audio features of the collected spectrum-missing audio and input them into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

[0082] As a further improvement, the feature extraction unit is specifically used for:

[0083] Perform triangular filtering on the frequency point energy of each frame signal in the spectrum-missing audio data set to obtain the frequency band energy of each frame signal;

[0084] Performing discrete cosine transform on the frequency band energy of each frame signal to obtain a transformed first-order difference signal and a second-order difference signal;

[0085] The audio features are generated based on the frequency band energy, total energy, first-order difference signal and second-order difference signal after frequency band energy transformation of each frame signal.

[0086] As a further improvement, the training unit is specifically used for:

[0087] The audio features of each audio sample in the extracted spectrum-missing audio dataset are input into the student model for prediction, and the loss value is calculated according to the audio prediction results of each audio sample and the true label and soft label of each audio sample;

[0088] The network parameters of the student model are updated through the loss value until the student model converges to obtain a trained high-fidelity audio generation model.

[0089] Among them, the calculation formula for training labels is:

[0090]

[0091] Where, mask is the frequency band gain; E a or c is a high-fidelity audio dataset or predicts the frequency band energy of each frame signal in a high-fidelity dataset; Eb It is the frequency band energy of each frame signal in the spectrum-missing audio dataset; mask_scale is the scaling value.

[0092] As a further improvement, the second prediction unit is specifically used for:

[0093] Extract the audio features of the collected spectral missing audio and input them into the high-fidelity audio generation model for prediction to obtain the frequency band gain prediction value;

[0094] The gain of each frequency band is obtained by multiplying the predicted value of the frequency band gain by the scaling value;

[0095] Perform an inverse triangular transformation on the gain of each frequency band to obtain the gain of each frequency point;

[0096] The gain of each frequency point is applied to the frequency point of the audio with missing spectrum, and inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

[0097] As a further improvement, the device further comprises: a pre-processing unit;

[0098] The preprocessing unit is used to interpolate the high-frequency spectrum signal by using the low-frequency spectrum signal of each frame signal in the spectrum-missing audio data set, and superimpose the high-frequency spectrum signal after interpolation with the high-frequency spectrum signal before interpolation; or, fill the high-frequency spectrum signal in each frame signal by adding white noise.

[0099] This application trains the student model through teacher model supervision, so that the student model learns the data probability distribution characteristics of the teacher model, and also prevents problems such as overfitting in the student model training, thereby migrating the knowledge of the highly complex teacher model to a lightweight student model with high real-time performance and low computational complexity. The student model is suitable for scenarios with high real-time requirements such as meetings, and can be deployed on the hardware side, improving the technical problems of existing large time domain convolution models, such as high latency, large computational complexity, and difficulty in hardware deployment. Furthermore, this application greatly reduces the computational complexity of real-time model operations and resource consumption of model deployment through methods such as frequency band compression and model structure optimization, further improving the real-time performance of the student model.

[0100] The embodiment of the present application also provides an electronic device, characterized in that the device includes a processor and a memory;

[0101] The memory is used to store the program code and transmit the program code to the processor;

[0102] The processor is used to execute the high-fidelity audio generation method in the aforementioned method embodiment according to the instructions in the program code.

[0103] An embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the high-fidelity audio generation method in the aforementioned method embodiment is implemented.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0106] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0111] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A high-fidelity audio generation method, characterized in that: include: Downsample, interpolate or low-pass filter the high-fidelity audio dataset to generate a spectrum-missing audio dataset; Inputting the spectrum-missing audio dataset into the teacher model for prediction to generate a predicted high-fidelity dataset; Performing audio feature extraction on the spectrum-missing audio data set; Taking the audio features as input, and using the real labels extracted from the high-fidelity audio dataset and the soft labels extracted from the predicted high-fidelity dataset as training labels to train a student model, thereby obtaining a high-fidelity audio generation model; The student model is a lightweight network; The audio features of the collected spectrally missing audio are extracted and input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

2. The high-fidelity audio generation method according to claim 1, characterized in that: The extracting audio features from the spectrum-missing audio data set includes: Performing triangular filtering on the frequency point energy of each frame signal in the spectrum-missing audio data set to obtain the frequency band energy of each frame signal; Performing discrete cosine transform on the frequency band energy of each frame signal to obtain a transformed first-order difference signal and a second-order difference signal; The audio features are generated based on the frequency band energy, total energy, first-order difference signal and second-order difference signal after frequency band energy transformation of each frame signal.

3. The high-fidelity audio generation method according to claim 1, characterized in that: The method of taking the audio features as input, using the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set as training labels to train the student model to obtain a high-fidelity audio generation model includes: The audio features of each audio sample in the spectrum-missing audio data set are input into the student model for prediction, and the loss value is calculated according to the audio prediction result of each audio sample and the real label and soft label of each audio sample; The network parameters of the student model are updated by the loss value until the student model converges to obtain a trained high-fidelity audio generation model.

4. The high-fidelity audio generation method according to claim 3, characterized in that: The calculation formula of the training label is: Where, mask is the frequency band gain; E a or c is a high-fidelity audio dataset or predicts the frequency band energy of each frame signal in a high-fidelity dataset; E b It is the frequency band energy of each frame signal in the spectrum-missing audio dataset; mask_scale is the scaling value.

5. The high-fidelity audio generation method according to claim 4, characterized in that: The extracting audio features from the spectrum-missing audio data set also includes: The high-frequency spectrum signal is interpolated by using the low-frequency spectrum signal of each frame signal in the spectrum-missing audio data set, and the high-frequency spectrum signal after interpolation is superimposed with the high-frequency spectrum signal before interpolation to obtain a completed spectrum-missing audio data set; Alternatively, a high frequency spectrum signal of each frame signal in the spectrum-missing audio data set is filled by adding white noise to obtain a completed spectrum-missing audio data set.

6. The high-fidelity audio generation method according to claim 3, characterized in that: The calculating of the loss value according to the audio prediction result of each audio sample and the real label and soft label of each audio sample includes: Calculate a first loss value according to the audio prediction result of each audio sample and the true label of each audio sample; Calculating a second loss value according to the audio prediction result of each audio sample and the soft label of each audio sample; A weighted sum is performed on the first loss value and the second loss value to obtain a final loss value.

7. The high-fidelity audio generation method according to claim 4, characterized in that: The extracted and collected audio features of the spectrally missing audio are input into the high-fidelity audio generation model for prediction to obtain high-fidelity audio, including: Extracting audio features of the collected spectral-missing audio and inputting them into the high-fidelity audio generation model for prediction to obtain a frequency band gain prediction value; Multiplying the predicted value of the frequency band gain by the scaling value to obtain the gain of each frequency band; Perform an inverse triangular transformation on the gain of each frequency band to obtain the gain of each frequency point; The gain of each frequency point is applied to the frequency point of the spectrum-missing audio, and inverse Fourier transform and delay superposition are performed to obtain high-fidelity audio.

8. The high-fidelity audio generation method according to any one of claims 1 to 7, characterized in that: The student model includes two fully connected layers and three gated recurrent layers; The first fully connected layer is used to process the input features; The first gated recurrent layer is used to process the output of the first fully connected layer; The second gated recurrent layer is used to process a first concatenated feature after concatenating the input feature, the output of the first fully connected layer, and the output of the first gated recurrent layer; The third gated recurrent layer is used to process a second concatenated feature after concatenating the input feature, the output of the first gated recurrent layer, and the output of the second gated recurrent layer; The second fully connected layer is used to process the output of the third gated recurrent layer and output the frequency band gain prediction value.

9. A high-fidelity audio generation device, characterized in that: include: A preprocessing unit, used to perform downsampling, interpolation or low-pass filtering on the high-fidelity audio data set to generate a spectrum-missing audio data set; A first prediction unit, configured to input the spectrum-missing audio data set into a teacher model for prediction, and generate a predicted high-fidelity data set; A feature extraction unit, used to extract audio features from the spectrum-missing audio data set; A training unit, configured to train a student model using the audio features as input and the real labels extracted from the high-fidelity audio data set and the soft labels extracted from the predicted high-fidelity data set as training labels, so as to obtain a high-fidelity audio generation model; The student model is a lightweight network; The second prediction unit is used to extract the audio features of the collected spectrum-missing audio and input them into the high-fidelity audio generation model for prediction to obtain high-fidelity audio.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and when the program codes are executed by a processor, the high-fidelity audio generation method according to any one of claims 1 to 8 is implemented.

Images