Training method and device of multi-scene vocoder based on value zero decomposition, and audio generation method and device

Abstract

The application provides a training method and device of a multi-scene vocoder based on value zero decomposition, including: obtaining a plurality of audio samples; determining a mel spectrum sample of each audio sample; inputting the mel spectrum sample and a mel filter corresponding to the mel spectrum sample into a vocoder model to train the vocoder model, and obtaining an audio prediction result corresponding to each mel spectrum sample, wherein the vocoder model includes a value space model and a zero space model, the vocoder model generates a first feature according to the mel spectrum sample and the mel filter corresponding to the mel spectrum sample through the value space model, the vocoder model performs linear amplitude domain reconstruction and phase information recovery on the first feature through the zero space model to obtain the audio prediction result; determining a loss value between the audio prediction result and the audio sample; adjusting parameters of the vocoder model according to the loss value to obtain a trained vocoder model. In this way, the vocoder model can adapt to multiple different scenes, thereby saving computing resources without training vocoders in different scenes.

Description

Training method, audio generation method and device for multi-scene vocoders based on value zero factorization Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a training method, audio generation method and apparatus for a multi-scene vocoder based on value-zero decomposition. Background Technology

[0002] Vocoders are used in tasks such as speech synthesis, audio generation, music generation, and speech enhancement. In these tasks, when a user provides a text description, a generative model estimates the corresponding Mel spectrogram of the audio. In speech enhancement tasks, the Mel spectrogram of noisy speech is extracted, and then a clean speech Mel spectrogram is recovered using a speech enhancement algorithm. The vocoder is responsible for converting this to the corresponding time-domain waveform. Although existing deep learning-based neural vocoders can effectively achieve the Mel spectrogram to waveform conversion...

[0003] However, current vocoders, after training, can only adapt to one Mel spectrum configuration. When generating audio for multiple scenarios, vocoders for different scenarios need to be trained to adapt to the Mel spectrum configurations for each scenario. This requires a large amount of computing resources, resulting in significant computational resource consumption. Summary of the Invention

[0004] This application provides a training method, audio generation method and apparatus for a multi-scene vocoder based on value zero decomposition, which enables the vocoder model to adapt to a variety of different scenarios, thereby eliminating the need to train vocoders for different scenarios and saving computational resources.

[0005] Firstly, this application provides a training method for a multi-scenario vocoder based on value-zero decomposition, the method comprising:

[0006] Obtain multiple audio samples and the label information for each audio sample;

[0007] The Mel spectrum sample of each audio sample is determined. The Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations. The Mel filters with different parameter configurations are adapted to different scenarios.

[0008] The Mel spectrum samples and their corresponding Mel filters are input into the vocoder model to train the vocoder model and obtain the audio prediction result for each Mel spectrum sample. The vocoder model includes a value space model and a null space model. The vocoder model generates the first feature based on the Mel spectrum samples and their corresponding Mel filters using the value space model. The vocoder model then reconstructs the first feature in the linear amplitude domain and recovers the phase information using the null space model to obtain the audio prediction result.

[0009] Determine the loss value between the audio prediction result and the audio sample;

[0010] Based on the loss value, adjust the parameters of the vocoder model to obtain the trained vocoder model.

[0011] According to the above scheme, Mel spectrum samples of audio samples in different scenarios are generated based on Mel filters with different parameter configurations. When training the vocoder model, the Mel filters and Mel spectrum samples from different scenarios enable the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter parameter configuration, thus saving computational resources. During the training process, to ensure the vocoder model can adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, thereby transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain the audio prediction result. Based on the loss value between the audio prediction result and the audio sample, the parameters of the vocoder model are adjusted to obtain the trained vocoder.

[0012] In one possible implementation, determining the Mel spectrum sample for each audio sample includes:

[0013] Based on the pre-configured band number range and frequency range, generate Mel filters with different parameter configurations;

[0014] Based on the audio samples, determine the target Mel filter corresponding to the audio samples and the target parameter configuration corresponding to the target Mel filter;

[0015] Based on the target Mel filter and target parameter configuration, Mel spectrum samples of the audio samples are generated.

[0016] In this way, it is possible to generate Mel spectrum samples of audio samples in different scenarios.

[0017] In one possible implementation, the first feature satisfies the following formula:

[0018]

[0019] in, Indicates the first feature, A + Let A represent the pseudo-inverse transform matrix of the Mel filter. + The size is |X mel | indicates a Mel spectrum sample.

[0020] In this way, the Mel spectrum samples are transformed from the Mel domain to the linear domain, and the Mel samples of audio samples in different scenarios are used to train the vocoder model, so that the trained vocoder model can be adapted to audio in different scenarios.

[0021] In one possible implementation, the vocoder model reconstructs the first feature in the linear amplitude domain and recovers its phase information using a null space model to obtain the audio prediction result, including:

[0022] The vocoder model uses a null space model to compress and encode the spectral features in the first feature in order from low frequency to high frequency, thus obtaining the encoded features;

[0023] The target audio is obtained by modeling the encoded features;

[0024] Amplitude and phase reconstruction are performed on the target audio to obtain the audio prediction result.

[0025] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0026] In one possible implementation, the audio prediction result Satisfy the following formula:

[0027]

[0028] in, This represents the magnitude estimate of the first feature. This represents the phase estimation of the first feature.

[0029] Let I represent the representation function of the null space model, and let I represent the identity matrix.

[0030] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0031] In one possible implementation, the loss value includes at least one of the following:

[0032] Logarithmic amplitude loss, phase loss, real-to-imaginary part loss, Mel spectrum loss, and uniformity loss;

[0033] Among them, the logarithmic magnitude loss value Satisfy the following formula:

[0034]

[0035] in, X represents the value of the audio prediction result at frequency index f and time frame index t. f,t This represents the value of the tag information when the frequency index is f and the time frame index is t, where F and T represent the frequency dimension size and the number of time frames, respectively. log(.) represents the logarithm operation, and ||.|2 represents the L2 norm.

[0036] Phase loss value Satisfy the following formula:

[0037]

[0038] Where "*" represents a two-dimensional convolution operation, f AW (·) denotes the unclosed function, round(.) denotes the rounding operation, and ||.|1 denotes the L1 norm;

[0039] The real-to-imaginary loss value satisfies the following formula:

[0040]

[0041] in, and These represent operations on extracting the real and imaginary parts of a complex number, respectively.

[0042] The Mel spectrum loss value satisfies the following formula:

[0043]

[0044] Among them, F mel The number of Melbands after compression of the frequency dimension. This represents the Mel spectrum of the audio prediction result at frequency index f and time frame index t. This represents the Mel spectrum of the tag information when the frequency index is f and the time frame index is t;

[0045] The consistency loss value satisfies the following formula:

[0046]

[0047] in, Indicates to Consistent estimates.

[0048] Secondly, this application provides an audio generation method, the method comprising:

[0049] Obtain Mel spectrum;

[0050] A Mel filter for obtaining the Mel spectrum;

[0051] The Mel spectrum and Mel filter are input into the vocoder model as provided in the first aspect to generate audio.

[0052] According to the above scheme, Mel spectrum samples of audio samples in different scenarios are generated based on Mel filters with different parameter configurations. When training the vocoder model, the Mel filters and Mel spectrum samples from different scenarios enable the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter parameter configuration, thus saving computational resources. During the training process, to ensure the vocoder model can adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain the audio prediction result. Based on the loss value between the audio prediction result and the audio sample, the parameters of the vocoder model are adjusted to obtain the trained vocoder. The vocoder trained through the above process can generate audio corresponding to Mel spectra in different scenarios, improving the audio generation efficiency in various scenarios.

[0053] Thirdly, this application provides a training device for a multi-scene vocoder based on value-zero decomposition, the device comprising:

[0054] The acquisition module is used to acquire multiple audio samples and the label information of each audio sample.

[0055] The first determining module is used to determine the Mel spectrum sample of each audio sample. The Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations.

[0056] The training module is used to input Mel spectrum samples and their corresponding Mel filters into the vocoder model to train the vocoder model and obtain the audio prediction result for each Mel spectrum sample. The vocoder model includes a value space model and a null space model. The vocoder model generates the first feature based on the Mel spectrum sample and its corresponding Mel filter using the value space model. The vocoder model then reconstructs the first feature in the linear amplitude domain and recovers the phase information using the null space model to obtain the audio prediction result.

[0057] The second determination module is used to determine the loss value between the audio prediction result and the audio sample;

[0058] The adjustment module is used to adjust the parameters of the vocoder model based on the loss value to obtain the trained vocoder model.

[0059] According to the above scheme, Mel spectrum samples of audio samples in different scenarios are generated based on Mel filters with different parameter configurations. When training the vocoder model, the Mel filters and Mel spectrum samples from different scenarios enable the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter parameter configuration, thus saving computational resources. During the training process, to ensure the vocoder model can adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, thereby transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain the audio prediction result. Based on the loss value between the audio prediction result and the audio sample, the parameters of the vocoder model are adjusted to obtain the trained vocoder.

[0060] In one possible implementation, the first determining module is used for:

[0061] Based on the pre-configured band number range and frequency range, generate Mel filters with different parameter configurations;

[0062] Based on the audio samples, determine the target Mel filter corresponding to the audio samples and the target parameter configuration corresponding to the target Mel filter;

[0063] Based on the target Mel filter and target parameter configuration, Mel spectrum samples of the audio samples are generated.

[0064] In this way, it is possible to generate Mel spectrum samples of audio samples in different scenarios.

[0065] In one possible implementation, the first feature satisfies the following formula:

[0066]

[0067] in, Indicates the first feature, A + Let A represent the pseudo-inverse transform matrix of the Mel filter. + The size is |X mel | indicates a Mel spectrum sample.

[0068] In this way, the Mel spectrum samples are transformed from the Mel domain to the linear domain, and the Mel samples of audio samples in different scenarios are used to train the vocoder model, so that the trained vocoder model can be adapted to audio in different scenarios.

[0069] In one possible implementation, the training module is used for:

[0070] The vocoder model uses a null space model to compress and encode the spectral features in the first feature in order from low frequency to high frequency, thus obtaining the encoded features;

[0071] The target audio is obtained by modeling the encoded features;

[0072] Amplitude and phase reconstruction are performed on the target audio to obtain the audio prediction result.

[0073] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0074] In one possible implementation, the audio prediction result Satisfy the following formula:

[0075]

[0076] in, This represents the magnitude estimate of the first feature. This represents the phase estimation of the first feature.

[0077] Let I represent the representation function of the null space model, I represent the identity matrix, and iSTFT(.) represent the inverse short-time Fourier transform.

[0078] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0079] In one possible implementation, the loss value includes at least one of the following:

[0080] Logarithmic amplitude loss, phase loss, real-to-imaginary part loss, Mel spectrum loss, and uniformity loss;

[0081] Among them, the logarithmic magnitude loss value Satisfy the following formula:

[0082]

[0083] in, X represents the value of the audio prediction result at frequency index f and time frame index t. f,t This represents the value of the tag information when the frequency index is f and the time frame index is t, where F and T represent the frequency dimension size and the number of time frames, respectively. log(.) represents the logarithm operation, and ||.|2 represents the L2 norm.

[0084] Phase loss value Satisfy the following formula:

[0085]

[0086] Where "*" represents a two-dimensional convolution operation, f AW (·) denotes the unclosed function, round(.) denotes the rounding operation, and ||.|1 denotes the L1 norm;

[0087] The real-to-imaginary loss value satisfies the following formula:

[0088]

[0089] in, and These represent operations on extracting the real and imaginary parts of a complex number, respectively.

[0090] The Mel spectrum loss value satisfies the following formula:

[0091]

[0092] Among them, F mel The number of Melbands after compression of the frequency dimension. This represents the Mel spectrum of the audio prediction result at frequency index f and time frame index t. This represents the Mel spectrum of the tag information when the frequency index is f and the time frame index is t;

[0093] The consistency loss value satisfies the following formula:

[0094]

[0095] in, Indicates to Consistent estimates.

[0096] Fourthly, this application provides an audio generation apparatus, the apparatus comprising:

[0097] The first acquisition module is used to acquire the Mel spectrum;

[0098] The second acquisition module is used to acquire the Mel filter of the Mel spectrum;

[0099] The generation module is used to input the Mel spectrum and Mel filter into the vocoder model as provided in the first aspect to generate audio.

[0100] Fifthly, this application provides a computing device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the method provided in the first aspect or any possible implementation of the first aspect, or implements the method provided in the second aspect.

[0101] Sixthly, this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect or any possible implementation of the first aspect, or to implement the method provided in the second aspect. Attached Figure Description

[0102] Figure 1 shows a flowchart illustrating a training method for a multi-scene vocoder based on value zero decomposition provided in an embodiment of this application;

[0103] Figure 2 shows a training flowchart of a vocoder model provided in an embodiment of this application;

[0104] Figure 3 shows a flowchart illustrating another training method for a multi-scene vocoder based on value zero decomposition provided in an embodiment of this application;

[0105] Figure 4 shows an exemplary flowchart of a phase loss function provided in an embodiment of this application;

[0106] Figure 5 shows a flowchart of an audio generation method provided in an embodiment of this application;

[0107] Figure 6 shows a schematic diagram of the structure of a training device for a multi-scene vocoder based on value zero decomposition according to an embodiment of this application;

[0108] Figure 7 shows a schematic diagram of an audio generation structure provided in an embodiment of this application;

[0109] Figure 8 shows a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0110] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0111] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0112] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0114] Vocoders are used in tasks such as speech synthesis, audio generation, music generation, and speech enhancement. In these tasks, when a user provides a text description, a generative model estimates the corresponding Mel spectrogram of the audio. In speech enhancement tasks, the Mel spectrogram of noisy speech is extracted, and then a clean speech Mel spectrogram is recovered using a speech enhancement algorithm. The vocoder is responsible for converting this to the corresponding time-domain waveform. Although existing deep learning-based neural vocoders can effectively achieve the Mel spectrogram to waveform conversion...

[0115] However, current vocoders, after training, can only adapt to one Mel spectrum configuration. When generating audio for multiple scenarios, vocoders for different scenarios need to be trained to adapt to the Mel spectrum configurations for each scenario. This requires a large amount of computing resources, resulting in significant computational resource consumption.

[0116] Based on this, this application provides a training method and apparatus for a multi-scene vocoder based on value-zero decomposition. It generates Mel spectrum samples of audio samples in different scenarios using Mel filters with different parameter configurations. During vocoder model training, the use of Mel filters and Mel spectrum samples from different scenarios allows the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter configuration and saving computational resources. In the process of training the vocoder model, to enable it to adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, thereby transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain audio prediction results. Based on the loss value between the audio prediction results and the audio samples, the parameters of the vocoder model are adjusted to obtain the trained vocoder.

[0117] Figure 1 illustrates a training method for a multi-scene vocoder based on value-zero decomposition provided in this application embodiment. This method can be applied to devices or platforms with computing and storage capabilities, such as computing devices (e.g., servers) or computing device clusters (e.g., cloud computing platforms). As shown in Figure 1, the training method for a multi-scene vocoder based on value-zero decomposition provided in this application embodiment includes the following steps S101 to S106.

[0118] S101, acquire multiple audio samples.

[0119] Audio samples can be generated in different scenarios.

[0120] In some embodiments, S101 includes the following process: first, generating Mel filters with different parameter configurations based on pre-configured band number intervals and frequency intervals; then, determining the target Mel filter corresponding to the audio sample and the target parameter configuration corresponding to the target Mel filter based on the audio sample; finally, generating Mel spectrum samples of the audio sample based on the target Mel filter and the target parameter configuration. This enables the generation of Mel spectrum samples of audio samples in different scenarios.

[0121] For example, as shown in Figure 2, in S101, by selecting the range of Mel-band number and the highest calculated frequency, sampling configuration is performed based on the Mel-band number and the highest calculated frequency range to obtain multiple Mel filter matrices. These multiple Mel filter matrices are then stored, thus enabling the acquisition of Mel filters with various parameter configurations. For example, assuming the Mel filter's band number range is... in and These refer to the minimum and maximum values ​​of the range, respectively, with the highest calculation frequency range being... in and These represent its minimum and maximum values, and the lowest calculated frequency value f, respectively. min It is usually set to 0. When setting the interval, the interval of the number of bands is set to δ. n The interval of the highest frequency is set to δ. f In this case, we can obtain M = M n M f The parameter configuration of the Mel filter, where M n With M f Let represent the maximum number of samples in the band number and highest frequency range, respectively, and their calculation method satisfies the following formula (1):

[0122]

[0123] in, This indicates the floor function. For example, and Set to 64 and 128 respectively, for sampling rates f s =24kHz audio generation scenario, will and Setting the frequencies to 8kHz and 12kHz respectively, practical experience has shown that this setting is sufficient to cover the vast majority of real-world application scenarios. n With δ f The Hz frequencies are set to 4 and 100Hz respectively. For M parameter configurations, the Mel filter matrix corresponding to each configuration is calculated and stored. The parameter configurations of different Mel filter matrices are different. For example, the set of Mel filter matrices is {A1,…,A1}. M}, then the parameter configuration matrix set corresponding to this matrix set is {C1,…,C}. M}, where C m With A m These represent the m-th parameter configuration and the corresponding Mel filter matrix, where m is a natural number, m = 1, 2, ..., M.

[0124] S102, determine the Mel spectrum sample for each audio sample. The Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations. Mel filters with different parameter configurations are adapted to different scenarios.

[0125] In this embodiment of the application, for a certain audio sample x(n), it is transformed to the time-frequency domain by Short-Time Fourier Transform (STFT), and the time-frequency domain representation of the audio sample x(n) is as follows: Where F and T represent the magnitudes in the frequency and time dimensions, respectively. For parameter configuration C... m The Mel filter, after the audio sample x(n) is compressed by the Mel filter, the Mel spectrum sample satisfies the following formula (2):

[0126] |X mel |=A|X| (2)

[0127] in, and These represent the Mel spectrum and amplitude spectrum, respectively. For simplicity, the configuration subscript (.) is omitted here. m F mel Let be the number of Mel bands after compression of the frequency dimension. For example, as shown in Figure 2, during model training, after acquiring audio samples, Mel filters are sampled from the stored Mel filter matrix, and the audio samples are processed based on the Mel filters to obtain Mel spectrum samples.

[0128] For example, the configuration of the Short-Time Fourier Transform (SFT) is as follows: a Hanning window is used as the window function, the window length for each frame is set to 1024, the number of frame shift points is 256, the number of Fourier Transform (FFT) points is 1024, and based on conjugate symmetry, the feature size F in the frequency dimension is 513. In network training, the Mel spectrum samples |X mel | The Mel filter A is used as the input to the network, while x(n) is used as the training target, that is, x(n) is the label information of the Mel spectrum sample.

[0129] S103, input the Mel spectrum samples and the corresponding Mel filters into the vocoder model, train the vocoder model, and obtain the audio prediction result corresponding to each Mel spectrum sample. The vocoder model includes a value space model and a null space model. The vocoder model generates the first feature based on the Mel spectrum samples and the corresponding Mel filters through the value space model. The vocoder model reconstructs the first feature in the linear amplitude domain and recovers the phase information through the null space model to obtain the audio prediction result.

[0130] The vocoder model uses a null-space model to compress and encode the spectral features in the first feature, following a low-frequency to high-frequency order, to obtain the encoded features. The encoded features are then modeled to obtain the target audio. Finally, amplitude and phase reconstructions are performed on the target audio to obtain the audio prediction result. In this way, by using a null-space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction result.

[0131] In some embodiments, the vocoder model is built based on a subband structured neural network (RNDVoC).

[0132] In this embodiment, as shown in Figure 3, the vocoder model comprises two parts: a rangespace model and a null space model. During training, the rangespace model aims to transform the input Mel spectrum from the Mel domain to the linear domain, achieved using the inverse transform of the Mel filter A. Since its inverse transform is difficult to perfectly reconstruct, a pseudo-inverse form is used in practice, i.e., A... + For example, the pseudo-inverse that can be used is the Moore-Penrose pseudo-inverse. Therefore, the operation process of the value space model can be expressed as the following formula (3):

[0133]

[0134] in, The output of the value space model, i.e., the first feature, is the pseudo-inverse transformation matrix A. + The size is |X mel | indicates a Mel spectrum sample.

[0135] In this way, the Mel spectrum samples are transformed from the Mel domain to the linear domain, and the Mel samples of audio samples in different scenarios are used to train the vocoder model, so that the trained vocoder model can be adapted to audio in different scenarios.

[0136] The null space model aims to reconstruct the missing spectral details in the linear domain. It is worth noting that since the Mel spectrum loses phase information, the null space model also needs to recover the phase information. The recovery process of the phase information satisfies the following formula (4):

[0137]

[0138] in, For the amplitude estimation of the first feature, This represents the phase estimation of the first feature.

[0139] This is the representation function of the null space model. Following the value-zero decomposition theory, the audio prediction results... The following formula (5) must be satisfied:

[0140]

[0141] As shown in Figure 3, the null space model mainly consists of three parts: a subband coding module, a subband decoding module, and a dual-path modeling module. The subband coding module utilizes the prior characteristic that the information content of the spectrum differs in different frequency ranges, compressing and encoding the spectral features from low to high frequency, and from fine to coarse. Assuming the number of compressed subbands is N, for a subband index n, its amplitude in the linear domain is... Where F n If the frequency of this sub-band is given, then the feature encoding process of this interval spectrum satisfies the following formula (6):

[0142]

[0143] Where LN(.) represents Layer Normalization, and Conv1d(.) represents one-dimensional convolution. This involves encoding N features... By splicing, a three-dimensional representation is obtained. Where C represents the channel dimension size after encoding. At a sampling rate of 24kHz, the frequency range of spectral compression is divided as follows: the 0-3kHz range is merged into a sub-band every 250Hz, the 3-7kHz range is merged into a sub-band every 500Hz, the 7-10kHz range is merged into a sub-band every 1kHz, and the range above 10kHz is merged into a sub-band. Therefore, N = 12 + 8 + 3 + 1 = 24.

[0144] After passing through the sub-band encoding module, the output F will be... in The target information is modeled using a dual-path modeling module. This module consists of B dual-path blocks, each containing one cross-subband module and P subband modules. For the cross-subband module, given input features of size N×C×T, it is first transposed. Then, a grouped one-dimensional convolution with kernel size J and number of groups H is used to model along the subband dimension. Next, a one-dimensional point convolution compresses the number of channels to one-quarter of the original. Linear layers are then used to model the global subband features on the compressed features, and a one-dimensional point convolution restores the number of channels. Finally, a grouped one-dimensional convolution with kernel size J and number of groups H is used again to model along the subband dimension. For the subband module, the transposition operation is performed first, followed by a convolution with kernel size J...′ The network is modeled along the time axis using a one-dimensional convolutional depthwise convolution, followed by a one-dimensional pointwise convolution and a global response normalization layer, and finally output through another one-dimensional pointwise convolution. The specific hyperparameters and settings of the network are described in Table 1, with C=256, B=6, P=2, J=3, H=8, and J... ′ =7, with a parameter count of 9.48M and a complexity of 24.98GMACs / 5s.

[0145] Table 1

[0146]

[0147] After passing through the dual-path modeling module, its output is denoted as This information is then fed into the subband decoding module. The subband decoding module involves two branches, one for reconstructing amplitude information and the other for reconstructing phase information. Taking the amplitude branch as an example, given input features O, for features at subband index n... The amplitude of the subband is then recovered by passing it through the decoding layer and then by the exponential operation exp(.). Finally, N amplitudes are concatenated along the frequency dimension to estimate the amplitude spectrum. This process satisfies the following formula (7):

[0148]

[0149] Here, GELU(.) represents the Gaussian Error Linear Unit, and Concat(.) represents the concatenation operation. In the phase branch, a decoding operation similar to that in the amplitude branch is used, the difference being that the phase is estimated at the end using the Atan2(.) operation.

[0150] During training, the network had a batch size of 16, the generator iterated for a total of one million steps, and the AdamW optimizer was used for training. The values ​​of {β1,β2} were set to 0.8 and 0.99, respectively. The learning rate was initialized to 0.0002 and an exponential decay strategy was used to adjust the learning rate, with a decay rate of 0.999.

[0151] S104, determine the loss value between the audio prediction result and the audio sample.

[0152] The loss function determines the difference between the audio prediction result and the audio sample, i.e., the loss value. To ensure the trained vocoder model can accurately generate audio for different scenarios, the loss function in this embodiment includes two parts: a reconstruction loss function and an adversarial loss function. The reconstruction loss function includes a logarithmic magnitude loss function, a phase loss function, a real-to-imaginary part loss function, a Mel loss function, and a consistency loss function.

[0153] The logarithmic magnitude loss function is the mean-square error (MSE) between the logarithmic estimated magnitude and the magnitude spectrum of the dialogue target. The logarithmic magnitude loss function satisfies the following formula (8):

[0154]

[0155] in, X represents the value of the audio prediction result at frequency index f and time frame index t. f,t This represents the value of the tag information when the frequency index is f and the time frame index is t, where F and T represent the frequency dimension size and the number of time frames, respectively. log(.) represents the logarithm operation, and ||.|2 represents the L2 norm.

[0156] The phase loss function can be an omnidirectional phase loss function, as shown in Figure 4. A design is provided. A fixed-parameter convolution kernel is used to calculate the difference between a given phase and its eight neighboring phases in the phase spectrum. The fifth 3×3 convolution of the kernel calculates the instantaneous phase. Given an estimated phase... Phase Φ with the target, through Convolve them respectively to obtain the omnidirectional phase difference after convolution. and The loss is then calculated using an anti-wrapping function. This loss function satisfies the following formula (9):

[0157]

[0158] Where "*" represents a two-dimensional convolution operation, and the unclosed function f AW (x) is defined as Here, round(.) represents the rounding operation, and ||.|1 represents the L1 norm.

[0159] The real-imaginary part loss function is the mean absolute error (MAE) between the estimated real-imaginary part and the target real-imaginary step. The real-imaginary part loss function satisfies the following formula (10):

[0160]

[0161] in, and These represent operations on extracting the real and imaginary parts of a complex number, respectively.

[0162] The Mel loss function is the average absolute error between the estimated Mel spectrum and the target Mel spectrum, and the Mel loss function satisfies the following formula (11):

[0163]

[0164] The consistency loss function satisfies the following formula (12):

[0165]

[0166] in, for Consistency estimation. The reconstruction loss function satisfies the following formula (13):

[0167]

[0168] Where, {λ a ,λ p ,λ ri ,λ mel ,λ c} represents the weighting coefficients for each loss term. For example, these weighting coefficients can be set to {45, 100, 45, 45, 45}. The specific weighting coefficients can be set according to the actual situation, and this application embodiment does not impose specific limitations.

[0169] For the adversarial loss function, two types of discriminators are used: a multi-period discriminator (MPD) and a multi-resolution spectrogram discriminator (MRSD). The MPD consists of five sub-discriminators with period values ​​set to {2, 3, 5, 7, 11}. The MRSD consists of three sub-discriminators, where the window length, frame shift, and Fourier transform coefficients are set to {(512, 128, 512), (1024, 256, 1024), (2048, 512, 2048)}. In this embodiment, Hinge is used as the adversarial loss, and the adversarial loss function of the discriminator part satisfies the following formula (14):

[0170]

[0171] Among them, D m (s) and Let M represent the outputs of the target and estimated audio in the sub-discriminators, respectively, where M is the number of sub-discriminators. For the generator, its adversarial loss function satisfies the following formula (15):

[0172]

[0173] In addition, a feature matching loss function is incorporated, which satisfies the following formula (16):

[0174]

[0175] in, Let E represent the output feature of the e-th layer of the m-th sub-discriminator, and E represent the total number of layers in the sub-discriminators. In this embodiment, for and Weighting coefficients {λ D ,λ g ,λ fm Set it to 1.

[0176] S105. Adjust the parameters of the vocoder model based on the loss value to obtain the trained vocoder model.

[0177] According to embodiments of this application, Mel spectrum samples of audio samples in different scenarios are generated based on Mel filters with different parameter configurations. When training the vocoder model, the Mel filters and Mel spectrum samples from different scenarios enable the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter parameter configuration, thus saving computational resources. During the training process, to enable the vocoder model to adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, thereby transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain the audio prediction result. Based on the loss value between the audio prediction result and the audio sample, the parameters of the vocoder model are adjusted to obtain the trained vocoder.

[0178] Figure 5 is a flowchart illustrating the audio generation method provided in an embodiment of this application. As shown in Figure 5, the audio generation method provided in an embodiment of this application includes the following steps S501 to S503.

[0179] S501, obtain Mel spectrum.

[0180] The user can provide a Mel spectrum.

[0181] S502, Obtain the Mel filter of the Mel spectrum.

[0182] Based on the Mel spectrum, the corresponding Mel filter can be obtained. For example, based on the Mel spectrum, the Mel filter matrix corresponding to the Mel spectrum is matched from the set of Mel spectrum filter matrices generated in the embodiment corresponding to S101, i.e., the Mel spectrum filter. If the set of Mel spectrum filter matrices does not include the Mel spectrum filter matrix corresponding to the Mel spectrum, then the Mel filter matrix of the Mel spectrum is generated.

[0183] S503, input the Mel spectrum and the Mel filter into the vocoder model to generate audio.

[0184] The Mel spectrum and Mel filter are input into the vocoder model to generate audio. This vocoder model is trained using the embodiment shown in Figure 1. Based on this vocoder model, audio for different scenarios can be generated.

[0185] Based on the same concept as the method embodiments of this application, this application also provides a training device for a multi-scene vocoder based on value zero decomposition. The training device for a multi-scene vocoder based on value zero decomposition includes several modules, each module being used to execute various steps in the training method for a multi-scene vocoder based on value zero decomposition provided in this application. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the training method for a multi-scene vocoder based on value zero decomposition provided in this application can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0186] Figure 6 illustrates a training apparatus for a multi-scene vocoder based on value-zero decomposition provided in an embodiment of this application. As shown in Figure 6, the training apparatus for a multi-scene vocoder based on value-zero decomposition provided in this embodiment includes:

[0187] The acquisition module 601 is used to acquire multiple audio samples and the label information of each audio sample among the multiple audio samples;

[0188] The first determining module 602 is used to determine the Mel spectrum sample of each audio sample. The Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations.

[0189] The training module 603 is used to input the Mel spectrum samples and the corresponding Mel filters into the vocoder model to train the vocoder model and obtain the audio prediction result corresponding to each Mel spectrum sample. The vocoder model includes a value space model and a null space model. The vocoder model generates a first feature based on the Mel spectrum samples and the corresponding Mel filters through the value space model. The vocoder model reconstructs the first feature in the linear amplitude domain and recovers the phase information through the null space model to obtain the audio prediction result.

[0190] The second determining module 604 is used to determine the loss value between the audio prediction result and the audio sample;

[0191] The adjustment module 605 is used to adjust the parameters of the vocoder model according to the loss value to obtain the trained vocoder model.

[0192] According to the above scheme, Mel spectrum samples of audio samples in different scenarios are generated based on Mel filters with different parameter configurations. When training the vocoder model, the Mel filters and Mel spectrum samples from different scenarios enable the trained vocoder model to adapt to different parameter configurations of the Mel filters, eliminating the need to train a separate vocoder model for each Mel filter parameter configuration, thus saving computational resources. During the training process, to ensure the vocoder model can adapt to Mel filters in different scenarios (i.e., different parameter configurations), a value space model is used to process the Mel spectrum samples and Mel filters, thereby transforming the Mel spectrum samples from the Mel domain to the linear domain. This allows the null space model to continue subsequent linear amplitude domain reconstruction and phase information recovery to obtain the audio prediction result. Based on the loss value between the audio prediction result and the audio sample, the parameters of the vocoder model are adjusted to obtain the trained vocoder.

[0193] In one possible implementation, the first determining module is used for:

[0194] Based on the pre-configured band number range and frequency range, generate Mel filters with different parameter configurations;

[0195] Based on the audio samples, determine the target Mel filter corresponding to the audio samples and the target parameter configuration corresponding to the target Mel filter;

[0196] Based on the target Mel filter and target parameter configuration, Mel spectrum samples of the audio samples are generated.

[0197] In this way, it is possible to generate Mel spectrum samples of audio samples in different scenarios.

[0198] In one possible implementation, the first feature satisfies the following formula:

[0199]

[0200] in, Indicates the first feature, A + Let A represent the pseudo-inverse transform matrix of the Mel filter. + The size is |X mel | indicates a Mel spectrum sample.

[0201] In this way, the Mel spectrum samples are transformed from the Mel domain to the linear domain, and the Mel samples of audio samples in different scenarios are used to train the vocoder model, so that the trained vocoder model can be adapted to audio in different scenarios.

[0202] In one possible implementation, the training module is used for:

[0203] The vocoder model uses a null space model to compress and encode the spectral features in the first feature in order from low frequency to high frequency, thus obtaining the encoded features;

[0204] The target audio is obtained by modeling the encoded features;

[0205] Amplitude and phase reconstruction are performed on the target audio to obtain the audio prediction result.

[0206] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0207] In one possible implementation, the audio prediction result Satisfy the following formula:

[0208]

[0209] in, This represents the magnitude estimate of the first feature. This represents the phase estimation of the first feature.

[0210] Let I represent the representation function of the null space model, I represent the identity matrix, and iSTFT(.) represent the inverse short-time Fourier transform.

[0211] Thus, by using the null space model to reconstruct the amplitude and phase of the target audio, the missing spectral details and phase information of the Mel spectrum sample can be recovered, improving the accuracy of the audio prediction results.

[0212] In one possible implementation, the loss value includes at least one of the following:

[0213] Logarithmic amplitude loss, phase loss, real-to-imaginary part loss, Mel spectrum loss, and uniformity loss;

[0214] Among them, the logarithmic magnitude loss value Satisfy the following formula:

[0215]

[0216] in, X represents the value of the audio prediction result at frequency index f and time frame index t. f,t This represents the value of the tag information when the frequency index is f and the time frame index is t, where F and T represent the frequency dimension size and the number of time frames, respectively. log(.) represents the logarithm operation, and ||.|2 represents the L2 norm.

[0217] Phase loss value Satisfy the following formula:

[0218]

[0219] Where "*" represents a two-dimensional convolution operation, f AW (·) denotes the unclosed function, round(.) denotes the rounding operation, and ||.|1 denotes the L1 norm;

[0220] The real-to-imaginary loss value satisfies the following formula:

[0221]

[0222] in, and These represent operations on extracting the real and imaginary parts of a complex number, respectively.

[0223] The Mel spectrum loss value satisfies the following formula:

[0224]

[0225] Among them, F mel The number of Melbands after compression of the frequency dimension. This represents the Mel spectrum of the audio prediction result at frequency index f and time frame index t. This represents the Mel spectrum of the tag information when the frequency index is f and the time frame index is t;

[0226] The consistency loss value satisfies the following formula:

[0227]

[0228] in, Indicates to Consistent estimates.

[0229] Based on the same concept as the method embodiments of this application, this application also provides an audio generation apparatus. The audio generation apparatus includes several modules, each module being used to execute various steps in the audio generation method provided in this application. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the audio generation method provided in this application can be assigned to different modules as needed, that is, the internal structure of the apparatus can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above apparatus can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0230] Figure 7 illustrates a training apparatus for a multi-scene vocoder based on value-zero decomposition provided in an embodiment of this application. As shown in Figure 7, the training apparatus for a multi-scene vocoder based on value-zero decomposition provided in this embodiment includes:

[0231] The first acquisition module 701 is used to acquire the Mel spectrum;

[0232] The second acquisition module 702 is used to acquire the Mel filter of the Mel spectrum;

[0233] The generation module 703 is used to input the Mel spectrum and Mel filter into the vocoder model provided in any of the above embodiments to generate audio.

[0234] The following describes a computing device provided by an embodiment of this application.

[0235] Figure 8 is a schematic diagram of a computer device provided in an embodiment of this application. As shown in Figure 8, the computer device provided in this embodiment of the application can be used to implement the Tibetan multilingual speech recognition and speech recognition joint model training method or speech recognition method described in the above method embodiments.

[0236] The computer device may include a processor 801 and a memory 802 storing computer program instructions.

[0237] Specifically, the processor 801 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0238] Memory 802 may include mass storage for data or instructions. For example, and not limitingly, memory 802 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 802 may include removable or non-removable (or fixed) media. Where appropriate, memory 802 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 802 is non-volatile solid-state memory.

[0239] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to this application.

[0240] The processor 801 reads and executes computer program instructions stored in the memory 802 to implement any of the Tibetan multilingual speech recognition and speech recognition joint model training methods or speech recognition methods in the above embodiments.

[0241] In one example, the electronic device may also include a communication interface 803 and a bus 810. As shown in Figure 8, the processor 801, memory 802, and communication interface 803 are connected via the bus 810 and communicate with each other.

[0242] The communication interface 803 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0243] Bus 810 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 810 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0244] Furthermore, in conjunction with the above embodiments, this application embodiment can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the methods described in the above embodiments.

[0245] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0246] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0247] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0248] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A training method for a multi-scene vocoder based on value-zero decomposition, characterized in that, The method includes: acquiring multiple audio samples; determining the Mel spectrum sample for each audio sample, wherein the Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations, and the Mel filters with different parameter configurations are adapted to different scenarios; inputting the Mel spectrum sample and the corresponding Mel filter into a vocoder model, training the vocoder model, and obtaining an audio prediction result corresponding to each Mel spectrum sample, wherein the vocoder model includes a value space model and a null space model, the vocoder model generates a first feature through the value space model based on the Mel spectrum sample and the corresponding Mel filter, and the vocoder model reconstructs the first feature in the linear amplitude domain and recovers the phase information through the null space model to obtain the audio prediction result; determining the loss value between the audio prediction result and the audio sample; and adjusting the parameters of the vocoder model according to the loss value to obtain the trained vocoder model.

2. The method according to claim 1, characterized in that, The step of determining the Mel spectrum sample of each audio sample includes: generating Mel filters with different parameter configurations based on pre-configured band number intervals and frequency intervals; determining the target Mel filter corresponding to the audio sample and the target parameter configuration corresponding to the target Mel filter based on the audio sample; and generating the Mel spectrum sample of the audio sample based on the target Mel filter and the target parameter configuration.

3. The method according to claim 1, characterized in that, The first feature satisfies the following formula: in, Indicates the first feature, This represents the pseudo-inverse transform matrix of the Mel filter. The size is ， This represents a Mel spectrum sample.

4. The method according to claim 1, characterized in that, The vocoder model reconstructs the linear amplitude domain and recovers the phase information of the first feature using the null space model to obtain the audio prediction result, including: the vocoder model compresses and encodes the spectral features in the first feature in order from low frequency to high frequency using the null space model to obtain coded features; models the coded features to obtain the target audio; and reconstructs the amplitude and phase of the target audio to obtain the audio prediction result.

5. The method according to claim 4, characterized in that, The audio prediction results Satisfy the following formula: in, This represents the magnitude estimate of the first feature. This represents the phase estimate of the first feature. The representation function of the null space model. Represents the identity matrix.

6. The method according to any one of claims 1-4, characterized in that, The loss value includes at least one of the following: logarithmic amplitude loss value, phase loss value, real-to-imaginary part loss value, Mel spectrum loss value, and uniformity loss value; wherein, the logarithmic amplitude loss value Satisfy the following formula: in, This indicates that the audio prediction result is at a frequency index of With time frame index The value at time, This indicates that the tag information is in the frequency index. With time frame index The value at time, and These represent the frequency dimension size and the number of time frames, respectively. This represents the logarithmic operation. express Norm; the phase loss value Satisfy the following formula: in," " indicates a two-dimensional convolution operation. This represents a convolution kernel with fixed parameters. Indicates the target phase. The unclosed function is defined as ， This indicates the rounding operation. express Norm; the real-to-imaginary loss value satisfies the following formula: in, and These represent the operations of taking the real and imaginary parts of a complex number, respectively; the Mel spectrum loss value satisfies the following formula: in, The number of Melbands after compression of the frequency dimension. This indicates that the audio prediction result is at a frequency index of With time frame index Mel's score at that time This indicates that the tag information is in the frequency index. With time frame index The Mel spectrum at that time; the consistency loss value satisfies the following formula: in, Indicates to Consistent estimates.

7. An audio generation method, characterized in that, The method includes: acquiring a Mel spectrum; acquiring a Mel filter for the Mel spectrum; and inputting the Mel spectrum and the Mel filter into a vocoder model as described in any one of claims 1-6 to generate audio.

8. A training device for a multi-scene vocoder based on value-zero decomposition, characterized in that, The device includes: an acquisition module for acquiring multiple audio samples and label information for each audio sample; a first determination module for determining the Mel spectrum sample of each audio sample, wherein the Mel spectrum samples of different audio samples are generated by Mel filters with different parameter configurations, and the Mel filters with different parameter configurations are adapted to different scenarios; a training module for inputting the Mel spectrum sample and the Mel filter corresponding to the Mel spectrum sample into a vocoder model to train the vocoder model and obtain an audio prediction result corresponding to each Mel spectrum sample, wherein the vocoder model includes a value space model and a null space model, the vocoder model generates a first feature through the value space model based on the Mel spectrum sample and the Mel filter corresponding to the Mel spectrum sample, and the vocoder model reconstructs the first feature in the linear amplitude domain and recovers the phase information through the null space model to obtain the audio prediction result; a second determination module for determining the loss value between the audio prediction result and the audio sample; and an adjustment module for adjusting the parameters of the vocoder model according to the loss value to obtain a trained vocoder model.

9. An audio generation device, characterized in that, The apparatus includes: a first acquisition module for acquiring a Mel spectrum; a second acquisition module for acquiring a Mel filter of the Mel spectrum; and a generation module for inputting the Mel spectrum and the Mel filter into a vocoder model as described in any one of claims 1-6 to generate audio.

10. A computing device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-6, or implements the method as described in claim 7.

Images