A speech synthesis method, a vocoder, and its training and optimization method

Abstract

This application provides a speech synthesis method, a vocoder, and its training and optimization method. The vocoder uses a ConvNext neural network as its backbone network. ConvNext replaces transposed convolution with channel upsampling, avoiding the periodic noise of transposed convolution. It employs depthwise convolution and separable convolution, which not only captures the time-frequency information of the vocoder input but also significantly reduces the computational load of the backbone network. Using frequency band fusion technology, the vocoder can simultaneously output audio from different frequency components of the synthesized audio, thereby reducing the output sampling rate and improving the vocoder's operating efficiency. Specifically, the vocoder uses multi-channel output, with each channel corresponding to an audio signal within a specific frequency band. The frequency ranges of the signals from each channel are orthogonal to each other, and the sampling rate is 1 / M of the original (M is the number of frequency bands). Finally, the audio signals from each frequency band are anti-mirror filtered and superimposed to obtain the audio at the original sampling rate.

Description

Technical Field

[0001] This invention relates to speech synthesis technology, and more particularly to a speech synthesis method, a vocoder, and a training and optimization method thereof. Background Technology

[0002] With the development of neural networks, speech synthesis technology has advanced rapidly. Most speech synthesis modules employ a two-stage process. The first stage involves using acoustic models to create a low-resolution acoustic representation of the text, such as mel features. The second stage uses a vocoder to reconstruct the audio from the intermediate representation. In other words, the vocoder process is the inverse of extracting mel features. The vocoder must solve two inverse problems: recovering the phase and amplitude of the original waveform, and understanding the frequency and time domain transformations required for upsampling the intermediate representation.

[0003] Because vocoders need to upsample mel features to the audio sampling rate (typically 16k+ samples per second), they are the bottleneck in the entire synthesis system's computational efficiency. To address this issue, researchers have made significant optimizations to the computational efficiency of vocoders. Currently, Generative Adversarial Network (GAN) vocoders with fully convolutional networks as their backbone are mainstream, with Hifi-Gan as a typical example. Hifi-Gan transforms mel features into audio waveforms in a non-autoregressive manner, greatly improving synthesis efficiency and sound quality. However, Hifi-Gan cannot effectively utilize the time-frequency information in mel features. The subsequent iSTFTNet vocoder improved the upsampling layer of Hifi-Gan to Inverse Short-Time Fourier Transform (iSTFT), enabling a faster and lighter model while maintaining audio quality. Subsequently, researchers improved iSTFTNet into Multi-Band iSTFTNet (MB-iSTFTNet) by adopting a multi-subband parallel generation strategy. This allows for the parallel output of multiple sub-band components, further reducing computational load and accelerating synthesis.

[0004] MB-iSTFTNet, as the most efficient vocoder currently available, still has the following two problems:

[0005] 1. The use of transposed convolutional structures in the backbone network to upsample the time channels can lead to frequency domain aliasing and periodic noise.

[0006] 2. The backbone network uses a multi-branch structure, which requires calculating and storing the output values ​​of each branch during model inference, resulting in low computational efficiency.

[0007] 3. The reconstruction of the inverse short-time Fourier transform requires a certain window overlap (generally 50% overlap rate) to ensure a smooth transition between adjacent frames. In other words, the number of sampling points generated by the neural network is greater than the number of sampling points in the recovered waveform. These extra sampling points are used for inter-frame smoothing. These redundant samples are all generated by the neural network, which to some extent wastes computational resources and is detrimental to the inference efficiency of the vocoder. Summary of the Invention

[0008] The purpose of this invention is to provide a speech synthesis method, vocoder, and training and optimization method that do not use transposed convolution and critical sampling.

[0009] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0010] According to a first aspect of the present invention, a speech synthesis method is provided, comprising the following steps:

[0011] Step S101: Input feature fea∈R into the vocoder T×D The feature fea is a low-resolution intermediate representation of the acoustic model from the text model, where T is the number of frames of the feature, D is the feature dimension, and H is the frame shift corresponding to the feature fea.

[0012] Step S102: Upsample the input feature fea using the vocoder backbone network to obtain the frequency domain representation y of each sub-band of the audio to be synthesized. i ∈C T×H′ , where i≤M is the sub-band index, M is the number of frequency bands, and H′ is the dimension of the frequency domain representation;

[0013] Step S103: The vocoder outputs the MDCT representation of the audio in each sub-band. As their respective frequency domain representations, the MDCT representations of each sub-band The frame shift is

[0014] Step S104: MDCT characterization of each sub-band in step S103. Perform inverse modified discrete cosine transform (IMDCT) to recover the audio corresponding to each sub-channel.

[0015] Step S106, for Upsample by M times to obtain

[0016]

[0017] Where upsample represents the upsampling operation;

[0018] Step S107: Use anti-mirror filter banks to filter the signals in each sub-band. Perform anti-mirror filtering; the output of each sub-band after anti-mirror filtering is

[0019]

[0020] Operators f represents the convolution operation. i For anti-mirror filter banks;

[0021] Step S108, convert each output channel of the anti-mirror filter bank in step S107 to... Add them together to get the output speech.

[0022]

[0023] In one embodiment, in step 102, the frequency domains of each sub-band are orthogonal to each other, and the audio sampling rate is 1 / 3 of the original sampling rate.

[0024] In one embodiment, in step 103, the sub-band frequency domain characterization The dimension H′ is equal to the frame shift, that is... The corresponding window length is

[0025] In one embodiment, in step 107, the impulse response of the anti-mirror filter bank is:

[0026]

[0027] Where N is the number of taps in the impulse response, M is the number of subbands in the filter bank, and the impulse response of the prototype filter h(n) is:

[0028]

[0029] Where cutoff is the cutoff frequency ratio.

[0030] According to a second aspect of the present invention, a vocoder is provided, comprising a backbone network and a sub-band fusion module. The backbone network is a ConvNext neural network model, its input being an acoustic representation, and its output being an MDCT representation of the audio components of each sub-band. The frequency domains of the sub-bands output by the backbone network are mutually orthogonal, and the audio sampling rate is 1 / 3 of the original sampling rate. M represents the number of frequency bands;

[0031] The sub-band fusion module includes an IMDCT inverse transform module, an upsampling module, and an anti-mirror filtering module, which are used to perform IMDCT inverse transform on the MDCT representation of the audio components of each sub-band, upsample the bandpass signal of each channel, and perform anti-mirror filtering, respectively.

[0032] In one embodiment, the backbone network is composed of multiple stacked ConvNextBlocks, each of which consists of a deep convolutional layer with a large convolutional kernel and a separable convolutional layer, wherein the deep convolutional layer is used to capture temporal information and the separable convolutional layer is used to capture frequency information.

[0033] According to a third aspect of the present invention, a training optimization method for the above-described vocoder is provided, comprising:

[0034] The acoustic representation extracted from the original audio x is input into the vocoder, and the vocoder outputs the reconstructed audio. Compare the reconstructed audio The differences from the original audio x are used to make them as consistent as possible;

[0035] Calculate the components of each sub-band of the original audio, and compare the components of each sub-band of the original audio with the audio of each sub-band reconstructed by the vocoder. The differences should be addressed to make the two as consistent as possible.

[0036] The reconstructed audio and the original audio are fed into the decision processor to make the output values ​​of the audio generated by the vocoder and the original audio as consistent as possible in each intermediate layer of the decision processor.

[0037] In one embodiment, the loss function is:

[0038] L = L target1 +L target2 +L adv

[0039] Among them, L target1 Audio generated for vocoder model MAE with GT Audio x;

[0040] L target2 For each sub-band of GT audio and the corresponding sub-band generated by the vocoder MAE;

[0041] L adv The cost of the decision-making device.

[0042] In one embodiment, L target1 The calculation formula is:

[0043]

[0044] Wherein, STFT represents Short Time Fourier Transform, x represents the audio reconstructed by the vocoder, and x represents the Ground Truth audio.

[0045] In one embodiment, L target2 The calculation process is as follows:

[0046] Step S204, set the Ground Truth audio x∈R T*H Anti-aliasing filtering is performed to obtain the audio of each sub-band.

[0047]

[0048] Where M is the number of sub-bands, f i ′ represents the anti-aliasing filter bank:

[0049]

[0050] Where N is the number of taps in the impulse response, M is the number of subbands in the filter bank, and the impulse response of the prototype filter h(n) is:

[0051]

[0052] Where cutoff is the cutoff frequency ratio;

[0053] Step S205, for Perform downsampling:

[0054]

[0055] Where downsample represents the... Perform downsampling;

[0056] L target2 The calculation formula is:

[0057]

[0058] The beneficial effects of the embodiments of the present invention are:

[0059] 1. ConvNext is used as the vocoder backbone. ConvNext replaces transposed convolution with convolution channel upsampling, avoiding the periodic noise of transposed convolution; it uses depthwise convolution and separable convolution, which not only captures the time-frequency information of the vocoder input, but also greatly reduces the computational load of the backbone network.

[0060] 2. By employing frequency band fusion technology, the vocoder can simultaneously output audio components of different frequencies in the synthesized audio, thereby reducing the output sampling rate and improving the vocoder's operating efficiency. Specifically, the vocoder uses multi-channel output, with each channel corresponding to an audio signal within a specific frequency band. The frequency ranges of the signals from each channel are orthogonal to each other, and the sampling rate is 1 / M of the original (where M is the number of frequency bands). Finally, the audio signals from each frequency band are anti-mirror filtered and superimposed to obtain the audio at the original sampling rate.

[0061] 3. Using the Modified Discrete Cosine Transform (MDCT) as the vocoder output eliminates the need for window overlap to smooth adjacent frame boundaries. The number of vocoder output sampling points is the same as the number of audio sampling points, avoiding redundant calculations and improving vocoder efficiency. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0064] Figure 1 This is a basic conceptual diagram of the present invention;

[0065] Figure 2 This is a flowchart of the vocoder of the present invention;

[0066] Figure 3 This is a structural diagram of the vocoder of the present invention;

[0067] Figure 4 This is a schematic diagram illustrating the optimization target of the vocoder provided by the present invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0069] The symbols used in this invention are explained below:

[0070] x: Ground Truth audio;

[0071] The audio reconstructed by the vocoder;

[0072] fea: Vocoder input feature;

[0073] T: Number of frames of vocoder input features;

[0074] D: Vocoder input feature dimension;

[0075] H: Vocoder input feature frame shift;

[0076] Sr: Audio sampling rate;

[0077] Frequency domain representation of each sub-band of the audio reconstructed by the vocoder;

[0078] The audio of each sub-band of the audio reconstructed by the vocoder;

[0079] M: Number of sub-bands;

[0080] like Figure 1 As shown, the original audio can be perfectly reconstructed using an encoder and a decoder. The encoder uses an anti-aliasing filter bank to divide the original audio into bandpass signals with non-overlapping (orthogonal) frequency ranges, and then downsamples the bandpass signals. The decoder first upsamples each sub-band signal output by the encoder, then performs anti-mirror filtering on the upsampled signals, and finally adds the sub-band signals together to obtain the reconstructed output, which is consistent with the original audio (i.e., achieves perfect reconstruction).

[0081] During the aforementioned perfect reconstruction process, the encoder output The sampling rate is 1 / 3 of the original audio. (M represents the number of sub-bands). Considering that although the frequency ranges of the sub-bands do not overlap, the speech production mechanism involves oral resonance and the frequency components are harmonics, therefore the frequency components of each sub-band are correlated. We can utilize the correlation between the sub-bands to compress information in the encoder. Therefore, the inventors conceived of using a neural network to represent the correlation between the frequency components of each sub-band, outputting the content of each sub-band using the neural network, and then using a decoder to perfectly reconstruct the original audio. When the input to the neural network is the acoustic representation of the original audio, the neural network is called a vocoder. It is easy to see that the neural network can simultaneously output the audio components of each sub-band, and the output sampling rate is only a fraction of the original audio. This greatly improves the generation efficiency of the vocoder.

[0082] like Figure 2 As shown, the present invention provides a speech synthesis method, including:

[0083] Step S101, the vocoder inputs features fea∈R T×D Where T is the number of frames for the feature and D is the feature dimension. This feature is a low-resolution intermediate representation of the text model from the acoustic model, such as the mel feature. The frame shift corresponding to feature fea is H. The audio reconstructed by the vocoder is... The sampling rate is Sr;

[0084] Step S102: Upsample the input features using the vocoder backbone network to obtain the frequency domain representation y of each sub-band of the audio to be synthesized. i ∈C T×H′ , where i≤M is the sub-band index, M is the number of frequency bands, and H′ is the dimension of the frequency domain representation.

[0085] It should be noted that the sub-frequency bands used in this invention are based on two premises: (Premise 1.) the frequency domains of the audio in each sub-frequency band are mutually orthogonal; (Premise 2.) the audio sampling rate of each sub-frequency band is reduced to... (sampling rate is the original) During the training process, the above two premises will be constrained, which will be explained in detail below.

[0086] In step S103, the vocoder outputs the MDCT representation of each sub-band audio as its respective frequency domain representation. Due to the downsampling in premise 2 of step S102, the MDCT representation of each sub-band... The frame shift is

[0087] To achieve critical sampling in the vocoder, this invention restricts the frequency domain representation of sub-bands. The dimension is equal to the frame shift, therefore we have Limiting MDCT characterization The corresponding window length is Therefore, there is This ensures Since there is no overlap between adjacent frames, the Inverse Modified Discrete Transform (IMDCT) can perfectly reconstruct frames that do not overlap. Therefore, the number of output sampling points of the vocoder is consistent with the number of output audio sampling points (reaching the critical sampling), avoiding redundant calculations and improving the efficiency of the vocoder.

[0088] Step S104: MDCT coefficients of each sub-band in step S103 Perform inverse IMDCT to recover the audio corresponding to each sub-channel. The pseudocode for implementing IMDCT is as follows: Input: MDCT coefficients of the i-th sub-band Output: The audio value of the i-th sub-band

[0089] Algorithm implementation of IMDCT:

[0090] for each frame's MDCT coefficients f in

[0091] By mirroring and reconstructing f, an odd function is formed:

[0092] f = cat(f, -flip(f))

[0093] Pre-rotation:

[0094] N-point IFFT: f = IFFT(f)

[0095] Rotation correction:

[0096] Get the real part:

[0097] Restoration Gain:

[0098] Add a window:

[0099] end for

[0100] For each frame signal f, For window moving, The output is obtained by stacking the window lengths.

[0101] CosineWindow represents a cosine window, and flip represents reversing the order of the signal.

[0102] Step S106, for Upsample by M times to obtain Thus, the original sampling rate is restored:

[0103]

[0104] upsample represents the upsampling operation, which is to... Insert M-1 zeros between any two adjacent sampling points. This is because upsampling will cause... A mirror image is generated in the frequency domain, and in the subsequent step S107, it needs to be... Perform anti-mirror filtering.

[0105] Step S107: Use the anti-mirror filter bank to filter the signals of each sub-band. Anti-mirror filtering is performed to eliminate the mirror image caused by signal upsampling in step S106.

[0106] The impulse response of the anti-mirror filter bank is:

[0107]

[0108] Where N is the number of taps in the impulse response, M is the number of sub-bands in the filter bank, and the impulse response of the prototype filter h(n) is:

[0109]

[0110] Where cutoff is the cutoff frequency ratio.

[0111] The outputs of each sub-band after anti-mirror filtering are:

[0112]

[0113] Operators This represents the convolution operation.

[0114] Step S108: Output each channel of the filter from step S107. Add them together to obtain the output speech of the vocoder.

[0115]

[0116] Figure 3 This is a structural diagram of the vocoder of the present invention, as shown below. Figure 3 As shown, the acoustic representation fea∈R T×DAfter being fed into the backbone network, the MDCT amplitude spectrum of each sub-band is obtained. and phase spectrum Therefore, we have the MDCT coefficients for each sub-band: Same as steps S104 to S108, The vocoder output is obtained after passing through IMDCT, upsampling, anti-mirror filtering, and superposition. The MDCT of each sub-band is generated by a neural network, and the MDCT data volume can be perfectly reconstructed with the same number of audio sampling points as the final output. There is no need to use inter-frame overlap to ensure a smooth transition between frames, thus improving the generation efficiency of the vocoder.

[0117] The vocoder of this invention uses a ConvNext backbone network, which consists of eight stacked ConvNextBlock layers. Each ConvNextBlock comprises deepwise convolutions (DeepwiseConv1D) with large kernels and pointwise convolutions (PointwiseConv1D). The deepwise convolutions (group convolutions with the same number of groups as the output channels) are used to capture temporal information, while the pointwise convolutions (convolutions with a kernel size of 1) are used to capture frequency domain information. By stacking multiple layers of deep convolutions with large kernels, the receptive field of the backbone model can be increased without significantly increasing the computational cost, allowing the vocoder to obtain greater contextual information. Furthermore, ConvNext replaces transposed convolutions with convolution channel upsampling, avoiding the periodic noise of transposed convolutions.

[0118] The training scheme of the present invention is as follows: Figure 3 As shown, its optimization objective is:

[0119] Objective 1. Audio recovered by the vocoder As consistent as possible with GroundTruth audio x;

[0120] Objective 2. Audio frequencies of each sub-band in the vocoder. The sub-bands should be as consistent as possible with those of GroundTruth.

[0121] Therefore, this invention directly minimizes the MAE of the spectra of both the model-generated audio and the Ground Truth audio in Objective 1 and Objective 2. Specifically, the loss function for Objective 1 is:

[0122]

[0123] Wherein, STFT represents Short Time Fourier Transform, x represents the audio reconstructed by the vocoder, and x represents the Ground Truth audio.

[0124] Objective 2: Minimize the sub-bands of the Ground Truth audio and their corresponding sub-bands generated by the vocoder. The difference is such that the components of each sub-band of GroundTruth need to be calculated first. Steps S201-S204-S205 are the process of calculating the components of each sub-band of GroundTruth. First, step S204 will calculate x∈R. T*H After performing anti-aliasing filtering, the audio of each sub-band is obtained.

[0125]

[0126] Where M is the number of sub-bands, f i ′ represents the anti-aliasing filter bank:

[0127]

[0128] Where N is the number of taps in the impulse response, M is the number of sub-bands in the filter bank, and the impulse response of the prototype filter h(n) is:

[0129]

[0130] Where cutoff is the cutoff frequency ratio.

[0131] Subsequently, step S205... Perform downsampling:

[0132]

[0133] downsample means that... Downsampling is performed, meaning one sample is taken every M sampling points. Due to the anti-mirror filtering in step S204, each sub-frequency band... Since the frequency ranges are mutually orthogonal bandpass signals, the downsampling in step S205 will not cause spectral aliasing and thus information loss.

[0134] Output of step S205 After upsampling, anti-mirror filtering, and superposition, perfect reconstruction is possible. This means that as long as the audio output of each sub-band of the vocoder is as consistent as possible with the Ground Truth, high-quality audio can be reconstructed. Therefore, we need to constrain the output of each sub-band. The output of step S205 As consistent as possible. Specifically, the spectrum of each sub-band is constrained to be as consistent as possible, and the corresponding loss function is:

[0135]

[0136] Consistent with the standard HiFiGAN training process, this invention employs an additional decision unit to assist in vocoder training. The basic idea is as follows: 1. Fix the vocoder and train the decision unit, enabling it to distinguish between audio generated by the generator and the original audio. 2. Fix the decision unit and train the vocoder, giving it stronger generation capabilities, making it impossible for the decision unit to distinguish between generated and original audio. 3. Ensure that the output values ​​of the audio generated by the vocoder and the original audio are as consistent as possible across the intermediate layers of the decision unit. For simplicity, this invention uses L... adv Refers to the cost of the decision-maker during the training process.

[0137] Finally, the training loss function is:

[0138] L = L target1 +L target2 +L adv .

[0139] In summary, this invention proposes a vocoder using transposed convolution and critical sampling, and also proposes a speech synthesis method based on this vocoder and a training and optimization method for the vocoder, which has the following advantages:

[0140] 1. ConvNext is used as the vocoder backbone. ConvNext replaces transposed convolution with convolution channel upsampling, avoiding the periodic noise of transposed convolution; it uses depthwise convolution and separable convolution, which not only captures the time-frequency information of the vocoder input, but also greatly reduces the computational load of the backbone network.

[0141] 2. By employing frequency band fusion technology, the vocoder can simultaneously output audio components of different frequencies in the synthesized audio, thereby reducing the output sampling rate and improving the vocoder's operating efficiency. Specifically, the vocoder uses multi-channel output, with each channel corresponding to an audio signal within a specific frequency band. The frequency ranges of the signals from each channel are orthogonal to each other, and the sampling rate is 1 / M of the original (where M is the number of frequency bands). Finally, the audio signals from each frequency band are anti-mirror filtered and superimposed to obtain the audio at the original sampling rate.

[0142] 3. Using the Modified Discrete Cosine Transform (MDCT) as the vocoder output eliminates the need for window overlap to smooth adjacent frame boundaries. The number of vocoder output sampling points is the same as the number of audio sampling points, avoiding redundant calculations and improving vocoder efficiency.

[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0144] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0145] The above description is merely a preferred example of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A speech synthesis method characterized by, Includes the following steps: Steps , inputting features to vocoder , features is a low-resolution intermediate representation modeled from text for an acoustic model, where is a number of frames of features, is a feature dimension, features corresponding frame shift is ; step Using the vocoder backbone network to process input features Upsampling is performed to obtain the frequency domain representation of each sub-band of the audio to be synthesized. ,in For sub-band indexing, For the number of frequency bands, For the dimension of frequency domain representation; step The vocoder backbone network outputs audio from each sub-band. Characterization As their respective frequency domain representations, each sub-frequency band Characterization The frame shift is ; step For the steps Each sub-band Perform inverse modified discrete cosine transform Recover the audio corresponding to each sub-channel. ; step ,right Upsampling Doubled : ， in, Indicates an upsampling operation; step Using anti-mirror filter banks to filter signals in each sub-band Perform anti-mirror filtering; the output of each sub-band after anti-mirror filtering is : ， Operators This represents the convolution operation. For anti-mirror filter banks; step , the steps The output channels of the anti-mirror filter bank Add them together to get the output speech. : 。 2. The speech synthesis method according to claim 1, characterized in that: In step 102, the frequency domains of each sub-band are orthogonal to each other, and the audio sampling rate is 1 / 3 of the original sampling rate. .

3. The speech synthesis method according to claim 2, characterized in that: In step 103, the sub-band frequency domain characterization Dimensions Equal to frame shift, that is , The corresponding window length is , .

4. The speech synthesis method according to claim 1, characterized in that: In step 107, the impulse response of the anti-mirror filter bank is: ， in The number of taps for the impact response. The number of sub-bands of the filter bank, the prototype filter The impulse response is: ， in This represents the cutoff frequency ratio.

5. A vocoder, characterized in that, A method for implementing the speech synthesis method as described in claim 1 includes a backbone network and a sub-frequency band fusion module, wherein the backbone network is... A neural network model whose input is an acoustic representation and whose output is the audio components of each sub-band. Characterized by the fact that the frequency domains of the various sub-bands output by the backbone network are mutually orthogonal, and the audio sampling rate is 1 / 3 of the original sampling rate. , Number of frequency bands; The sub-frequency band fusion module includes The inverse transform module, upsampling module, and anti-mirror filtering module are used to process the audio components of each sub-band. Characterization Inverse transform, upsampling of bandpass signals for each channel, and anti-mirror filtering.

6. The vocoder according to claim 5, characterized in that: The backbone network consists of multiple Stacked together, each Both consist of deep convolutional layers with large kernels and separable convolutional layers, where deep convolutional layers are used to capture temporal information and separable convolutional layers are used to capture frequency information.

7. A training and optimization method for the vocoder as described in claim 5, characterized in that, include: Original audio The extracted acoustic representation is input into the vocoder, and the vocoder outputs the reconstructed audio. Compare the reconstructed audio With the original audio The differences should be addressed to make them as consistent as possible. Calculate the components of each sub-band of the original audio, and compare the components of each sub-band of the original audio with the audio of each sub-band reconstructed by the vocoder. The differences should be addressed to make them as consistent as possible. The reconstructed audio and the original audio are fed into the decision processor to make the output values ​​of the audio generated by the vocoder and the original audio as consistent as possible in each intermediate layer of the decision processor.

8. The training optimization method according to claim 7, characterized in that, The loss function is: ， in, Audio generated for vocoder model With GT Audio MAE; for Each sub-band of the audio and its corresponding sub-band generated by the vocoder MAE; The cost of the decision-making device.

9. The training optimization method according to claim 8, characterized in that, The calculation formula is: ， in, Represents the short-time Fourier transform, Audio reconstructed for vocoder for Audio.

10. The training optimization method according to claim 8, characterized in that, The calculation process is as follows: step ,Will audio Anti-aliasing filtering is performed to obtain the audio of each sub-band. : ， in The number of sub-bands, For anti-aliasing filter banks: ； ， in The number of taps for the impact response. The number of sub-bands of the filter bank, the prototype filter The impulse response is: ； in This is the cutoff frequency ratio; step ,right Perform downsampling: ， in Indicates to Perform downsampling; The calculation formula is: 。

Images