A multi-scale discrimination method and a multi-scale discriminator for an audio signal
Patent Information
- Application Number
- CN202311415952.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-27
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2043-10-27
AI Technical Summary
[0005]本发明提供了一种音频信号的多尺度判别方法及多尺度判别器，能够解决现有判别器无法在不同的频段上拥有不同的时频域分辨率来捕捉重点不同的信息的问题
[0038](1)本发明提供的音频信号的多尺度判别方法及多尺度判别器，通过使用常数Q变换(Constant Q Transform，CQT)得到的频谱作为特征。该特征在低频区域拥有更高的频域分辨率，从而可以更好地追踪基频的变化；在高频区域拥有更好的时域分辨率，从而可以追踪快速变化的泛音。相比于短时傅里叶变换得到的频谱，该特征对歌声、音乐等信号拥有更好的表征能力。与目前常用的多尺度短时傅里叶变换判别器相比，本发明训练出来的多尺度判别器，在频谱恢复质量与基频的稳定性上都取得了更好的效果，从而获得了更高的音质。
Smart Images
Figure CN117423352B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multi-scale discrimination method and a multi-scale discriminator for audio signals, belonging to the field of audio processing technology. Background Technology
[0002] As an important module in speech synthesis systems, vocoders are used to convert textual linguistic / acoustic features into audio waveform signals. In recent years, with the application of deep learning technology in speech synthesis, neural network-based vocoders have been widely used.
[0003] A neural network-based vocoder comprises two key components: a generator and a discriminator. The generator produces samples that resemble real samples, while the discriminator classifies the received sample data, determining whether it is a real sample or a generated sample, and feeds the result back to the generator. Currently, discriminators are mainly divided into time-domain discriminators and time-frequency domain discriminators.
[0004] Most time-frequency discriminators are based on the short-time Fourier transform (SFT). However, since the time-frequency resolution of the spectrum obtained by the SFT remains unchanged, for certain audio signals, such as singing, this transform cannot use different time-frequency resolutions in different frequency bands to capture information with different focuses. Therefore, such time-frequency discriminators based on the SFT cannot effectively model such signals. Summary of the Invention
[0005] This invention provides a multi-scale discrimination method and a multi-scale discriminator for audio signals, which can solve the problem that existing discriminators cannot capture information with different focus by having different time-frequency domain resolutions in different frequency bands.
[0006] On one hand, the present invention provides a multi-scale discrimination method for audio signals, the method comprising:
[0007] S1. Perform multiple constant Q transforms on the audio signal based on different numbers of frequency points to obtain multiple spectra;
[0008] S2. Perform probability prediction on the multiple spectra respectively to obtain multiple predicted probabilities;
[0009] S3. The audio signal is judged according to the multiple predicted probabilities to obtain the judgment result of the audio signal.
[0010] Optionally, S2 specifically includes:
[0011] Feature extraction is performed on the multiple spectra to obtain the implicit representation of each spectrum;
[0012] Probabilistic predictions are performed on the implicit representation of each spectrum to obtain multiple predicted probabilities.
[0013] Optionally, feature extraction is performed on the multiple spectra to obtain an implicit representation of each spectrum, specifically including:
[0014] For each spectrum, features are extracted from the real and imaginary parts respectively to obtain real part features and imaginary part features;
[0015] The implicit representation of each spectrum is obtained by concatenating the real part features and the corresponding imaginary part features.
[0016] Optionally, feature extraction is performed on the real and imaginary parts of each spectrum separately to obtain real part features and imaginary part features, specifically including:
[0017] The real and imaginary parts of each spectrum are divided into multiple sub-bands by octaves, and features are extracted from each sub-band to obtain the implicit representation of each sub-band.
[0018] The implicit representations of the sub-bands corresponding to the real part of the spectrum are spliced together to obtain the real part features of the spectrum, and the implicit representations of the sub-bands corresponding to the imaginary part of the spectrum are spliced together to obtain the imaginary part features of the spectrum.
[0019] Optionally, feature extraction is performed on each subband to obtain the implicit representation of each subband, specifically:
[0020] Each subband is fed into a convolutional layer to obtain an implicit representation of each subband;
[0021] The convolutional kernel of the convolutional layer is 3×9.
[0022] Optionally, S2 specifically includes:
[0023] The multiple spectra are input into the convolution module to obtain multiple predicted probabilities; wherein the convolution module contains multiple 2D convolutional layers.
[0024] On the other hand, the present invention provides a multi-scale discriminator for audio signals, the discriminator comprising:
[0025] Multiple sub-discriminators; each sub-discriminator includes a transformation module and a prediction module. The transformation module performs a constant Q-transform on the audio signal to obtain a spectrum; the prediction module performs probability prediction on the spectrum to obtain a predicted probability; the transformation modules of the multiple sub-discriminators have different numbers of frequency points when performing the constant Q-transform.
[0026] An integrated model is used to integrate multiple predicted probabilities to obtain the discrimination result of the audio signal.
[0027] Optionally, the prediction module includes:
[0028] The feature extraction submodule is used to extract features from the spectrum to obtain an implicit representation of the spectrum;
[0029] The convolutional submodule is used to perform probabilistic prediction on the implicit representation of the spectrum to obtain the predicted probability.
[0030] Optionally, the feature extraction submodule specifically includes:
[0031] The sub-band processing unit is used to extract features from the real and imaginary parts of the spectrum, respectively, to obtain real part features and imaginary part features;
[0032] The splicing unit is used to splice the real part features and the corresponding imaginary part features of the spectrum to obtain the implicit representation of the spectrum.
[0033] Optionally, the sub-band processing unit specifically includes:
[0034] Divide into sub-units to divide the real and imaginary parts of the spectrum into multiple sub-bands in octaves;
[0035] Convolutional sub-units are used to extract features from each sub-band separately, obtaining the implicit representation of each sub-band;
[0036] The splicing subunit is used to splice the implicit representations of the sub-bands corresponding to the real part of the spectrum to obtain the real part features of the spectrum, and to splice the implicit representations of the sub-bands corresponding to the imaginary part of the spectrum to obtain the imaginary part features of the spectrum.
[0037] The beneficial effects that this invention can produce include:
[0038] (1) The multi-scale discrimination method and discriminator for audio signals provided by this invention use the spectrum obtained by constant Q transform (CQT) as a feature. This feature has higher frequency domain resolution in the low-frequency region, thus better tracking the changes in the fundamental frequency; and better time domain resolution in the high-frequency region, thus tracking rapidly changing overtones. Compared with the spectrum obtained by short-time Fourier transform, this feature has better representation ability for signals such as singing and music. Compared with the commonly used multi-scale short-time Fourier transform discriminators, the multi-scale discriminator trained by this invention achieves better results in both spectrum recovery quality and fundamental frequency stability, thus obtaining higher sound quality.
[0039] (2) The multi-scale discrimination method and multi-scale discriminator for audio signals provided by the present invention utilize the sub-band processing unit to obtain an implicit representation with time-domain information alignment, and use the implicit representation as the representation used by the discriminator. This can solve the problem of misalignment of time-domain information between different octaves in the existing CQT spectrum algorithm (i.e., the amount of time-domain information contained in bins corresponding to different octaves in frequency is inconsistent in each time frame). Attached Figure Description
[0040] Figure 1 This is a flowchart of the multi-scale discrimination method for audio signals provided in an embodiment of the present invention;
[0041] Figure 2 This is a block diagram of a multi-scale discriminator structure provided in an embodiment of the present invention;
[0042] Figure 3 A schematic diagram of the CQT spectrum and implicit characterization curve provided for embodiments of the present invention. Detailed Implementation
[0043] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.
[0044] This invention provides a multi-scale discrimination method for audio signals, such as... Figure 1 As shown, the method includes:
[0045] S1. Perform multiple constant Q transformations on the audio signal based on different numbers of frequency points to obtain multiple spectra.
[0046] The number of frequency points refers to the number of frequency bins in an octave.
[0047] The same audio signal is subjected to constant Q transform multiple times to obtain multiple spectra. The number of frequency bins in each octave is different during each constant Q transform to achieve a multi-scale effect.
[0048] S2. Perform probability predictions on multiple spectra separately to obtain multiple prediction probabilities.
[0049] In practical applications, multiple spectra can be input into the convolutional module to obtain the predicted probability of each spectrum; wherein, the convolutional module contains multiple 2D convolutional layers. This embodiment of the invention does not limit the number of 2D convolutional layers; those skilled in the art can set it according to actual conditions. For example, it can be 3 or 5, etc.
[0050] It should be noted that when using this spectrum directly for probability prediction, the number of channels in the convolutional layer needs to be increased because the temporal information of the spectrum is not aligned.
[0051] S3. Based on multiple predicted probabilities, the audio signal is discriminated to obtain the discrimination result of the audio signal.
[0052] Specifically, S2 may include:
[0053] Feature extraction is performed on multiple spectra separately to obtain the implicit representation of each spectrum.
[0054] Specifically, this includes: (1) extracting features from the real and imaginary parts of each spectrum to obtain real features and imaginary features.
[0055] In this embodiment of the invention, firstly, the real and imaginary parts of each spectrum are divided into multiple sub-bands in octaves.
[0056] Then, feature extraction is performed on multiple sub-bands to obtain the implicit representation of each sub-band; specifically, the sub-bands are input into the convolutional layer to obtain the implicit representation of each sub-band; wherein, the convolutional kernel of the convolutional layer is 3×9.
[0057] Finally, the implicit representations of all sub-bands divided by the real part of the spectrum are spliced together to obtain the real part features of the spectrum, and the implicit representations of all sub-bands divided by the imaginary part of the spectrum are spliced together to obtain the imaginary part features of the spectrum.
[0058] (2) The real part features and the corresponding imaginary part features of the spectrum are concatenated to obtain the implicit representation of each spectrum.
[0059] Then, the implicit representation of each spectrum is used for probability prediction to obtain multiple predicted probabilities.
[0060] This invention uses the spectrum obtained by constant Q-transform as a feature. This feature has higher frequency domain resolution in the low-frequency region, thus better tracking changes in the fundamental frequency; and better time domain resolution in the high-frequency region, thus tracking rapidly changing overtones. Compared to the spectrum obtained by short-time Fourier transform, this feature has a better representation ability for signals such as singing and music. Compared with commonly used multi-scale short-time Fourier transform discriminators, the multi-scale discriminator trained by this invention achieves better results in both spectrum recovery quality and fundamental frequency stability, thus obtaining higher sound quality.
[0061] Another embodiment of the present invention provides a multi-scale discriminator for audio signals, the structural block diagram of which is shown below. Figure 2 As shown, this multi-scale subband processing discriminator based on constant Q-transform mainly consists of multiple structurally consistent sub-discriminators. Each sub-discriminator includes a subband processing unit that aligns the temporal information between different octaves of the CQT spectrum, and a series of convolutional layers that produce the discriminator output. Specifically, the multi-scale discriminator includes:
[0062] Multiple sub-discriminators; each sub-discriminator includes a transform module and a prediction module. The transform module performs a constant Q transform on the audio signal to obtain the spectrum; the prediction module performs probability prediction on the spectrum to obtain the predicted probability. The number of frequency points used by the transform modules of the multiple sub-discriminators when performing the constant Q transform is different.
[0063] First, the same audio signal is input to different sub-discriminators. Then, in the transformation module of each sub-discriminator, the audio signal is transformed by a constant Q-transform to obtain the spectrum. The number of frequency bins per octave is different for each sub-discriminator during the constant Q-transform to achieve a multi-scale effect. For example, this invention uses three sub-discriminators, with 24, 36, and 48 bins respectively selected as the number of bins per octave in the CQT spectrum.
[0064] Furthermore, the prediction module includes:
[0065] The feature extraction submodule is used to extract features from the spectrum to obtain an implicit representation of each spectrum.
[0066] The feature extraction submodule specifically includes:
[0067] The sub-band processing unit is used to extract features from the real and imaginary parts of the spectrum, respectively, to obtain real part features and imaginary part features.
[0068] Specifically, the subband processing unit first divides the real and imaginary parts of the spectrum into multiple subbands by octaves by using a partitioning subunit. Then, it uses a convolution subunit to extract features from each subband, obtaining the implicit representation of each subband. Finally, a concatenation subunit concatenates the implicit representations of all subbands divided by the real part of the spectrum to obtain the real part features of the spectrum, and concatenates the implicit representations of all subbands divided by the imaginary part of the spectrum to obtain the imaginary part features of the spectrum.
[0069] refer to Figure 2 As shown, the real and imaginary parts of the spectrum of the constant Q-transform are input into the sub-band processing unit proposed in this invention. The sub-band processing unit first divides the input matrix into different sub-bands by octaves. Then, each sub-band is passed through a convolutional sub-unit (i.e., a convolutional layer) to obtain the implicit representation (or latent representation) of that sub-band. The concatenation sub-unit concatenates all the implicit representations to obtain the output of the sub-band processing unit. The kernel size of the convolutional layer can be set to (3, 9).
[0070] The splicing unit splices the real and imaginary features of the spectrum to obtain an implicit representation of the spectrum.
[0071] The splicing unit splices together the outputs of the sub-band processing units corresponding to the real and imaginary parts to obtain the temporal information aligned representation, that is, the implicit representation of the spectrum.
[0072] The convolutional submodule is used to perform probabilistic prediction on the implicit representation of the spectrum to obtain the predicted probability of the spectrum.
[0073] The implicit representation of the spectrum is input into a series of convolutional layers to obtain the output of the sub-discriminator, namely the predicted probability of the spectrum and the feature map obtained from the output of each layer in the series of convolutional layers. This invention does not limit the specific number of convolutional layers included in the convolutional sub-module; those skilled in the art can set it according to actual conditions. For example, the convolutional module mainly consists of five 2D convolutional layers. The first 2D convolutional layer has a kernel of size (3, 8) and 32 channels. The next three convolutional layers have dilation rates of 1, 2, and 4 in the time dimension and a convolutional stride of size 2 in the frequency dimension. The last convolutional layer has a kernel of size (3, 3) and a convolutional stride of size (1, 1).
[0074] An integrated model is used to integrate the predicted probabilities of multiple spectra to obtain the discrimination result of the audio signal.
[0075] The ensemble model combines the outputs of all sub-discriminators into a list to obtain the output of the entire discriminator, which includes the discrimination results of the audio signal.
[0076] The spectrum obtained by the existing constant Q transform suffers from a misalignment of time-domain information; that is, frequency points at different octaves within the same time frame contain different time-domain information. This can lead to issues such as... Figure 3 The problem shown in the lower half, as demonstrated by experiments, is that such features are difficult for neural networks to extract, negatively impacting network learning. This invention employs sub-band processing units to alleviate this problem, such as... Figure 3 As shown in the upper part, an equivalent temporal information aligned representation is obtained through representation learning, and this representation is used as the object of subsequent network learning. Experiments show that this approach can significantly improve the learning effect.
[0077] This invention utilizes a subband processing unit to obtain an implicit representation of time-domain information alignment, and uses this implicit representation as the representation used by the discriminator. This can solve the problem of misalignment of time-domain information between different octaves in the existing CQT spectrum algorithm (i.e., the amount of time-domain information contained in bins corresponding to different octaves in frequency is inconsistent in each time frame).
[0078] The short-time Fourier transform (SFT) spectrum and the constant Q-transform spectrum have different time-frequency domain resolution distributions, which allows these two different features to be "complementary." Therefore, discriminators designed based on them can be naturally used together. Thus, in practical applications, jointly training the multi-scale SFT discriminator with the constant Q-transform-based multi-scale discriminator of this invention can further improve the quality of synthesized audio.
[0079] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.
Claims
1. A multi-scale discrimination method for audio signals, characterized in that, The method includes: S1. Multiple sub-discriminators perform multiple constant Q-transforms on the audio signal based on different numbers of frequency points to obtain multiple spectra; S2. The multiple sub-discriminators extract features from the multiple spectra respectively to obtain an implicit representation of each spectrum; and perform probability prediction on the implicit representation of each spectrum to obtain multiple predicted probabilities; the sub-discriminators are used to align the time-domain information between different octaves of the CQT spectrum. S3. The audio signal is judged according to the multiple predicted probabilities to obtain the judgment result of the audio signal.
2. The method according to claim 1, characterized in that, Feature extraction is performed on the multiple spectra to obtain an implicit representation of each spectrum, specifically including: For each spectrum, features are extracted from the real and imaginary parts respectively to obtain real part features and imaginary part features; The implicit representation of each spectrum is obtained by concatenating the real part features and the corresponding imaginary part features.
3. The method according to claim 2, characterized in that, For each spectrum, feature extraction is performed on the real and imaginary parts separately to obtain real part features and imaginary part features, specifically including: The real and imaginary parts of each spectrum are divided into multiple sub-bands by octaves, and features are extracted from each sub-band to obtain the implicit representation of each sub-band. The implicit representations of the sub-bands corresponding to the real part of the spectrum are spliced together to obtain the real part features of the spectrum, and the implicit representations of the sub-bands corresponding to the imaginary part of the spectrum are spliced together to obtain the imaginary part features of the spectrum.
4. The method according to claim 3, characterized in that, Feature extraction is performed on each subband to obtain the implicit representation of each subband, specifically: Each subband is fed into a convolutional layer to obtain an implicit representation of each subband; The convolutional kernel of the convolutional layer is 3×9.
5. The method according to claim 1, characterized in that, S2 specifically includes: The multiple spectra are input into the convolution module to obtain multiple predicted probabilities; wherein the convolution module contains multiple 2D convolutional layers.
6. A multi-scale discriminator for audio signals, characterized in that, The discriminator includes: Multiple sub-discriminators are included; each sub-discriminator comprises a transform module and a prediction module. The transform module performs a constant Q-transform on the audio signal to obtain a spectrum; the prediction module performs probability prediction on the spectrum to obtain a prediction probability. The transform modules of the multiple sub-discriminators have different numbers of frequency points when performing the constant Q-transform. The prediction module includes: a feature extraction sub-module for extracting features from the spectrum to obtain an implicit representation of the spectrum; and a convolution sub-module for performing probability prediction on the implicit representation of the spectrum to obtain a prediction probability. The sub-discriminators are used to align the temporal information between different octaves of the CQT spectrum. An integrated model is used to integrate multiple predicted probabilities to obtain the discrimination result of the audio signal.
7. The discriminator according to claim 6, characterized in that, The feature extraction submodule specifically includes: The sub-band processing unit is used to extract features from the real and imaginary parts of the spectrum, respectively, to obtain real part features and imaginary part features; The splicing unit is used to splice the real part features and the corresponding imaginary part features of the spectrum to obtain the implicit representation of the spectrum.
8. The discriminator according to claim 7, characterized in that, The sub-band processing unit specifically includes: Divide into sub-units to divide the real and imaginary parts of the spectrum into multiple sub-bands in octaves; Convolutional sub-units are used to extract features from each sub-band separately, obtaining the implicit representation of each sub-band; The splicing subunit is used to splice the implicit representations of the sub-bands corresponding to the real part of the spectrum to obtain the real part features of the spectrum, and to splice the implicit representations of the sub-bands corresponding to the imaginary part of the spectrum to obtain the imaginary part features of the spectrum.