Speech vocoder fusing spatial perception and attention mechanism and establishment method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 湖南工商大学
- Filing Date
- 2022-08-12
- Publication Date
- 2026-08-07
AI Technical Summary
WaveNet神经网络声码器相对传统声码器显著提升了重构语音的质量,但重建语音信号时存在运算效率低下的问题
[0029] In the frame-level feature extraction module, to better capture spatial auditory features, ordinary convolution is no longer used. Instead, coordinate convolution is applied to one-dimensional speech. First, the one-dimensional speech signal is up-scaled, and then additional two-dimensional constant coordinate information, i.e., (x, y) coordinate channels, is filled into the up-scaled features to generate speech features with location information. This addresses the problems of weak generalization ability, excessive training time, and overfitting that ordinary convolution suffers from in supervised rendering and supervised coordinate classification scenarios. It improves the numerical regression and fitting capabilities of convolution and solves the problem of muffled sound caused by insufficient extraction of high-frequency harmonic components. Secondly, to enhance the robustness of the speech, the features with location information are sequentially passed through fully connected layers, channel attention, and spatial attention. Channel attention corrects the features, retaining valuable features; spatial attention acquires important features from the channels.
Smart Images
Figure CN115346541B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio processing technology, specifically to a speech vocoder and its creation method that integrates spatial perception and attention mechanisms. Background Technology
[0002] Speech synthesis is a technology that converts text into natural and fluent speech, and it has significant application value in intelligent voice interaction. Statistical parametric speech synthesis is one of the mainstream technologies in speech synthesis, generally consisting of modules such as text analysis, acoustic models, and vocoders. First, the text analysis module analyzes the input raw text to obtain text features; second, the acoustic model module predicts the acoustic features of the speech based on the text features obtained from the text analysis module; finally, the vocoder converts the acoustic features into the final speech waveform. Among these, the performance of the vocoder significantly affects the quality of the synthesized speech and has become one of the key core issues in speech synthesis research.
[0003] With the rapid development of deep learning technology, its application in signal processing has become increasingly widespread. Currently popular neural network vocoders include WaveNet, WaveRNN, Melgan, Waveglow, FastSpeech, and LPCNet. In 2016, Google proposed WaveNet, a neural network-based speech waveform generation model, and built a speech synthesis vocoder based on this model. The WaveNet neural network vocoder significantly improved the quality of reconstructed speech compared to traditional vocoders, but it suffers from low computational efficiency when reconstructing speech signals. In 2019, Jean-Marc Valin proposed an improved vocoder based on WaveRNN, LPCNet. LPCNet has advantages such as low complexity and can synthesize speech in real time, but its generation quality is lower than previous mainstream vocoders. Current research on vocoders mainly focuses on improving sound quality and reducing complexity. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, this invention proposes a speech vocoder that integrates spatial perception and attention mechanisms. It uses coordinate convolution to capture spatial auditory features and attention mechanisms to capture important harmonic components. This improves the quality of speech generation and significantly enhances synthesis efficiency, thereby achieving real-time speech synthesis.
[0005] To achieve the above objectives, one of the technical solutions adopted by this invention is: a method for establishing a speech vocoder that integrates spatial perception and attention mechanisms, comprising the following steps:
[0006] Step 1: Use the spectrogram as input to the acoustic feature extraction module, and calculate the pitch period information M using the autocorrelation function method. pThe BFCC feature M is calculated using the Bark frequency partitioning method. f ;
[0007] Step 2: Extract the BFCC features M from Step 1 f and gene cycle information M p As input, the frame-level feature extraction module generates frame-level features x”. c ;
[0008] Step 3: Extract the BFCC features M from Step 1 f As input, the LPC coefficients M are obtained through the LPC coefficient calculation module. l ;
[0009] Step 4: Use the speech sample A obtained from the analysis and synthesis module. t and the LPC coefficients M generated in step 3 l The speech signal C is generated after the linear prediction module. t ;
[0010] Step 5: Extract frame-level features x” c Speech signal C t 1. Analyze the speech sample A obtained by the synthesis module t The excitation signal S obtained by the multi-scale sampling module t Sequence data x is obtained through the feature connection module. t ;
[0011] Step 6: Convert the output x from Step 5 t As input, the hidden information h is obtained through the context information capture module. t .
[0012] Step 7: Convert the output h from step 6 t The input is fed into the sample space mapping generation module, and the combined feature d is obtained through feature concatenation. t .
[0013] Step 8: Combine the features d from Step 7 t As input, the excitation signal S is obtained through a multi-scale sampling module. t .
[0014] Step 9: The excitation signal S obtained in Step 8 t The speech signal C obtained in step 4 t The analysis and synthesis module generates speech sample A. t .
[0015] To achieve the above objectives, the second technical solution adopted by this invention is: a speech vocoder integrating spatial perception and attention mechanisms, comprising:
[0016] The module includes an acoustic feature extraction module, a frame-level feature extraction module, an LPC coefficient calculation module, a linear prediction module, a feature connection module, a context information capture module, a sample space mapping generation module, a multi-scale sampling module, and an analysis and synthesis module.
[0017] The acoustic feature extraction module extracts relevant speech features from the spectrogram; the frame-level feature extraction module endows each feature with coordinate information and retains and filters important features to extract several frame-level acoustic features; the LPC coefficient calculation module is mainly used to calculate LPC coefficients; the linear prediction module predicts the current speech sample from the LPC coefficients; the feature concatenation module combines the excitation signal, predicted speech, speech sample, and frame-level features; the context information capture module obtains the context information of the combined features; the sample space mapping generation module processes the context information to generate a probability distribution; the multi-scale sampling module samples the probability distribution to generate the excitation signal; and the analysis and synthesis module mainly adds the predicted speech and the excitation signal to obtain the synthesized speech.
[0018] The frame-level feature extraction module consists of coordinate convolution, fully connected layers, and attention. Coordinate convolution adds two-dimensional constant coordinates to the features to obtain the coordinate information of each feature, and then performs convolution operation on them. The convolved features are then fed into the fully connected layers and the attention module. The attention module mainly consists of channel attention and spatial attention. Its main function is to filter and retain important speech features to ensure the robustness of the generated speech.
[0019] The abbreviations involved in this invention are explained as follows:
[0020] FFT: Fast Fourier Transform
[0021] IFFT: Inverse Fast Fourier Transform
[0022] DCT: Discrete Cosine Transform
[0023] IDCT: Inverse Discrete Cosine Transform
[0024] LPC: Linear Prediction Coefficient
[0025] BFCC: Bark-Frequency Cepstral Coefficients
[0026] MLP: Multilayer Perceptron
[0027] MOS: Mean Opinion Score
[0028] The beneficial effects of this invention are:
[0029] In the frame-level feature extraction module, to better capture spatial auditory features, ordinary convolution is no longer used. Instead, coordinate convolution is applied to one-dimensional speech. First, the one-dimensional speech signal is up-scaled, and then additional two-dimensional constant coordinate information, i.e., (x, y) coordinate channels, is filled into the up-scaled features to generate speech features with location information. This addresses the problems of weak generalization ability, excessive training time, and overfitting that ordinary convolution suffers from in supervised rendering and supervised coordinate classification scenarios. It improves the numerical regression and fitting capabilities of convolution and solves the problem of muffled sound caused by insufficient extraction of high-frequency harmonic components. Secondly, to enhance the robustness of the speech, the features with location information are sequentially passed through fully connected layers, channel attention, and spatial attention. Channel attention corrects the features, retaining valuable features; spatial attention acquires important features from the channels.
[0030] To reduce the hidden parameters of the network structure and improve the speech synthesis speed, a minimum-gated recurrent unit (MGRU) is used instead of a gated recurrent neural network in the context information capture module. The MGRU retains only one gate structure and uses a forget gate to filter information, ensuring that the synthesis speed is improved without compromising speech synthesis quality.
[0031] This invention utilizes coordinate convolution to capture spatial auditory features in the frame-level feature extraction module, perfectly solving the problems of weak generalization ability, excessive training time, and overfitting that occur in supervised rendering and supervised coordinate classification scenarios. It also improves the numerical regression and fitting capabilities of convolution and solves the problem of muffled sound caused by insufficient extraction of high-frequency harmonic components. Attached Figure Description
[0032] Figure 1 This is a flowchart of the speech synthesis method of the present invention.
[0033] Figure 2 This is a flowchart of the frame-level feature extraction method of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0035] The speech vocoder model includes the following modules:
[0036] The system comprises an acoustic feature extraction module, a frame-level feature extraction module, an LPC coefficient calculation module, a linear prediction module, a feature connection module, a context information capture module, a sample space mapping generation module, a multi-scale sampling module, and an analysis and synthesis module. The acoustic feature extraction module primarily extracts speech features from the spectrogram and sends them as input to the frame-level feature extraction module to generate a 128-dimensional feature condition vector. This 128-dimensional feature condition vector, along with the LPC coefficients calculated by the LPC coefficient calculation module and the current speech sample predicted by the linear prediction module, are fed into the feature connection module to obtain sequence data. The sequence data then passes through the context information capture module to capture the context information of the features. This acquired context information is then fed into the sample space mapping generation module and the multi-scale sampling module for sampling processing to generate an excitation signal. Finally, the excitation signal is processed by the analysis and synthesis module to generate speech.
[0037] To address the issues of weak generalization, excessive training time, and overfitting that convolution suffers in supervised rendering and supervised coordinate classification scenarios, as well as the problem of muffled sound due to insufficient extraction of high-frequency harmonic components, the frame-level feature extraction module performs a special convolution operation on speech features. The generated features are then fed into an attention module to capture important features, ensuring good generalization ability. The frame-level feature extraction module consists of coordinate convolution, fully connected layers, and an attention module. Coordinate convolution is a special type of convolution that adds constant 2D coordinates to features, obtaining the coordinate information of each feature before performing the convolution operation, ensuring good generalization ability. The convolutional features are then fed into a fully connected layer and an attention module. The attention module mainly consists of channel attention and spatial attention, and its main function is to filter and retain important speech features, ensuring the robustness of the generated speech.
[0038] A method for establishing a speech vocoder that integrates spatial perception and attention mechanisms, see [link to relevant documentation]. Figure 1 The method includes the following steps:
[0039] Step 1: Use the spectrogram as input to the acoustic feature extraction module, calculate the pitch period information using the autocorrelation function method, and calculate the BFCC features using the Bark frequency division method;
[0040] Step 2: Using the BFCC features and gene cycle information extracted in Step 1 as input, the frame-level feature extraction module generates frame-level features;
[0041] Step 3: Simultaneously, the BFCC features extracted in Step 1 are used as input, and the LPC coefficients are obtained through the LPC coefficient calculation module.
[0042] Step 4: Use the analysis and synthesis module to obtain speech samples and the LPC coefficients generated in step 3 to generate a speech signal through the linear prediction module;
[0043] Step 5: Combine frame-level features, speech signals, speech samples obtained from the analysis and synthesis module, and excitation signals obtained from the multi-scale sampling module through the feature connection module to obtain sequence data;
[0044] Step 6: Take the output sequence data from Step 5 as input and obtain the hidden information through the context information capture module.
[0045] Step 7: Input the hidden information output from Step 6 into the sample space mapping module, and obtain combined features through feature concatenation.
[0046] Step 8: Using the combined features from Step 7 as input, obtain the excitation signal through the multi-scale sampling module.
[0047] Step 9: The excitation signal obtained in Step 8 and the speech signal obtained in Step 4 are analyzed and synthesized by the speech synthesis module to generate speech samples.
[0048] The method for speech synthesis using the vocoder model described above specifically includes the following steps:
[0049] Step 1: Obtain BFCC features M from the spectrogram f and fundamental period information M p .
[0050] Step 1.1: Obtain the fundamental period information M using the autocorrelation function method. p .
[0051] Step 1.2: M f Extraction process: The spectrogram is divided into 18 frequency bands according to Bark frequencies. The energy in each frequency band is calculated. The cepstral spectrum is obtained by taking the base-10 logarithm of the energy. Finally, the 18-dimensional BFCC feature M is obtained by decorrelation through DCT. f .
[0052] Step 2: Extract the 18-dimensional BFCC features M from Step 1 f As input, the frame-level feature extraction module generates a 128-dimensional frame-level feature x”. c ,See Figure 2 The specific steps are as follows: First, the 18-dimensional BFCC features are filled with additional 2-dimensional constant coordinate information, namely (x, y) coordinate channels. The x-coordinate channel is an h×w rank-1 matrix, with the first row filled with 0, the second row with 1, the third row with 2, and so on. The y-coordinate channel is similar, except that each column of y is filled with a constant. Then, the x and y coordinate values are linearly scaled to fall within the range [-1, 1], generating 128-dimensional features. After residual connection with the BFCC features while keeping the dimensions unchanged, the features are fed into a fully connected layer, generating a 128-dimensional feature vector x through two fully connected layers. C .
[0053] Then, the feature vector x C As input to the channel attention module, it is first subjected to dimensionality increase processing to obtain... To more efficiently compute channel attention features, speech features... Compression is achieved using average pooling and max pooling, resulting in two different types of context information after pooling. and Then and The weight coefficients W are obtained by feeding them into a parameter-sharing linear layer and using the Sigmoid function. t Weighting coefficient W t and Multiplying them together yields the channel attention feature map x'. C x' C The calculation formula is as follows:
[0054]
[0055]
[0056] In the formula, σ is the sigmoid function.
[0057] Next, the channel attention feature map x' C As input to the spatial attention module, x' C Perform global max pooling and mean pooling along the channel axis to obtain the pooled result. and The features generated by average pooling and max pooling are then concatenated. A convolutional operation is then used to generate a spatial attention map W′. t Then, the two-dimensional spatial attention map W′ t With x'C The feature map x”' is obtained by element-wise multiplication. c Finally, for x”' c Dimensionality reduction to generate 128-dimensional frame-level features x” c x” c The calculation formula is as follows:
[0058]
[0059]
[0060] x c "=f(x c "′),
[0061] Among them, f 7×7 σ is the convolution size, σ is the sigmoid function, and f is the dimensionality reduction process.
[0062] Step 3: Extract the BFCC features M from Step 1 f As input, first M f Perform an IDCT operation, then take a base-10 exponential transform to obtain the band power. Next, perform linear interpolation on the band power to obtain the power spectrum. Then, perform an IFFT transform on the power spectrum to obtain the autocorrelation sequence. Finally, use the Levinson-Durbin algorithm to process the autocorrelation sequence to obtain the LPC coefficients M. l .
[0063] Step 4: Analyze the speech sample A obtained from the synthesis module. t , and the LPC coefficient M generated in step 3 l The speech signal C is generated through linear weighting. t The calculation is as follows:
[0064]
[0065] In the formula, p is the order of linear prediction.
[0066] Step 5: Extract frame-level features x” c Speech signal C t 1. Analyze the speech sample A obtained by the synthesis module t The excitation signal S obtained by the multi-scale sampling module t Sequence data x is obtained through the feature connection module. t .
[0067] Step 6: Take the sequence data x obtained in Step 5 t As input, the important context information h′ is obtained through the loop gate unit. t Meanwhile, in order to reduce the hidden parameters of the network structure and improve the speech synthesis speed, h′ tWith x” c The input is fed into the smallest gated loop unit, h′ t With x” c Only through one forget gate f t Use a forget gate to perform all filtering and generate h. t The formula for the minimum gate unit is shown below:
[0068] f t =σ(W f [h t-1 ,x′ t ]+b f ),
[0069]
[0070]
[0071] In the formula x′ t For [h′ t ;x c "], h t-1 For past information, ⊙ represents element-wise multiplication, f t For the forget gate information, W f This is the weight matrix.
[0072] Step 7: Convert the output h from step 6 t As distributed features, the features are integrated together, weighted, and then mapped to the sample label space to obtain the combined feature d. t .
[0073] Step 8: Convert the output d from step 7 to... t As input, for d t For unsupervised classification, the output distribution is represented as a full binary tree of depth 3, resulting in a hierarchical clustering structure. By constructing a binary tree, the computational complexity of the target probability is reduced from the initial N to log2. N The order of magnitude is [value missing]. Then, the probability P(S_{n+1}) of each branch is calculated using the sigmoid function. t Then, for the probability P(S) t The excitation signal S is obtained by sampling. t Direct sampling introduces a large amount of noise, necessitating the introduction of a constant i to control the overall sampling process. If the current voiced tone has pitch, assign a value to P(S). t A larger weight is applied to ensure deterministic sampling of voiced sounds. The sampling formula is:
[0074] i = 1 + max(0, 1.5J) p -0.5)
[0075]
[0076] In the formula For standardization, J p For fundamental frequency correlation (0 <J p <1), L=0.002.
[0077] Step 9: Excitation signal S from step 8 t The linear prediction C from step 4 t Add to generate speech A t .
[0078] After the test audio was synthesized using the method of this invention, it improved the MOS score by 0.2 points (out of 5) compared to LPCNet, and the synthesis speed was increased by 8%. Therefore, the method of this invention uses coordinate convolution to capture spatial auditory features in the frame-level feature extraction module, which improves synthesis efficiency and speech generation quality.
[0079] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A method for establishing a speech vocoder that integrates spatial perception and attention mechanisms, characterized in that... The speech vocoder includes the following modules: an acoustic feature extraction module, a frame-level feature extraction module, an LPC coefficient calculation module, a linear prediction module, a feature connection module, a context information capture module, a sample space mapping generation module, a multi-scale sampling module, and an analysis and synthesis module. The acoustic feature extraction module extracts relevant speech features from the spectrogram; the frame-level feature extraction module assigns coordinate information to each feature and retains and filters important features to extract several frame-level acoustic features; the LPC coefficient calculation module calculates LPC coefficients; the linear prediction module predicts the current speech sample from the LPC coefficients; the feature connection module combines the excitation signal, predicted speech, speech sample, and frame-level features; the context information capture module acquires the context information of the combined features; the sample space mapping generation module processes the context information to generate a probability distribution; the multi-scale sampling module samples the probability distribution to generate an excitation signal; and the analysis and synthesis module adds the predicted speech and the excitation signal to obtain synthesized speech. The method includes the following steps: Step 1, using the spectrogram as input to the acoustic feature extraction module, and calculating the pitch period information using the autocorrelation function method. BFCC features were calculated using the Bark frequency partitioning method. ; Step 2, extract the BFCC features from Step 1. and fundamental frequency information As input, the frame-level feature extraction module generates frame-level features. Specifically, the frame-level feature extraction module consists of a coordinate convolution module, a fully connected layer, and an attention module. The coordinate convolution module adds two-dimensional constant coordinates to the features to obtain the coordinate information of each feature, and then performs a convolution operation on them. The convolved features are then fed into the fully connected layer and the attention module. The attention module consists of channel attention and spatial attention to filter and retain important speech features. Step 3, extract the BFCC features from Step 1. As input, the LPC coefficients are obtained through inverse discrete cosine transform, exponential transform, linear interpolation, inverse fast Fourier transform, and the Levinson-Durbin algorithm processed by the LPC coefficient calculation module. ; Step 4: Speech samples and LPC coefficients generated in Step 3 The speech signal is generated by linear weighting after linear prediction module. ; Step 5, use the frame-level features obtained in Step 2. The speech signal obtained in step 4 Speech samples and excitation signals are processed by a feature connection module to obtain sequence data. ; Step 6, take the output of step 5. As input, the hidden information is obtained by capturing context information through a minimum gated loop unit by the context information capture module. ; Step 7: Obtain the hidden information from Step 6. The input is fed into the sample space mapping generation module. As distributed features, the features are combined, weighted, and then mapped to the sample label space to obtain combined features. ; Step 8, combine the features obtained in step 7. As input, a full binary tree of depth 3 is constructed through a multi-scale sampling module for hierarchical clustering. The probability of each branch is calculated using the sigmoid function, and the excitation signal is obtained through sampling. ; Step 9, the excitation signal obtained in step 8 The speech signal obtained in step 4 Speech samples are generated after analysis and synthesis module. .
2. The method for establishing a speech vocoder that integrates spatial perception and attention mechanisms according to claim 1, characterized in that... Step 1 includes: s1.1 Obtaining the pitch period information using the autocorrelation function method. ,s1.2 The extraction process involves dividing the spectrogram into 18 frequency bands based on Bark frequencies, calculating the energy within each band, obtaining the cepstrum by taking the base-10 logarithm of the energy, and finally obtaining the 18-dimensional BFCC features through DCT decorrelation. .
3. The method for establishing a speech vocoder that integrates spatial perception and attention mechanisms according to claim 2, characterized in that... Step 2 specifically involves: processing the 18-dimensional BFCC features extracted in step 1. As input, the frame-level feature extraction module generates 128-dimensional frame-level features. .