Non-parallel arbitrary-to-arbitrary speech conversion method based on attention feature fusion

Abstract

The application belongs to the field of intelligent voice, and discloses a non-parallel arbitrary-to-arbitrary voice conversion method based on attention feature fusion. First, content features, speaker features and prosody features are extracted from original voice and target voice respectively. A bottleneck layer composed of time instance normalization attention is used to couple the extracted features to obtain a feature map. Time instance normalization and efficient channel attention mechanism are combined to dynamically integrate local phoneme details and global acoustic features in the time-frequency domain, and acoustic migration is completed. Finally, a neural vocoder is used to generate synthesized voice. The application uses instance normalization and twin loss enhancement to improve the adaptability and generalization of noise and non-parallel data, so as to achieve voice synthesis effect considering content integrity and speaker feature similarity. In addition, the application sets three different training and conversion scenes to improve the generalization and robustness of the model, so as to realize high-quality multi-item voice conversion.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent speech technology, specifically to the field of speech conversion, and more specifically to a method for realizing non-parallel speech conversion through an encoder-decoder architecture and an attention mechanism, which is suitable for scenarios such as speech synthesis, speech enhancement, and personalized voice assistants. Background Art

[0002] Speech conversion technology, a major application of generative AI, involves converting the speech characteristics of an original speaker into those of a target speaker. This technology aims to modify individual characteristics of the speech, such as timbre and intonation, while preserving the semantic content of the original speech. This technology has broad application prospects in areas such as speech synthesis, speech enhancement, and personalized voice assistants. Traditional speech conversion methods primarily rely on signal processing techniques, such as linear predictive coding and spectral envelope conversion, which analyze and modify the spectral characteristics of speech signals to achieve timbre conversion. However, these traditional methods often have limited performance when processing complex speech features, especially in cross-language or multi-speaker scenarios, where the conversion effect fails to meet practical requirements.

[0003] With the rapid development of deep learning technology, neural network-based speech conversion methods have gradually become mainstream. These methods utilize models such as deep neural networks, convolutional neural networks, and recurrent neural networks to more effectively capture the nonlinear characteristics of speech signals, thereby achieving higher-quality speech conversion. For example, autoencoder-based speech conversion methods significantly improve the naturalness and similarity of the conversion by encoding the speech signal into a latent feature representation and then decoding it into the speech characteristics of the target speaker.

[0004] Although deep learning technology has made significant progress in the field of speech conversion, it still faces several challenges. First, high-quality speech conversion typically requires a large amount of parallel corpus (i.e., speech data of the same text recorded simultaneously by the original and target speakers), which is expensive to obtain. Second, speech conversion using non-parallel corpus remains a difficult problem, especially when the speech styles of the original and target speakers differ significantly, which can easily distort the conversion effect. In addition, how to better preserve the emotional information and prosodic features of the speech during the conversion process is also a key research direction.

[0005] Patent publication number CN118969023A discloses a speech conversion method based on an attention mechanism and a stream matching strategy, comprising: the feature extraction module extracts features from the acquired audio to be converted to obtain audio features; the feature coupling module learns the correspondence between the pitch features, emotional features, and content features based on the attention mechanism, and obtains a latent vector based on the correspondence and the preset identity features of the target speaker; the stream matching module performs Mel-spectrometric calculation on the latent vector based on a trained Mel-spectrometric model to obtain Mel-spectrometric features, and outputs the Mel-spectrometric features to a voice coding output module; the voice coding output module processes the Mel-spectrometric features that meet preset decoding conditions to obtain processed Mel-spectrometric features, and obtains audio after identity conversion based on the processed Mel-spectrometric features.

[0006] Patent publication number CN119274565A discloses a speech conversion method based on data enhancement, which includes: performing data enhancement on the speech to be converted to obtain a sample speech; inputting the sample speech into a prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; inputting the sample speech into a content encoder for processing to obtain content features of the sample speech; inputting the prosody features and content features into an attention-based alignment model for alignment processing to obtain aligned prosody features; inputting the aligned prosody features, content features, spectral features and a speaker vector with a set timbre into a decoder for synthesis to obtain synthesized speech; and intercepting part of the content corresponding to the speech to be converted from the synthesized speech as the target speech.

[0007] Patent publication number CN112331183B discloses a non-parallel corpus speech conversion method and system based on an autoregressive network. The method includes: phoneme delay probability extraction, extracting phoneme delay probability features from the input speech; capturing contextual information in the phoneme delay probability features to obtain a text feature representation that integrates the contextual information; using an adaptive attention mechanism to fuse the text features at the current moment with the acoustic features at the previous moment to obtain an augmented feature representation; using a long short-term memory network to predict the acoustic features of the target speaker, and finally using a vocoder to synthesize speech.

[0008] The above three patents all belong to speech conversion methods in intelligent speech. Among them, the CN118969023A patent focuses on using a method combining attention mechanism and stream matching to improve the problem of poor speech conversion effect. It uses a feature coupling module based on the attention mechanism to learn the correspondence between the corresponding pitch features and content features, and then uses the Mel spectrum model to calculate the Mel spectrum to obtain the Mel spectrum features. The CN119274565A patent focuses on robustness enhancement and constructs a complete pipeline including data enhancement-feature alignment-joint decoding. This solution extracts phoneme-level spectrum and semantic features respectively through the prosody encoder and content encoder, uses the attention mechanism to achieve cross-modal feature alignment, and finally combines the voiceprint features with the set timbre vector to complete high-fidelity speech synthesis, and achieves accurate target speech output through time series interception. The CN112331183B patent uses an autoregressive network and an adaptive attention mechanism to achieve the conversion of non-parallel corpus.

[0009] While these existing technologies have made some progress in improving speech conversion, they fail to fully consider the balance between converted speech similarity and content accuracy, and to some extent ignore other vocal features of the speaker besides timbre. This, to a certain extent, limits the accuracy and robustness of the speech conversion model. To address this issue, the present invention proposes a method for arbitrary speech conversion based on attention feature fusion. This method uses temporally normalized attention fusion to dynamically integrate local phoneme details with global acoustic features in the temporal dimension, completing self-acoustic transfer and ensuring speaker feature similarity, content feature integrity, and the prosodic characteristics of the target speaker. The present inventors have designed a speech conversion method that is different from the aforementioned patents, and a search has found no similar patent documents. Summary of the Invention

[0010] The purpose of the present invention is to provide a non-parallel arbitrary-to-arbitrary speech conversion method based on attention feature fusion. To address the problem that existing deep learning methods cannot simultaneously take into account both content integrity and speaker feature similarity, the present invention uses an attention fusion mechanism combined with efficient multi-scale attention as the backbone neural network to improve the quality of synthesized speech.

[0011] The technical solution of the present invention is:

[0012] A non-parallel any-to-any speech conversion method based on attention feature fusion, i.e., a speech conversion model for any-to-any speech conversion, comprises: first, extracting content features, speaker features, and rhythm features from the original and target speech, respectively; coupling the extracted features using a bottleneck layer consisting of time-series instance normalization attention to obtain a feature map; combining time-series instance normalization with an efficient channel attention mechanism to dynamically integrate local phoneme details and global acoustic features in the time-frequency domain to achieve adaptive acoustic migration; finally, generating high-quality speech through a neural vocoder, and enhancing adaptability to noise and non-parallel data through instance normalization and twin loss; splitting the training dataset into a certain proportion, performing preprocessing, and using the dataset for training; the method comprises the following steps:

[0013] Step 1: Obtain a speech dataset and preprocess each speech in it;

[0014] Step 2: Build a speech conversion model, which includes:

[0015] The feature extraction module is used to extract the corresponding speech content, rhythm, pitch, timbre and other speech component features of the original speech and the target speech;

[0016] The bottleneck layer consists of time-series instance normalized attention fusion and a threshold unit, which is used to combine the speaker characteristics and prosody characteristics of the target speaker with the speaker characteristics of the original speaker;

[0017] The EMAFF module, consisting of time-series instance normalized attention fusion and efficient channel attention, dynamically integrates the local phoneme details and global acoustic features of the original and target speech in the time-frequency domain to achieve acoustic transfer;

[0018] Vocoder, generates the final speech based on the feature map;

[0019] Step 3: Use the dataset to train the model;

[0020] Step 4: Use the trained model to perform performance testing on the test set.

[0021] More specific steps are:

[0022] Step 1: Obtain the VCTK speech dataset, preprocess all audio, and reduce the sampling rate to 16kHz; the dataset is used for training, validation, and test sets in a ratio of 3:1:1. 20 speakers are selected, and 600 utterances are generated for "seen-to-seen" scenarios, "seen-to-unseen" scenarios, and "unseen-to-unseen" scenarios to evaluate the model's ability to learn examples from the same distribution it was trained on, and its ability to learn examples from the same distribution it was trained on;

[0023] Step 2: Construct a speech conversion model for arbitrary speech conversion, which mainly includes: a feature extraction module, a bottleneck layer, an EMAFF module and a vocoder; the feature extraction module is divided into two parts: using the fine-tuned pre-trained model wav2vec2.0 to extract the comparative predictive coding CPC of the original speech and the target speech as content features and speaker features; using the distributed inline filtering algorithm with overlap to extract the logarithmic fundamental frequency f0 of the target speech to represent its pitch and fundamental frequency.

[0024] Step 3: The CPC features of the extracted target speech, f0 and CPC features of the original speech are passed through the bottleneck layer to obtain a new feature map, which is then used as input and fed into the EMAFF module composed of time-series instance normalized attention fusion and efficient multi-scale attention to obtain the converted feature map; finally, the converted speech is generated through the vocoder. The loss function used in this training method is composed of two sets of loss functions, namely reconstruction loss and twin loss; for the reconstruction loss, the L1 loss function is used to compare the Mel spectrum extracted from the original audio. Mel-spectrogram with model prediction For the twin loss, after the input x is enhanced by temporal masking, the L1 loss function is used to compare the original Mel spectrogram y with the enhanced Mel spectrogram predicted by the model. The combined loss function is as follows:

[0025]

[0026] in

[0027] Step 4: The generated converted speech in the test set is evaluated objectively and subjectively for content accuracy, similarity, and naturalness. The objective evaluations include word error rate (WER) and character error rate (EER) to assess content accuracy; speaker similarity verification to assess speaker similarity; and naturalness using the NISQA model to predict mean opinion scores (MOS). NISQA is a deep learning model for speech quality prediction. Its model weights are used to predict the quality of speech samples sent through a communication system, ranging from 1 (worst) to 5 (best). The subjective evaluation follows the requirements of the Voice Conversion Challenge (VGG). Twenty listeners randomly listen to the generated speech and provide subjective scores. Subjective scores are categorized as: different, very certain; different, uncertain; identical, uncertain; identical, very certain.

[0028] The present invention has the following characteristics:

[0029] 1. This invention improves attention feature fusion and regularizes it in the time dimension based on the characteristics of speech data, encouraging the extraction of more detailed speaker features during model training and improving the learning performance of the model.

[0030] 2. The present invention adopts a dual-channel encoder to extract speech content features (CPC features) and speaker features (fundamental frequency f0) respectively, and uses an attention-based feature fusion module to enable the model to retain the original speech content while taking into account other features such as timbre and pitch, thereby improving the similarity of the speech generated by the model.

[0031] 3. During the training process, the present invention divides the data set into three different transformation scenarios to improve the generalization of the model.

[0032] 4. The present invention uses reduced learning in the training process of the speech conversion model. The data in the training set is reduced when training the model, and the reconstruction loss (L1 loss) and twin loss are combined to improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the voice conversion method of the present invention;

[0034] Figure 2 It is the specific structure of the present invention;

[0035] Figure 3 It is the temporal instance normalized attention feature fusion TInAFF module structure of the present invention;

[0036] Figure 4 It is the structural flow of the EMAFF module of the present invention;

[0037] Figure 5 is the MOS distribution of the subjective score of the generated speech;

[0038] Figure 6 is the distribution of subjective similarity scores of the generated speech according to the voice conversion challenge (VGG). DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0040] See also Figure 1-4 A non-parallel any-to-any speech conversion method based on attention feature fusion is proposed. By fusing improved time-series instance-normalized attention feature fusion for speech sequence data with efficient multi-scale attention, any-to-any speech conversion is achieved. The method includes the following steps:

[0041] Step 1: Create the required dataset.

[0042] Step 2: Preprocess and extract features of the original speech and target speech respectively

[0043] Step 3: Use the dataset to train the model;

[0044] Step 4: Use the trained model to perform performance testing on the test set and compare its performance with AUTO-VC, AGAIN-VC, and VQVC+, which were also trained using the VCTK dataset.

[0045] The specific steps are:

[0046] Step 1. Obtain the VCTK speech dataset, preprocess all audio, and reduce the sampling rate to 16kHz. The dataset is used for training, validation, and test sets in a ratio of 3:1:1. 20 speakers are selected, and 600 "seen-to-seen", "seen-to-unseen" and "unseen-to-unseen" utterances are generated to evaluate the model's ability to learn examples from the same distribution it was trained on, and its ability to learn examples from the same distribution it was trained on;

[0047] Step 2: Use the fine-tuned pre-trained model wav2vec2.0 to extract the comparative predictive coding (CPC) of the original and target speech as content features and speaker features; use the distributed inline filtering algorithm with overlap to extract the logarithmic fundamental frequency (f0) of the target speech to represent its pitch and fundamental frequency;

[0048] Step 3: The original speech features and target speech features extracted in step 2 are input into the bottleneck layer composed of time series instance normalized attention feature fusion to further extract features in the time series dimension. The structure is as follows: Figure 3 As shown. For a given input Two attention feature maps S g and S l Generated by the following formula:

[0049] S g =σ(Conv1d(SiLU(T(Conv1d(GAP(X))))))

[0050] S l =σ(Conv1d(SiLU(T(Conv1d(X)))))

[0051] Among them, GAP is a one-dimensional global average pooling (Global Average Pooling) along the channel dimension to extract global information; SiLU is a Sigmoid linear unit (Sigmoid-Weighted Linear Unit); σ represents the Sigmoid function; T represents the instance normalization of the time step, that is, the normalization of the features in the time dimension. The final attention map Then there is S g With S l Combination of:

[0052]

[0053] in Represents the broadcast dot product.

[0054] The two source features X and Y are first added together and fed into TInAFF as input. The final attention map S is used as the fusion weight. Considering that only features from one side need to be transferred to the other side during the conversion process to reduce redundant calculations, the final fused feature Z is obtained:

[0055]

[0056] The obtained fusion feature Z is used as input together with the original speech feature and the target speech feature, and then the EMAFF module is constructed by combining the sequential instance normalized attention feature fusion and the efficient multi-scale attention (structure as shown in the figure). Figure 4 to generate synthesized speech.

[0057] The loss functions used for training are reconstruction loss and twin loss. For reconstruction loss, the L1 loss function is used to compare the Mel spectrogram extracted from the original audio. Mel-spectrogram with model prediction For the twin loss, after the input x is enhanced by temporal masking, the L1 loss function is used to compare the original Mel spectrogram y with the enhanced Mel spectrogram predicted by the model. The combined loss function is as follows:

[0058]

[0059] where loss(y,y x )=||yy x ||1 / T, through y siam The calculated additional loss is used to improve the robustness and consistency of the model in the temporal masking task and remove content information during training. This method dynamically masks local abnormal regions in the time series and combines it with a contrastive learning strategy to suppress abnormal deviations while retaining valid temporal features, thereby enhancing the model's adaptability to temporal masking operations.

[0060] Step 4: For the converted speech generated by the test set, objective and subjective evaluations are performed on the three aspects of content accuracy, similarity, and naturalness. The objective evaluations are word error rate and character error rate to evaluate content accuracy, similarity speaker verification to evaluate speaker similarity, and NISQA model to evaluate naturalness. The subjective evaluation adopts the requirements of the voice conversion challenge (VGG) speech conversion challenge, and 20 listeners are asked to randomly listen to the generated speech and give subjective evaluation scores. The present invention combines the characteristics of speech data and the scenarios where parallel data training is not required for arbitrary conversion, improves the attention feature fusion, and modifies it into a time-series instance normalized attention feature fusion that is more in line with the characteristics of speech data, and combines it with efficient multi-scale attention to generate an EMAFF module, so that the model can learn more original speaker speech features.

[0061] As shown in Table 1, the proposed IAFF-VC model achieves a WER of 18.02% in the S2S scenario, 19.45% in the S2U scenario, and 20.7% in the U2U scenario, with an average of 19.39%. The CER reaches 9.34% in the S2S scenario, 9.98% in the S2U scenario, and 10.17% in the U2U scenario, with an average of 9.83%. Table 2 shows that in terms of similarity, IAFF-VC achieves 94.17%, 91.24%, 89.34%, and 91.58% in the S2S, S2U, U2U, and average count scenarios, respectively. For naturalness, IAFF-VC achieves a score of 3.4, demonstrating the best average performance. However, its score in the U2U scenario is slightly lower than that of the AUTO-VC model. The proposed model demonstrates superiority over other methods in balancing content and speaker characteristics, with minimal impact from changing scenarios. In terms of preserving content integrity, this method achieved the lowest WER and CER in S2S, S2U, and U2U scenarios, significantly outperforming other methods. It also achieved the best performance on similarity metrics, with approximately 90% of sample results exceeding the threshold; it also performed well on naturalness. In contrast, while AUTO-VC performed well in terms of content preservation and naturalness, it was inferior to our method on similarity metrics. The results demonstrate that our method can effectively preserve both the original content and the characteristics of the target speaker.

[0062] Table 1 Comparison of the synthesis results of the proposed model and other models in terms of content completeness

[0063]

[0064] Table 2 Comparison of the synthesis results of the proposed model and other models in terms of similarity and naturalness

[0065]

[0066] Subjective evaluation results such as Figure 5 and Figure 6 As shown, Figure 5 The mean opinion score (MOS) of the model in S2S, S2U and U2U scenarios is shown. Figure 6 The results of the VGG subjective similarity test are presented, along with a reference speech (oracle) reconstructed by a vocoder. Consistent with the objective evaluation results, both the MOS and VGG tests show that IAFF-VC performs better in similarity metrics, with both similarity and naturalness close to the reference speech level. Furthermore, IAFF-VC significantly outperforms existing methods in similarity and naturalness evaluation, indicating that the proposed model can generate relatively unbiased results.

[0067] The present invention achieves a closed-loop decoupling, fusion, and reconstruction of content and speaker characteristics in non-parallel any-to-any speech conversion through dual encoder decoupling, temporal alignment, and multi-scale attention. Semantic integrity is ensured through CPC feature extraction and improved temporal instance normalization attention feature fusion, dynamic acoustic migration is achieved through conditional attention, and instance normalization and twin loss enhance adaptability to noise and non-parallel data. Experimental results show that, when trained on the VCTK dataset, the present invention achieves a WER of 18.02% in the S2S scenario, 19.45% in the S2U scenario, and 20.7% in the U2U scenario, with an average of 19.39%. The CER reaches 9.34% in the S2S scenario, 9.98% in the S2U scenario, and 10.17% in the U2U scenario, with an average of 9.83%. In terms of similarity, IAFF-VC achieved scores of 94.17%, 91.24%, 89.34%, and 91.58% in S2S, S2U, U2U, and average count, respectively. For naturalness, IAFF-VC scored 3.4. In subjective evaluation, IAFF-VC also achieved impressive results in S2S, S2U, and U2U scenarios.

[0068] In summary, the present invention is based on an encoder-decoder structure and combined with attention feature fusion to retain the original speaker's speech features as much as possible while preserving the original semantics, and can realize arbitrary speech conversion in different S2S, S2U and U2U scenarios.

[0069] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A non-parallel any-to-any speech conversion method based on attention feature fusion, characterized by: Firstly, the content features, speaker features and rhythm features of the original speech and the target speech are extracted respectively; A bottleneck layer consisting of time-series instance normalization and attention is used to couple the extracted features to obtain a feature map. Time-series instance normalization is combined with an efficient channel attention mechanism to dynamically integrate local phoneme details and global acoustic features in the time-frequency domain to achieve adaptive acoustic migration. Finally, high-quality speech is generated through a neural vocoder. Instance normalization and twin loss enhance adaptability to noisy and non-parallel data. The training dataset is split according to a certain ratio, preprocessed, and then used for training. The following steps are included: Step 1: Obtain a speech dataset and preprocess each speech in it; Step 2: Build a speech conversion model, which includes: The feature extraction module is used to extract the corresponding speech content, rhythm, pitch, timbre and other speech component features of the original speech and the target speech; The bottleneck layer consists of time-series instance normalized attention fusion and a threshold unit, which is used to combine the speaker characteristics and prosody characteristics of the target speaker with the speaker characteristics of the original speaker; The EMAFF module, consisting of time-series instance normalized attention fusion and efficient channel attention, dynamically integrates the local phoneme details and global acoustic features of the original and target speech in the time-frequency domain to achieve acoustic transfer; Vocoder, generates the final speech based on the feature map; Step 3: Use the dataset to train the model; Step 4: Use the trained model to perform performance testing on the test set; Specifically, step 1 involves obtaining a VCTK speech dataset, preprocessing all audio, and reducing the sampling rate to 16kHz; using the dataset for training, validation, and test sets in a 3:1:1 ratio; selecting 20 speakers and generating 600 utterances for "speaker visible" scenarios, "speaker visible to invisible" scenarios, and "speaker invisible" scenarios to evaluate the model's ability to learn examples from the same distribution on which it was trained; The feature extraction module of step 2 is divided into two parts: using the fine-tuned pre-trained model wav2vec2.0 to extract the comparative predictive coding CPC of the original speech and the target speech as content features and speaker features; using the distributed inline filtering algorithm with overlap to extract the logarithmic fundamental frequency f0 of the target speech to represent its pitch and fundamental frequency; The loss functions of step 3 are reconstruction loss and twin loss respectively; for reconstruction loss, the L1 loss function is used to compare the Mel spectrogram extracted from the original audio. Mel-spectrogram with model prediction For the twin loss, after the input x is enhanced by temporal masking, the L1 loss function is used to compare the original Mel spectrogram y with the enhanced Mel spectrogram predicted by the model. ; The combined loss function is as follows: ； in .

2. The non-parallel any-to-any speech conversion method based on attention feature fusion according to claim 1, characterized in that: In step 3, the extracted CPC features of the target speech, f0, and the CPC features of the original speech are passed through the bottleneck layer to obtain a new feature map, which is then used as input and enters the EMAFF module composed of time-series instance normalized attention fusion and efficient multi-scale attention to obtain the converted feature map.

3. The non-parallel any-to-any speech conversion method based on attention feature fusion according to claim 1, characterized in that: Specifically, step 4 includes objectively and subjectively evaluating the converted speech generated in the test set in terms of content accuracy, similarity, and naturalness. The objective evaluations include word error rate (WER) and character error rate (EER) to evaluate content accuracy; similarity speaker verification to evaluate speaker similarity; naturalness to evaluate the NISQA model, which is used to predict the mean opinion score (MOS). NISQA is a deep learning model for speech quality prediction. Its model weights can be used to predict the quality of speech samples sent through a communication system, ranging from "1" for the worst to "5" for the best. The subjective evaluation uses the requirements of the VGG speech conversion challenge, in which 20 listeners randomly listen to the generated speech and give subjective evaluation scores. The evaluation level is divided into four levels to judge similarity.

Images