A speech conversion method and device based on attention mechanism and stream matching strategy

Abstract

The present application provides a method and device for voice conversion based on an attention mechanism and a stream matching strategy, which effectively solves the problem of poor conversion effect existing in most voice conversion methods. The method includes: the feature extraction module extracts features from the acquired audio to be converted to obtain audio features; the feature coupling module learns the correspondence between the pitch features, emotional features and content features based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker; the stream matching module performs a Mel spectrum calculation on the latent vector based on a trained Mel spectrum model to obtain a Mel spectrum feature, so as to output the Mel spectrum feature to a voice code output module; the voice code output module processes the Mel spectrum feature that meets the preset decoding conditions to obtain a processed Mel spectrum feature, so as to obtain the audio after identity conversion based on the processed Mel spectrum feature.

Description

Technical Field

[0001] The present application relates to the field of speech conversion technology, and in particular to a speech conversion method and device based on an attention mechanism and a stream matching strategy. Background Art

[0002] The research on speech conversion is to convert a speech segment of one speaker (source speech) into a speech segment with a similar pronunciation to that of another speaker (target speaker) without changing the speech content. Speech conversion can be regarded as style transfer in the field of speech. It transfers the timbre of the target speaker to the source speech, so that the timbre between the input speech and the output speech changes while the content remains the same. Speech conversion was once classified as a branch of speech synthesis. The main tasks of speech synthesis include text-to-speech synthesis and synthesis of changing speech attributes (such as speaker identity, speaking emotion, etc.). The research focus of speech conversion is to change the speaker identity information of the speech.

[0003] The mainstream conversion system framework requires a certain amount of source and target speaker speech data to be obtained in advance, and then the conversion is completed in two stages. In the training stage, the acquired speech data is used to learn the relationship between the source and target speech features to establish a conversion model; in the conversion stage, the obtained conversion model is used to convert the newly input source speaker data. At present, most of the sound conversion methods are based on this framework to complete the conversion. According to the different conversion relationship modeling methods, these methods can be mainly divided into three categories. The first category is based on vector quantization. The sound conversion method based on vector quantization represents the feature space of the source and target as a set of codebooks respectively. The mapping relationship between the source and target codebooks is learned in the training stage. When converting, the codeword corresponding to the input source speaker's feature is first found, and then the conversion is performed according to the codebook mapping relationship. The final generated conversion spectrum is a linear combination of the target codewords. A basic problem with this method is that the discrete representation of the feature space leads to poor continuity of the generated converted speech and insufficient conversion effect. The second category is based on the Gaussian mixture model. Compared with the method based on vector quantization, this type of method uses this continuous probability model to represent the feature space, which improves the robustness of the system and the sound quality of the converted speech. However, this modeling method still has some problems that affect the conversion speech effect. These problems mainly include the following aspects: the conversion process does not consider the correlation between consecutive frames, the features of each frame are mapped independently, the model is overfitted, and the conversion spectrum is too smooth, which makes the converted speech sound dull. The third category is based on frequency regularization. The idea of ​​this type of algorithm is to use a regularization function to directly adjust the spectrum of the source speaker so that the distance between the regularized spectrum and the target spectrum is minimized. Relatively speaking, the method based on frequency regularization directly adjusts the spectrum, retains the detailed information of the natural spectrum, and can obtain better conversion sound quality. However, this amplitude scaling technology can only work under the premise of accurate spectrum adjustment. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a speech conversion method and device based on an attention mechanism and a stream matching strategy. This speech conversion method and device based on an attention mechanism and a stream matching strategy effectively solves the problem of poor conversion effect existing in most sound conversion methods.

[0005] In a first aspect, an embodiment of the present application provides a speech conversion method based on an attention mechanism and a stream matching strategy, which is applicable to a speech conversion system, wherein the speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module, and a vocoding output module, and the method includes:

[0006] The feature extraction module extracts features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features;

[0007] The feature coupling module learns the correspondence between the pitch feature, the emotion feature and the content feature based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity feature of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features;

[0008] The stream matching module performs Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features;

[0009] The voice coding output module processes the Mel-spectrogram features that meet the preset decoding conditions to obtain processed Mel-spectrogram features, so as to obtain the identity-converted audio based on the processed Mel-spectrogram features.

[0010] In combination with the first aspect, an embodiment of the present application provides a first possible implementation of the first aspect, wherein the voice coding output module processes the Mel spectrum feature that meets the preset decoding condition to obtain the processed Mel spectrum feature, and obtains the identity-converted audio based on the processed Mel spectrum feature, including:

[0011] The voice coding output module performs time correlation and channel correlation learning on the obtained Mel spectrum features that meet the preset decoding conditions to obtain the learned Mel spectrum features;

[0012] The voice code output module identifies the learned Mel spectrum features to obtain the audio after identity conversion.

[0013] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the specific steps of the voice coding output module obtaining the mel spectrum features that meet the preset decoding conditions include:

[0014] The vocoding output module determines whether the Mel spectrum feature meets the optimization target of the loss function in the stream matching module;

[0015] If it meets the requirements, the voice coding output module obtains the Mel spectrum features that meet the preset decoding conditions.

[0016] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the specific steps of continuously training the mel spectrum model with mel spectrum features include:

[0017] The loss function is used to calculate the difference between the Mel spectrum feature and the actual Mel spectrum feature, and the parameters of the Mel spectrum model are updated with the difference; the actual Mel spectrum feature is extracted from the audio to be converted;

[0018] Repeat the above steps until the difference between the mel spectrum feature output by the mel spectrum model and the actual mel spectrum feature is lower than the preset threshold.

[0019] In combination with the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein the feature coupling module learns the correspondence between the tone features, the emotion features, and the content features based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker, including:

[0020] The feature coupling module extracts and learns the correspondence between the tone features, emotion features and content features based on the attention mechanism;

[0021] The feature coupling module couples the corresponding relationship with the identity feature of the target speaker to obtain a latent vector.

[0022] In combination with the first aspect, the embodiment of the present application provides a fifth possible implementation of the first aspect, wherein the feature extraction module extracts features from the acquired audio to be converted to obtain audio features, including:

[0023] The feature extraction module outputs the audio to be converted to multiple extractors in the feature extraction module;

[0024] The feature extraction module calls a plurality of preset extractors to extract the audio to be converted respectively, and obtains audio features of multiple dimensions.

[0025] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the feature extraction module calls a plurality of preset extractors to extract the audio to be converted respectively to obtain audio features of multiple dimensions, including:

[0026] The content extractor among the plurality of extractors encodes the audio to be converted into continuous encoded audio through an encoder;

[0027] The content extractor converts the continuous encoded audio into content features of a corresponding discrete unit sequence through a clusterer.

[0028] In a second aspect, an embodiment of the present application provides a speech conversion device based on an attention mechanism and a stream matching strategy, which is applicable to a speech conversion system. The speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a voice coding output module. The device includes:

[0029] An extraction module, used for the feature extraction module to extract features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features;

[0030] A learning module, used for the feature coupling module to learn the correspondence between the pitch feature, the emotion feature and the content feature based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity feature of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features;

[0031] A calculation module, used for the stream matching module to perform Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the vocoding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features;

[0032] The conversion module is used for the voice coding output module to process the Mel spectrum features that meet the preset decoding conditions to obtain the processed Mel spectrum features, so as to obtain the audio after identity conversion based on the processed Mel spectrum features.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, any step of a speech conversion method based on an attention mechanism and a stream matching strategy is performed.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes any one of the steps of a speech conversion method based on an attention mechanism and a stream matching strategy.

[0035] The embodiment of the present application provides a speech conversion method based on an attention mechanism and a stream matching strategy, which is applicable to a speech conversion system. The speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a voice code output module. The method first extracts features from the acquired audio to be converted by the feature extraction module to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include pitch features, emotional features and content features; secondly, the feature coupling module learns the correspondence between the pitch features, emotional features and content features based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker; the latent vector is a numerical composition corresponding to multiple audio features matrix; then the stream matching module performs Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features; finally, the voice coding output module processes the Mel spectrum features that meet the preset decoding conditions to obtain the processed Mel spectrum features, so as to obtain the audio after identity conversion based on the processed Mel spectrum features, based on the speech conversion system including the feature extraction module, the feature coupling module, the stream matching module and the voice coding output module, the effect of converting the identity features of the audio to be converted is achieved, the problems existing in the existing sound conversion method are avoided, and the emotion and tone of the audio after identity conversion are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a flow chart of a first speech conversion method based on an attention mechanism and a stream matching strategy provided in an embodiment of the present application is shown;

[0038] Figure 2 A training schematic diagram of a speech conversion system provided in each embodiment of the present application is shown;

[0039] Figure 3 A schematic diagram of audio conversion to be converted provided by an embodiment of the present application is shown;

[0040] Figure 4 A schematic diagram of the structure of a first speech conversion device based on an attention mechanism and a stream matching strategy provided in an embodiment of the present application is shown;

[0041] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0043] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0044] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0045] The current sound conversion modeling methods can be mainly divided into three categories. The first category is based on vector quantization. The sound conversion method based on vector quantization is to represent the feature space of the source and target as a set of codebooks respectively. The mapping relationship between the source and target codebooks is learned in the training stage. When converting, the codeword corresponding to the input source speaker's feature is first found, and then the conversion is performed according to the codebook mapping relationship. The final generated conversion spectrum is a linear combination of the target codewords. A basic problem with this method is that the discrete representation of the feature space leads to poor continuity of the generated conversion speech and insufficient conversion effect; the second category is based on Gaussian mixture model. Compared with the method based on vector quantization, this type of method uses this continuous probability model to represent the feature space, which improves the robustness of the system and the sound quality of the converted speech. However, this modeling method still has some problems that affect the conversion speech effect. These problems mainly include the following aspects: the conversion process does not consider the correlation between consecutive frames, each frame feature is independently mapped, the model is overfitting, and the conversion spectrum is too smooth, resulting in a dull sound of the converted speech; the third category is based on frequency regularization. The idea of ​​this type of algorithm is to use a regularization function to directly adjust the spectrum of the source speaker so that the distance between the regularized spectrum and the target spectrum is minimized. In contrast, the frequency regularization method directly adjusts the spectrum, retains the details of the natural spectrum, and can obtain better converted sound quality. However, this amplitude scaling technology can only work under the premise of accurate spectrum adjustment.

[0046] Based on this, the embodiments of the present application provide a speech conversion method and device based on an attention mechanism and a stream matching strategy, which are described below through embodiments.

[0047] Example 1

[0048] To facilitate understanding of this embodiment, a speech conversion method based on an attention mechanism and a stream matching strategy disclosed in an embodiment of this application is first described in detail. Figure 1 The flowchart of a speech conversion method based on an attention mechanism and a stream matching strategy is shown in the figure. The speech conversion method based on an attention mechanism and a stream matching strategy provided in the present application is applicable to a speech conversion system. The speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a vocoding output module. The method includes:

[0049] S101, the feature extraction module extracts features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features;

[0050] S102, the feature coupling module learns the correspondence between the tone features, the emotion features and the content features based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features;

[0051] S103, the stream matching module performs Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, and outputs the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features;

[0052] S104, the voice coding output module processes the Mel-spectrogram features that meet the preset decoding conditions to obtain processed Mel-spectrogram features, and obtains the audio after identity conversion based on the processed Mel-spectrogram features.

[0053] The speech conversion system is pre-trained, such as Figure 2 As shown, the training method is: first, collect audios of several speakers and target speakers, and collect several high-quality, multi-scene, and multi-speaker audios to train and improve the sound output module, use the emotion2vec model to extract the emotional features of the audios of several speakers and target speakers respectively, specifically by encoding them with speaker encoder and emotion encoder, and use the Hubert model to extract content features and pitch features, specifically by encoding them with pitch encoder and content encoder, and finally use the collected audios of several speakers and target speakers, the extracted emotional features, content features, and pitch features, and train to obtain a speech conversion system.

[0054] In step S101, the pre-acquired audio to be converted is output to the speech conversion system. After receiving the audio to be converted, the feature extraction module in the speech conversion system extracts features of the acquired audio to be converted in a predefined manner, such as encoding by speaker emcoder, so as to obtain audio features of the audio to be converted. The audio features at this time are encoded index sequences. In order to avoid an increase in the amount of calculation, adjacent repeated index sequences are removed to obtain a discrete unit sequence. The audio features are the characteristics that best represent the audio to be converted. The audio features extracted by the feature extraction module include tone features, emotion features and content features. The tone features are the tone of the speaker in the extracted audio to be converted, such as high or low, the emotion features are the emotions of the speaker in the extracted audio to be converted, such as happy or sad, and the content features are the content expressed by the speaker in the extracted audio to be converted.

[0055] In the specific implementation process of step S101, there is an embodiment in which the feature extraction module extracts features from the acquired audio to be converted to obtain audio features, including:

[0056] S1011, the feature extraction module outputs the audio to be converted to multiple extractors in the feature extraction module;

[0057] S1012: The feature extraction module calls a plurality of preset extractors to extract the audio to be converted respectively, and obtains audio features of multiple dimensions.

[0058] In steps S1011-S1012, there are multiple extractors in the feature extraction module, and different extractors can extract different features of the audio to be converted. For example, for pitch features, a pitch extractor is provided, which is specifically extracted by encoding with a pitch emcoder; for emotion features, an emotion extractor is provided, which is specifically extracted by encoding with an emotion emcoder; for content features, a content extractor is provided, which is specifically extracted by encoding with a content emcoder. The feature extraction module outputs the audio to be converted to multiple extractors respectively, and the multiple extractors extract the audio to be converted in parallel according to a predefined method, that is, the effect of simultaneously obtaining audio features of multiple dimensions is achieved, and the multiple dimensions include pitch, content, and emotion, etc. The feature extraction module integrates the extracted multiple dimensions so that the obtained audio features have multiple dimensions.

[0059] In the specific implementation process of step S10121, there is an embodiment in which the feature extraction module calls a plurality of preset extractors to extract the audio to be converted respectively to obtain audio features of multiple dimensions, including:

[0060] S10121. A content extractor among the multiple extractors encodes the audio to be converted into continuous encoded audio through an encoder;

[0061] S10122. The content extractor converts the continuous encoded audio into content features of a corresponding discrete unit sequence through a clusterer.

[0062] In steps S10121-S10122, the multiple extractors in the feature extraction module of the present application extract features in different ways. For example, the content extractor among the multiple extractors in the feature extraction module has an encoder and a clusterer, that is, the content extractor is a pre-trained HuBERT self-supervised model, and the clusterer is a pre-trained HuBERT self-supervised model, which can be a trained k-means clusterer. The encoder encodes the speech to be converted into continuous encoded audio, and then converts the continuous encoded audio into a series of clustering index sequences through the clusterer, and removes adjacent repeated index sequences to obtain a discrete unit sequence, and then uses a BERT-like MLM self-supervised pre-training method to enable the model to predict the target value of the content feature through the masked speech signal. HuBERT not only achieves the same performance as wav2vec in the ASR task, but also achieves the same performance as wav2vec in the ASR task. 2.0, and more importantly, it also shows good versatility and potential in more non-ASR tasks (such as speaker recognition, intent recognition, emotion recognition, etc.). The HuBERT model is trained in advance by an unsupervised method to obtain a clustering model, which generates a discretized target sequence for all unlabeled speech signals. The clusterer determines in advance the cluster center sequence corresponding to the audio to be converted, that is, the discretized unit sequence, and the total number of cluster centers, and then uses the cross entropy loss function to pre-train the clusterer. Since the discrete unit sequence provided by the clusterer contains not only vocabulary information related to ASR, but also other information such as speakers and emotions, the pre-trained HuBERT model can learn richer multi-faceted speech information, thereby showing stronger generalization performance in a variety of downstream speech processing tasks.

[0063] As described above, the emotion extractor among the multiple extractors in the feature extraction module is also pre-trained, and is implemented using the pre-trained emotion2vec emotion representation model, which is trained by self-supervised pre-training on 262 hours of open source emotion data and using an online distillation paradigm. Given that both global information and local details in audio can express emotions, a pre-training strategy combining sentence-level loss and frame-level loss is proposed to better capture emotional information. This strategy can help the model learn richer and more accurate emotional features, thereby improving the performance of extracting various emotional features.

[0064] In step S102, the feature coupling module is responsible for effectively integrating the emotional features, content features and tone features in the audio features, such as Figure 3As shown, in order to generate speech that conforms to the target speaker, the feature coupling learns the correspondence between the pitch features, emotional features and content features based on the attention mechanism, and ignores information other than the pitch features, emotional features and content features and the correspondence between the three, and dynamically adjusts the weights of these features in the process of generating speech. The attention mechanism is the Attention Based Fusion Model. For example, when expressing strong emotions, the weight of emotional features may increase, and when emphasizing specific words, the weight of content features may increase, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features; the audio features are all expressed in numerical values, that is, the numerical values ​​corresponding to multiple audio features are weighted by the attention mechanism.

[0065] In the specific implementation process of step S102, there is an embodiment in which the feature coupling module learns the correspondence between the tone features, the emotion features and the content features based on the attention mechanism to obtain a latent vector based on the correspondence and the preset identity features of the target speaker, including:

[0066] S1021, the feature coupling module extracts and learns the corresponding relationship between the tone feature, the emotion feature and the content feature based on the attention mechanism;

[0067] S1022: The feature coupling module couples the corresponding relationship with the identity feature of the target speaker to obtain a latent vector.

[0068] In steps S1021-S1022, the feature coupling module extracts the correspondence between the tone features, emotional features and content features based on the attention mechanism. For example, when you are happy, the value corresponding to the tone features will increase, the value corresponding to the emotional features will also increase, and the content in the content features will be slightly more interesting and flexible, and learns to extract the correspondence between the tone features, emotional features and content features. The feature coupling module is coupled with the identity features of the target speaker based on the correspondence, that is, attention weights are assigned to the four features of tone features, emotional features, content features and the identity features of the target speaker through the attention mechanism, and a latent vector including multiple audio feature values ​​is calculated based on the attention weights corresponding to the four features of tone features, emotional features, content features and the identity features of the target speaker and the values ​​corresponding to the tone features, emotional features, content features and the identity features of the target speaker.

[0069] In step S103, after receiving the latent vector output by the feature coupling module, the flow matching module performs Mel-spectrogram calculation on the latent vector and the attention weights corresponding to the four features of the pitch feature, the emotion feature, the content feature and the identity feature of the target speaker contained in the latent vector based on the trained Mel-spectrogram model to obtain the Mel-spectrogram feature. The flow matching module is Flow matching. The Mel-spectrogram model is usually a neural network, which may include multiple layers, such as a fully connected layer, a convolutional layer, etc., to capture the complex features in the latent vector and convert them into Mel-spectrogram representations. Through the Mel-spectrogram model, the latent vector output by the feature coupling module is converted into a series of Mel-spectrogram frames. Each Mel-spectrogram frame is a representation of the frequency distribution of the audio signal in a specific time window, but is converted by Mel Scale to better match the auditory characteristics of the human ear, and the Mel-spectrogram feature is output to the voice code output module; the Mel-spectrogram model is obtained by continuously training the Mel-spectrogram feature to obtain the most suitable Mel-spectrogram model.

[0070] In the specific implementation process of step S103, there is an embodiment in which the specific steps of continuously training the mel spectrum model with mel spectrum features include:

[0071] S1031, using a loss function to calculate the difference between the Mel-spectrogram feature and the actual Mel-spectrogram feature, and using the difference to update the parameters of the Mel-spectrogram model; the actual Mel-spectrogram feature is extracted from the audio to be converted;

[0072] S1032, repeat the above steps until the difference between the mel spectrum feature output by the mel spectrum model and the actual mel spectrum feature is lower than a preset threshold.

[0073] In steps S1031-S1032, in order to obtain the best or most suitable Mel-spectrogram model, the stream matching module sets a loss function for the Mel-spectrogram model, and calculates the difference between the Mel-spectrogram feature calculated by the stream matching module and the actual Mel-spectrogram feature, where the actual Mel-spectrogram feature is extracted from the audio to be converted; if the difference between the Mel-spectrogram feature calculated by the stream matching module and the actual Mel-spectrogram feature is large, the loss function will optimize the Mel-spectrogram model again based on the difference, and calculate the difference between the Mel-spectrogram feature calculated by the stream matching module and the actual Mel-spectrogram feature again, and repeat the above steps until the difference between the Mel-spectrogram feature calculated by the stream matching module and the actual Mel-spectrogram feature becomes smaller and smaller, that is, the difference between the Mel-spectrogram feature output by the Mel-spectrogram model and the actual Mel-spectrogram feature is lower than a preset threshold, then the Mel-spectrogram feature output by the Mel-spectrogram model is closer to or even consistent with the actual Mel-spectrogram feature, and the preset threshold is specifically set according to actual conditions.

[0074] In step S104, the vocoder output module is similar to the feature extraction module, the feature coupling module and the stream matching module, and is also a trained HiFiGAN neural network model, namely, Vocoder. The goal is to enhance the ability of the vocoder output module to distinguish synthetic or real audio as much as possible. The vocoder output module processes the Mel-spectrogram features that meet the preset decoding conditions to obtain the processed Mel-spectrogram features, and obtains the identity-converted audio based on the processed Mel-spectrogram features, thereby realizing the conversion of the identity of the speaker of the audio to be converted to the identity of the target speaker.

[0075] In the specific implementation process of step S104, there is an embodiment in which the voice coding output module processes the mel spectrum features that meet the preset decoding conditions to obtain the processed mel spectrum features, and obtains the identity-converted audio based on the processed mel spectrum features, including:

[0076] S1041, the voice coding output module performs time correlation and channel correlation learning on the obtained Mel spectrum features that meet the preset decoding conditions to obtain the learned Mel spectrum features;

[0077] S1042: The voice coding output module identifies the learned Mel-spectrogram features to obtain audio after identity conversion.

[0078] In steps S1041-S1042, the vocoding output module judges the Mel spectrum features output by the stream matching module to obtain Mel spectrum features that meet the preset decoding conditions, and decodes the Mel spectrum features that meet the preset decoding conditions through the generator of the HiFiGAN neural network model in the vocoding output module. The generator of the HiFiGAN neural network model of the vocoding output module mainly includes two parts, one is an upsampling structure, which is specifically composed of a one-dimensional convolution; the other is a multi-receptive field fusion module, which is mainly responsible for optimizing the sampling points obtained by upsampling, and is specifically composed of a residual network, that is, the Mel spectrum features that meet the preset decoding conditions are respectively learned by time correlation and channel correlation. This method is more helpful for the HiFiGAN neural network model of the vocoding output module to restore the speech waveform using the Mel spectrum, and relatively reduces the HiFiGAN neural network model of the vocoding output module. The learning burden is helpful for the rapid learning of the HiFiGAN neural network model of the vocoding output module, and the learned Mel spectrum features are identified from multiple angles by the identifier of the HiFiGAN neural network model in the vocoding output module. The vocoding output module HiFiGAN neural network model has two identifiers, namely a multi-scale identifier and a multi-cycle identifier, which identify the speech from two different angles. The multi-scale identifier is derived from the MelGAN vocoder, which continuously averages and pools the speech sequence, successively halves the length of the speech sequence, and then applies several layers of convolution at different scales of the speech, and finally flattens it as the output of the multi-scale identifier. The multi-cycle identifier folds the one-dimensional audio sequence into a two-dimensional plane with different sequence lengths, and applies two-dimensional convolution on the two-dimensional plane, thereby obtaining the audio after identity conversion, and ensuring the quality of the audio after identity conversion.

[0079] In the specific implementation process of step S1041, there is an embodiment in which the specific steps of the voice coding output module obtaining the mel spectrum features that meet the preset decoding conditions include:

[0080] S10411, the voice coding output module determines whether the Mel spectrum feature meets the optimization target of the loss function in the stream matching module;

[0081] S10412: If yes, the voice coding output module obtains the Mel spectrum features that meet the preset decoding conditions.

[0082] In steps S10411-S10412, the optimization target of the loss function in the flow matching module is expressed by formula (1):

[0083] (1)

[0084] in, To optimize the goal, is the edge weighted integration condition, are the parameters to be learned, Even distribution , is the target probability density path, is a vector field, is the vector field to be learned, The conditional probability distribution is the conditional probability distribution. The voice coding output module is pre-set to determine whether the Mel-spectrogram feature meets the optimization target of the loss function in the stream matching module. The voice coding output module determines that the Mel-spectrogram feature meets the optimization target of the loss function in the stream matching module, that is, the Mel-spectrogram feature output by the Mel-spectrogram model in the stream matching module is close to or even consistent with the actual Mel-spectrogram feature. After the voice coding output module obtains the Mel-spectrogram feature that meets the preset decoding conditions, the Mel-spectrogram feature is processed to obtain the audio that completes the identity conversion.

[0085] Example 2

[0086] The present application also provides a speech conversion device based on an attention mechanism and a stream matching strategy, such as Figure 4 The block diagram of a speech conversion device based on an attention mechanism and a stream matching strategy is shown. The functions implemented by the speech conversion device based on an attention mechanism and a stream matching strategy correspond to the steps of executing a speech conversion method based on an attention mechanism and a stream matching strategy on a terminal device. The device can be understood as a component of a server including a processor, which is applicable to a speech conversion system. The speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a voice code output module. The device includes:

[0087] Extraction module 401, used for the feature extraction module to extract features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features;

[0088] A learning module 402 is used for the feature coupling module to learn the correspondence between the tone features, the emotion features and the content features based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity features of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features;

[0089] The calculation module 403 is used for the stream matching module to perform Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features;

[0090] The conversion module 404 is used for the voice coding output module to process the Mel-spectrogram features that meet the preset decoding conditions to obtain processed Mel-spectrogram features, so as to obtain the audio after identity conversion based on the processed Mel-spectrogram features.

[0091] In a feasible implementation, the conversion module includes:

[0092] A first learning module is used for the voice coding output module to learn the Mel spectrum features that meet the preset decoding conditions respectively by time correlation and channel correlation to obtain the learned Mel spectrum features;

[0093] The identification module is used for the voice code output module to identify the learned Mel spectrum features to obtain the audio after identity conversion.

[0094] In a feasible implementation manner, the conversion module further includes:

[0095] A judgment module, used for the voice coding output module to judge whether the Mel spectrum feature meets the optimization target of the loss function in the stream matching module;

[0096] The obtaining module is used for, if it meets the conditions, the voice coding output module obtains the Mel spectrum features that meet the preset decoding conditions.

[0097] In a feasible implementation, the computing module includes:

[0098] An updating module, configured to calculate the difference between the Mel spectrum feature and the actual Mel spectrum feature using a loss function, and update the parameters of the Mel spectrum model using the difference; the actual Mel spectrum feature is extracted from the audio to be converted;

[0099] The first output module is used to repeat the above steps until the difference between the mel spectrum feature output by the mel spectrum model and the actual mel spectrum feature is lower than a preset threshold.

[0100] In a feasible implementation, the learning module includes:

[0101] A second learning module is used for the feature coupling module to extract and learn the corresponding relationship between the tone feature, the emotion feature and the content feature based on the attention mechanism;

[0102] A coupling module is used for coupling the feature coupling module with the identity feature of the target speaker based on the corresponding relationship to obtain a latent vector.

[0103] In a feasible implementation, the extraction module includes:

[0104] A second output module, used for the feature extraction module to output the audio to be converted to multiple extractors in the feature extraction module;

[0105] The calling module is used for the feature extraction module to call multiple preset extractors to extract the audio to be converted respectively to obtain audio features of multiple dimensions.

[0106] In a feasible implementation manner, the extraction module further includes:

[0107] An encoding module, used for a content extractor among the plurality of extractors to encode the audio to be converted into continuous encoded audio through an encoder;

[0108] The first conversion module is used for the content extractor to convert the continuous encoded audio into content features of a corresponding discrete unit sequence through a clusterer.

[0109] Example 3

[0110] The present application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502 and a bus 503, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 through the bus 503. When the machine-readable instructions are executed by the processor 501, the steps of the speech conversion method based on the attention mechanism and the stream matching strategy are performed.

[0111] Example 4

[0112] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the speech conversion method based on the attention mechanism and the stream matching strategy are executed.

[0113] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0114] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0117] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech conversion method based on attention mechanism and stream matching strategy, characterized in that: Applicable to a speech conversion system, the speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a vocoding output module, and the method includes: The feature extraction module extracts features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features; The feature coupling module learns the correspondence between the pitch feature, the emotion feature and the content feature based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity feature of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features; The stream matching module performs Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features; The vocoding output module processes the Mel spectrum features that meet the preset decoding conditions to obtain processed Mel spectrum features, so as to obtain the audio after identity conversion based on the processed Mel spectrum features; The vocoding output module processes the mel spectrum features that meet the preset decoding conditions to obtain the processed mel spectrum features, and obtains the identity-converted audio based on the processed mel spectrum features, including: The vocoding output module performs time correlation and channel correlation learning on the obtained Mel spectrum features that meet the preset decoding conditions to obtain the learned Mel spectrum features; The voice code output module identifies the learned Mel spectrum features to obtain the audio after identity conversion; The specific steps of the vocoding output module obtaining the mel spectrum features that meet the preset decoding conditions include: The vocoding output module determines whether the Mel spectrum feature meets the optimization target of the loss function in the stream matching module; If yes, the vocoding output module obtains the Mel spectrum features that meet the preset decoding conditions; The specific steps of continuously training the mel spectrum model with mel spectrum features include: The loss function is used to calculate the difference between the Mel spectrum feature and the actual Mel spectrum feature, and the parameters of the Mel spectrum model are updated with the difference; the actual Mel spectrum feature is extracted from the audio to be converted; Repeat the above steps until the difference between the mel spectrum feature output by the mel spectrum model and the actual mel spectrum feature is lower than the preset threshold.

2. The method according to claim 1, characterized in that The feature coupling module learns the correspondence between the tone features, the emotion features and the content features based on the attention mechanism to obtain a latent vector based on the correspondence and the preset identity features of the target speaker, including: The feature coupling module extracts and learns the correspondence between the tone features, emotion features and content features based on the attention mechanism; The feature coupling module couples the corresponding relationship with the identity feature of the target speaker to obtain a latent vector.

3. The method according to claim 1, characterized in that The feature extraction module extracts features from the acquired audio to be converted to obtain audio features, including: The feature extraction module outputs the audio to be converted to multiple extractors in the feature extraction module; The feature extraction module calls a plurality of preset extractors to extract the audio to be converted respectively, and obtains audio features of multiple dimensions.

4. The method according to claim 3, characterized in that The feature extraction module calls multiple preset extractors to extract the audio to be converted respectively, and obtains audio features of multiple dimensions, including: The content extractor among the plurality of extractors encodes the audio to be converted into continuous encoded audio through an encoder; The content extractor converts the continuous encoded audio into content features of a corresponding discrete unit sequence through a clusterer.

5. A speech conversion device based on attention mechanism and stream matching strategy, characterized in that: Applicable to a speech conversion system, the speech conversion system includes a feature extraction module, a feature coupling module, a stream matching module and a voice code output module, and the device includes: An extraction module, used for the feature extraction module to extract features from the acquired audio to be converted to obtain audio features; the audio features are encoded discrete unit sequences; the audio features include tone features, emotion features and content features; A learning module, used for the feature coupling module to learn the correspondence between the pitch feature, the emotion feature and the content feature based on the attention mechanism, so as to obtain a latent vector based on the correspondence and the preset identity feature of the target speaker; the latent vector is a matrix composed of numerical values ​​corresponding to multiple audio features; A calculation module, used for the stream matching module to perform Mel spectrum calculation on the latent vector based on the trained Mel spectrum model to obtain Mel spectrum features, so as to output the Mel spectrum features to the voice coding output module; the Mel spectrum model is obtained by continuously training the Mel spectrum features; A conversion module, used for the voice coding output module to process the Mel spectrum features that meet the preset decoding conditions to obtain the processed Mel spectrum features, so as to obtain the audio after identity conversion based on the processed Mel spectrum features; The conversion module comprises: A first learning module is used for the voice coding output module to learn the Mel spectrum features that meet the preset decoding conditions respectively by time correlation and channel correlation to obtain the learned Mel spectrum features; An identification module, used for the voice code output module to identify the learned Mel spectrum features to obtain the audio after identity conversion; The conversion module further includes: A judgment module, used for the voice coding output module to judge whether the Mel spectrum feature meets the optimization target of the loss function in the stream matching module; An obtaining module is used for obtaining the Mel spectrum features that meet the preset decoding conditions by the voice coding output module if the conditions are met; The computing module comprises: An updating module, configured to calculate the difference between the Mel spectrum feature and the actual Mel spectrum feature using a loss function, and update the parameters of the Mel spectrum model using the difference; the actual Mel spectrum feature is extracted from the audio to be converted; The first output module is used to repeat the above steps until the difference between the mel spectrum feature output by the mel spectrum model and the actual mel spectrum feature is lower than a preset threshold.

6. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of a speech conversion method based on an attention mechanism and a stream matching strategy as described in any one of claims 1 to 4 are performed.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a speech conversion method based on an attention mechanism and a stream matching strategy as described in any one of claims 1 to 4.

Images