Voice generation model construction method and device, electronic equipment and readable medium

Abstract

The invention provides a voice generation model construction method and device, electronic equipment and a readable medium, and the method comprises the steps: inputting a preset training voice into a preset vector quantizer, and obtaining a training semantic discrete feature of the training voice; the training semantic discrete features comprise the language style of the training voice; obtaining a training text corresponding to the training voice, and training a preset autoregression voice model by using the training text and the training semantic discrete features to obtain a semantic discrete feature generation model; obtaining a training Mel spectrogram corresponding to the training semantic discrete features; training a preset optimal transmission condition flow matching model by using the training semantic discrete features and the training Mel spectrogram to obtain a Mel spectrogram generation model; and constructing a voice generation model based on the Mel spectrogram generation model and the semantic discrete feature generation model. According to the invention, multi-language-style voice or dialect voice generation is realized through a single voice generation model, the annotation cost is reduced, and the mechanical feeling generated in audio generation is reduced.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for constructing a speech generation model, a device for constructing a speech generation model, an electronic device and a computer-readable medium. Background Art

[0002] Text-to-Speech (TTS) technology refers to the technology of converting text into speech. In traditional speech generation schemes, the input text is first preprocessed, including word segmentation, part-of-speech tagging, prosody prediction and other steps. Then, the characters or letters (graphemes) in the text are converted into corresponding speech phonemes (phonemes) through the G2P (Grapheme-to-Phoneme) module, which serves as the input of the speech generation system. When processing multi-dialect speech generation, data annotation for multi-dialect speech is required, and the requirements for data annotation are high. Moreover, the G2P module needs to adapt to the pronunciation rules of different dialects, that is, for different dialects, a set of dedicated G2P modules needs to be developed. This usually means that the G2P model and TTS model need to be trained separately for each dialect, resulting in high development and maintenance costs. In addition, the speech generated using speech generation technology has the problem of insufficient anthropomorphism. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for constructing a speech generation model to solve the problems that when processing multi-dialect speech generation, data annotation for multi-dialect speech needs to be performed, and the requirements for data annotation are high; for different dialects, a set of dedicated G2P modules need to be developed; and speech generated using speech generation technology lacks anthropomorphism.

[0004] The embodiment of the present invention discloses a method for constructing a speech generation model, the method comprising:

[0005] Inputting a preset training speech into a preset vector quantizer to obtain training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech;

[0006] Acquire a training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model;

[0007] Obtaining a training Mel-spectrogram corresponding to the training semantic discrete feature;

[0008] The preset optimal transmission condition stream matching model is trained by using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model;

[0009] A speech generation model is constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model.

[0010] Optionally, constructing a speech generation model based on the mel-spectrogram generation model and the semantic discrete feature generation model includes:

[0011] Using the semantic discrete feature generation model to construct a model to be processed;

[0012] Acquire a training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed;

[0013] Fine-tuning the model to be processed using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model;

[0014] The speech generation model is constructed based on the hybrid expert model and the mel-spectrogram generation model.

[0015] Optionally, the hybrid expert model includes at least one expert model.

[0016] Optionally, the hybrid expert model further includes a language router; and the method includes:

[0017] Inputting a preset text into the hybrid expert model to obtain a probability distribution of the at least one expert model output by the language router;

[0018] Extracting semantic discrete features of the text using the expert model;

[0019] Based on the probability distribution, the semantic discrete features of the text are fused to obtain target semantic discrete features.

[0020] Optionally, the method comprises:

[0021] Inputting the target semantic discrete features into the Mel-spectrogram generation model to obtain a Mel-frequency graph of the text;

[0022] Based on the Mel frequency map, using a preset vocoder to obtain an audio waveform of the text;

[0023] Based on the audio waveform and the text, the speech corresponding to the text is obtained.

[0024] Optionally, the using the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model includes:

[0025] During the training process of the autoregressive speech model, the training semantic discrete features are used as the inference output of the autoregressive speech model.

[0026] Optionally, the fusing the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features includes:

[0027] Based on the probability distribution, determining a weight of the at least one expert model;

[0028] Based on the weights, the semantic discrete features of the text are fused to obtain the target semantic discrete features.

[0029] The embodiment of the present invention also discloses a device for constructing a speech generation model, the device comprising:

[0030] A training semantic discrete feature obtaining module is used to input a preset training speech into a preset vector quantizer to obtain the training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech;

[0031] An autoregressive speech model training module is used to obtain a training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model;

[0032] A training Mel-spectrogram acquisition module is used to acquire a training Mel-spectrogram corresponding to the training semantic discrete feature;

[0033] An optimal transmission condition stream matching model training module is used to train a preset optimal transmission condition stream matching model using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model;

[0034] The speech generation model construction module is used to construct a speech generation model based on the Mel-spectrogram generation model and the semantic discrete feature generation model.

[0035] Optionally, the speech generation model construction module includes:

[0036] A model-to-be-processed construction submodule is used to generate a model using the semantic discrete features to construct a model to be processed;

[0037] A submodule for obtaining training text to be processed, used for obtaining training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed;

[0038] A hybrid expert model obtaining submodule is used to fine-tune the model to be processed by using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model;

[0039] The speech generation model construction submodule is used to construct the speech generation model based on the hybrid expert model and the Mel-spectrogram generation model.

[0040] Optionally, the hybrid expert model includes at least one expert model.

[0041] Optionally, the hybrid expert model further includes a language router; the device includes:

[0042] A probability distribution obtaining module, used for inputting a preset text into the hybrid expert model to obtain a probability distribution of the at least one expert model output by the language router;

[0043] An extraction module, used for extracting semantic discrete features of the text using the expert model;

[0044] The fusion module is used to fuse the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features.

[0045] Optionally, the device comprises:

[0046] A mel frequency map obtaining module, used for inputting the target semantic discrete features into the mel frequency map generation model to obtain the mel frequency map of the text;

[0047] An audio waveform acquisition module, used to acquire the audio waveform of the text using a preset vocoder based on the Mel frequency map;

[0048] The speech acquisition module is used to acquire the speech corresponding to the text based on the audio waveform and the text.

[0049] Optionally, the autoregressive speech model training module includes:

[0050] The inference output is used as a submodule to use the training semantic discrete features as the inference output of the autoregressive speech model during the training process of the autoregressive speech model.

[0051] Optionally, the fusion module includes:

[0052] A weight determination submodule, configured to determine a weight of the at least one expert model based on the probability distribution;

[0053] The fusion submodule is used to fuse the semantic discrete features of the text based on the weights to obtain the target semantic discrete features.

[0054] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0055] The memory is used to store computer programs;

[0056] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0057] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0058] The embodiments of the present invention include the following advantages:

[0059] In an embodiment of the present invention, a preset training speech is input into a preset vector quantizer to obtain training semantic discrete features of the training speech, and the training semantic discrete features include the language style of the training speech. A training text corresponding to the training speech is obtained, and a preset autoregressive speech model is trained using the training text and the training semantic discrete features to obtain a semantic discrete feature generation model, a training mel-spectrogram corresponding to the training semantic discrete features is obtained, and a preset optimal transmission condition stream matching model is trained using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model. A speech generation model is constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model. The generation of speech or dialect speech in multiple language styles is achieved through a single speech generation model, thereby reducing the cost and complexity of development and deployment, and having good scalability, significantly reducing the dependence on linguistic knowledge and pronunciation dictionaries of each dialect, reducing the cost of annotation, and the speech generation model generates speech based on the semantic discrete features corresponding to the language style, effectively reducing the mechanical feeling generated by the traditional speech generation method in audio generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flowchart of a method for constructing a speech generation model provided in an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of a shared layer and an expert layer provided in an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of the relationship between the gating network and the dialect expert provided in an embodiment of the present invention;

[0063] Figure 4is a flowchart of the steps of another method for constructing a speech generation model provided in an embodiment of the present invention;

[0064] Figure 5 It is a structural block diagram of a device for constructing a speech generation model provided in an embodiment of the present invention;

[0065] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0066] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Reference Figure 1 , shows a flowchart of a method for constructing a speech generation model provided in an embodiment of the present invention, which may specifically include the following steps:

[0069] Step 101, inputting a preset training speech into a preset vector quantizer to obtain training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech;

[0070] In the embodiment of the present invention, a supervised trained semantic speech quantizer is used to extract training semantic discrete features of large-scale Mandarin and English speech signals in the training set. It should be noted that the training semantic discrete features are a set of discrete symbols or values, which are discrete sequences rich in style and semantics, and contain information such as the user's pronunciation habits.

[0071] In the embodiment of the present invention, the open source quantizer of FunAudioLLM is reused, which is based on the pre-trained speech recognition model SenseVoice-Large, and a vector quantizer is added after the first six layers of its encoder. Among them, FunAudioLLM is an open source project, and SenseVoice-Large is a speech recognition model.

[0072] The vector quantizer is a "dictionary" containing 4096 channels, which can map the input speech signal into discrete semantic discrete features. Therefore, in an embodiment of the present invention, the preset training speech is input into the vector quantizer to obtain the training semantic discrete features of the training speech. The training semantic discrete features include the language style of the training speech. When the user reads the text corresponding to the speech of this language style, he has a specific pronunciation habit. If the speech is generated using this pronunciation habit, the speech has an anthropomorphic effect.

[0073] Step 102, obtaining a training text corresponding to the training speech, and using the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model;

[0074] In an embodiment of the present invention, texts corresponding to large-scale Mandarin and English speech signals can be used as training texts, and training semantic discrete features of the training speech of large-scale Mandarin and English speech signals are obtained using a vector quantizer.

[0075] In the embodiment of the present invention, a training text corresponding to the training speech is obtained, and a preset autoregressive speech model is trained using the training text and training semantic discrete features to obtain a semantic discrete feature generation model. It should be noted that the autoregressive speech model is a large model.

[0076] In some embodiments of the present invention, the step of training a preset autoregressive speech model using the training text and the training semantic discrete features to obtain a semantic discrete feature generation model includes:

[0077] During the training process of the autoregressive speech model, the training semantic discrete features are used as the inference output of the autoregressive speech model.

[0078] In an embodiment of the present invention, the autoregressive speech model is trained using a teacher-forcing paradigm, that is, during the training process of the autoregressive speech model, the training semantic discrete features are used as the inference output of the autoregressive speech model, thereby reducing the accumulated errors during the training process and accelerating the convergence of the model.

[0079] Specifically, the training semantic discrete features and training text are combined into a unified input sequence, and the input sequence is designed as follows:

[0080] [START,text_id_seq,semantic_token_seq,END]

[0081] Among them, START refers to the start of the sequence, END refers to the end of the sequence, text_id_seq is the training text, and semantic_token_seq refers to the training semantic discrete features. The four cannot overlap in numerical representation. The loss function in the training process of the autoregressive speech model is designed as:

[0082]

[0083] Among them, L LM It is the name of the loss function, usually used to represent a specific type of loss. is a normalization factor that ensures that the loss value does not vary with the size of the dataset. This summation symbol represents the accumulation of all terms from 1 to L + 1. log q(μ1) is a logarithmic function that calculates the logarithm of q(μ1).

[0084] In the inference stage, the training text and training semantic discrete features are merged, and the training semantic discrete features are regarded as the pre-generated part. Through the input sequence, the recursive autoregressive speech model gradually predicts the semantic discrete representation corresponding to the text to be synthesized until the "sequence end" Token (END) is encountered.

[0085] Step 103, obtaining a training Mel-frequency spectrum corresponding to the training semantic discrete feature;

[0086] In the embodiment of the present invention, a training mel-spectrogram corresponding to the training semantic discrete feature may be obtained.

[0087] Step 104, using the training semantic discrete features and the training mel-spectrogram to train a preset optimal transmission condition stream matching model to obtain a mel-spectrogram generation model;

[0088] In the embodiment of the present invention, the Mel-spectrogram generation model can be obtained by training the preset optimal transmission condition flow matching model (OT-CFM) by training the semantic discrete features and training the Mel-spectrogram. It should be noted that the audio waveform can be synthesized from the training Mel-spectrogram using a HiFTNet-based vocoder.

[0089] In the embodiment of the present invention, an optimal transmission conditional flow matching model is adopted. The model uses a convolutional Transformer U-Net to determine the vector field from the prior distribution to the target distribution. From the prior distribution to the target distribution, the conditional probability P(S|X,v,S ref ), where X and v represent speech token and speaker vector respectively, S ref is the target distribution, and S is the prior distribution. In the embodiment of the present invention, the semantic-to-speech part reuses the FunAudioLLM open source Flow model and HiFt model. The vector field describes how to map from the prior distribution (using the training semantic discrete features to obtain the corresponding known probability distribution of the training Mel-spectrogram) to the target distribution (using the training semantic discrete features to obtain the target probability distribution of the corresponding training Mel-spectrogram).

[0090] Step 105: construct a speech generation model based on the mel-spectrogram generation model and the semantic discrete feature generation model.

[0091] In the embodiment of the present invention, a speech generation model can be constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model.

[0092] In some embodiments of the present invention, the step of constructing a speech generation model based on the mel-spectrogram generation model and the semantic discrete feature generation model includes:

[0093] Using the semantic discrete feature generation model to construct a model to be processed;

[0094] Acquire a training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed;

[0095] Fine-tuning the model to be processed using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model;

[0096] The speech generation model is constructed based on the hybrid expert model and the mel-spectrogram generation model.

[0097] In an embodiment of the present invention, a model to be processed can be constructed by using a semantic discrete feature generation model. Specifically, the base GPT model is composed of a series of pre-trained Transformer models, with a total of L layers. The multi-layer Transformer structure stacked in the base GPT model is divided into a shared layer and an expert layer. Among them, the first N layers are shared layers, and the (LN) layer is an MDE (Mixture-of-Dialect-Experts, dialect expert mixture model) layer. For the MDE layer, the feedforward network layer FFN (Feedforward Network) in the Transformer structure is replaced with MDE FFN. At this time, the base GPT model is the model to be processed, which realizes the use of semantic discrete feature generation model to construct the model to be processed.

[0098] Reference Figure 2 , showing a schematic diagram of a shared layer and an expert layer provided in an embodiment of the present invention. The stacked multi-layer Transformer structure in the base GPT model is divided into a shared layer and an expert layer. Among them, the first N layers are shared layers, and the (LN) layer is an MDE layer. For the MDE layer, the feedforward network layer FFN (Feedforward Network) in the Transformer structure is replaced by MDE FFN.

[0099] It should be noted that GPT (Generative Pre-trained Transformer) is a deep learning model based on the Transformer architecture, which is good at handling natural language generation tasks. Its core idea is to pre-train on a large amount of text data and predict the next word through autoregression. In the field of speech generation large models, the GPT model accepts input text and generates corresponding discrete high-level semantic representations based on its context information.

[0100] In the embodiment of the present invention, a training text to be processed in the target language style and semantic discrete features of speech corresponding to the training text to be processed are obtained.

[0101] Specifically, the training text to be processed in the target language style can be a dialect text. The dialect data preparation stage is the basis for ensuring that the model can generate high-quality, supernatural dialect speech. A variety of dialect speech data is obtained by combining web crawlers and manual recording. Under the premise of retaining the dialect label, corresponding transcribers are recruited for the corresponding dialect, and detailed transcription rules need to be formulated to ensure the consistency and accuracy of the transcribed text. After the transcription is completed, a quality inspection mechanism needs to be set up to ensure the writing quality. The dialect speech data is input into the vector quantizer, and the vector quantizer is used to obtain the semantic discrete features of the dialect speech, that is, the speech discrete features of the speech corresponding to the training text to be processed.

[0102] In an embodiment of the present invention, the processing model is fine-tuned using the semantic discrete features of the training text to be processed and the speech corresponding to the training text to be processed to obtain a hybrid expert model. Based on the hybrid expert model and the Mel-spectrogram generation model, a speech generation model can be constructed. It should be noted that MoE (Mixture of Experts) is a machine learning model architecture that decomposes complex problems into multiple sub-problems, divides the space through a gating mechanism that acts as a router, and guides data to specialized experts. Each sub-problem is processed by a specialized "expert" sub-network, thereby improving the efficiency and performance of the model. Hybrid expert models can be divided into two types: one that implicitly divides the problem space through a gating network optimized by a loss function, and the other that explicitly divides the space, usually using clustering technology to identify subspaces before training begins.

[0103] In some embodiments of the present invention, the hybrid expert model includes at least one expert model.

[0104] In the embodiment of the present invention, the hybrid expert model includes at least one expert model.

[0105] During the training of the model to be processed, there is a language router before the MDE FFN module of the model to be processed. Through the gating mechanism, based on the input dialect label d, the text to be processed for training, and the semantic discrete feature sequence x of the speech corresponding to the text to be processed for training, a vector with a dimension of n is obtained. The dialect label represents the type of dialect. Each value in the vector represents the probability that the input is routed to the corresponding expert model. The gating mechanism can be specifically expressed as:

[0106] G(x) = Softmax(KeepTopK(x, d), k)

[0107] According to the probability distribution output by the gating network, the top k most suitable expert models (k << n) (n is the total number of expert models) are selected, and the unselected expert models will not be activated. Each activated expert model has its own independent parameters, and finally the output results of the selected expert models are weighted and fused according to the probabilities generated by the gating network.

[0108] For example: If k expert models are activated, their outputs are combined according to their respective weights to generate the final dialect-specific inference semantic discrete representation:

[0109]

[0110] where G i (x, d) represents the weight of the i-th expert model, and E i (x) represents the processing result of the i-th expert model on the input.

[0111] The expert model is also called a dialect expert. Referring to Figure 3 , a schematic diagram showing the relationship between the gating network and the dialect expert provided in the embodiment of the present invention is shown. The preset text is input into the mixture of experts model, and the probability distribution of at least one expert model output by the gating network is obtained. There are activated dialect experts and unactivated dialect experts in the mixture of experts model.

[0112] It should be noted that the gating mechanism: is a technique specifically used to control the flow of information in a neural network. The gating mechanism determines which input signals should be passed to subsequent layers or outputs by learning one or more gating functions. The gating network is often used to process complex tasks such as multi-modal data fusion, feature selection, and model sparsity.

[0113] In some embodiments of the present invention, the mixture of experts model further includes a language router; the method includes:

[0114] Input the preset text into the mixture of experts model to obtain the probability distribution of the at least one expert model output by the language router;

[0115] Extracting semantic discrete features of the text using the expert model;

[0116] Based on the probability distribution, the semantic discrete features of the text are fused to obtain target semantic discrete features.

[0117] In an embodiment of the present invention, the hybrid expert model also includes a language router. A preset text is input into the hybrid expert model to obtain the probability distribution of at least one expert model output by the language router. The semantic discrete features of the text are extracted using the expert model; based on the probability distribution, the semantic discrete features of the text extracted by multiple expert models are fused to obtain the target semantic discrete features. It should be noted that in the process of model training to obtain the hybrid expert model and the application of the hybrid expert model, the relationship between the language router and the at least one expert model is the same, and the language routers are used to determine the probability distribution of at least one expert model through a gating mechanism, which will not be described in detail in the present invention.

[0118] In some embodiments of the present invention, the step of fusing the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features includes:

[0119] Based on the probability distribution, determining a weight of the at least one expert model;

[0120] Based on the weights, the semantic discrete features of the text are fused to obtain the target semantic discrete features.

[0121] In the embodiment of the present invention, the weight of at least one expert model is determined based on the probability distribution. According to the weight, the semantic discrete features of the text are fused to obtain the target semantic discrete features.

[0122] In some embodiments of the invention, the method comprises:

[0123] Inputting the target semantic discrete features into the Mel-spectrogram generation model to obtain a Mel-frequency graph of the text;

[0124] Based on the Mel frequency map, using a preset vocoder to obtain an audio waveform of the text;

[0125] Based on the audio waveform and the text, the speech corresponding to the text is obtained.

[0126] In an embodiment of the present invention, the target semantic discrete features extracted by the hybrid expert model are input into the mel-spectrogram generation model to obtain the mel-frequency graph of the text. Based on the mel-frequency graph, the audio waveform of the text can be obtained using a preset vocoder. By processing the text according to the audio waveform, the speech corresponding to the text can be obtained.

[0127] In an embodiment of the present invention, a preset training speech is input into a preset vector quantizer to obtain training semantic discrete features of the training speech, and the training semantic discrete features include the language style of the training speech. A training text corresponding to the training speech is obtained, and a preset autoregressive speech model is trained using the training text and the training semantic discrete features to obtain a semantic discrete feature generation model, a training mel-spectrogram corresponding to the training semantic discrete features is obtained, and a preset optimal transmission condition stream matching model is trained using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model. A speech generation model is constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model. The generation of speech or dialect speech in multiple language styles is achieved through a single speech generation model, thereby reducing the cost and complexity of development and deployment, and having good scalability, significantly reducing the dependence on linguistic knowledge and pronunciation dictionaries of each dialect, reducing the cost of annotation, and the speech generation model generates speech based on the semantic discrete features corresponding to the language style, effectively reducing the mechanical feeling generated by the traditional speech generation method in audio generation.

[0128] In an embodiment of the present invention, a hybrid expert model is obtained by fine-tuning the semantic discrete feature generation model obtained based on pre-training, which effectively reduces the mechanical feeling produced by the traditional speech generation method in audio generation. The present invention simplifies the system architecture of dialect speech generation and realizes efficient and scalable multi-dialect speech generation. Through the introduction of the hybrid expert model, speech generation of multiple dialects can be realized in a single model, the pre-trained semantic discrete feature generation model is introduced into the dialect speech generation, and the mel-spectrogram generation model is used to obtain the mel-spectrogram and audio waveform to generate a more natural and more anthropomorphic dialect speech.

[0129] The embodiment of the present invention adopts a hybrid expert model, which can divide the speech generation of multiple dialects into different subspaces, and route the input of different dialects to the most suitable expert model through a gating mechanism, thus realizing the "divide and conquer" strategy. In this way, it is possible to complete the generation of speech in multiple dialects on a single model, significantly reducing the cost of development and deployment, simplifying the system architecture in the dialect generation application scenario, and no longer needing to deploy a separate speech generation system for each dialect.

[0130] The present invention adopts a word-level modeling approach, directly through the GPT model, based on the contextual information of the text representation, relying on the powerful representation ability of the model to automatically learn the mapping relationship between text and high-dimensional semantic discrete representation, predict discrete speech mark sequence, and realize speech generation. The need to maintain a set of traditional G2P (character to phoneme) models for each dialect is omitted, and there is no need to rely on a large amount of linguistic knowledge, rules, pronunciation dictionaries and other resources and large-scale accurate phoneme annotation data. Only speech and corresponding text transcription annotations are required as input, which greatly reduces the data annotation cost and model development cost. In addition, through direct modeling at the word level, the model can better capture global contextual information, rather than being limited to local phoneme representations, so that the speech generated by the model can be more natural, more coherent, and more anthropomorphic.

[0131] The present invention uses a semantic discrete feature generation model pre-trained on more than 100,000 hours of Mandarin and English speech data to fine-tune the model to be processed to obtain a hybrid expert model. The large-scale training data covers a richer and wider range of speech features, including diverse intonations, speech speeds, rhythms, pronunciation details, etc., and has learned powerful context understanding capabilities on massive data, while also possessing powerful domain knowledge transfer and generalization capabilities. Through this method, the mechanical feeling brought by traditional speech generation methods is reduced, and the naturalness, coherence and fluency of the generated speech are improved.

[0132] Reference Figure 4 , shows a flowchart of the steps of another method for constructing a speech generation model provided in an embodiment of the present invention, which may specifically include the following steps:

[0133] Step 401: extracting discrete speech quantization representation of large-scale speech data.

[0134] Step 402, pre-training the text-to-semantics and semantics-to-speech stages of the base speech generation model.

[0135] Step 403: dialect data collection, manual transcription and quality inspection.

[0136] Step 404, construct a dialect mixed expert model to fine-tune the text to the semantic stage.

[0137] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0138] Reference Figure 5 , shows a structural block diagram of a device for constructing a speech generation model provided in an embodiment of the present invention, which may specifically include the following modules:

[0139] The training semantic discrete feature obtaining module 501 is used to input the preset training speech into the preset vector quantizer to obtain the training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech;

[0140] The autoregressive speech model training module 502 is used to obtain the training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model;

[0141] A training mel-spectrogram acquisition module 503 is used to acquire a training mel-spectrogram corresponding to the training semantic discrete feature;

[0142] The optimal transmission condition stream matching model training module 504 is used to train a preset optimal transmission condition stream matching model using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model;

[0143] The speech generation model construction module 505 is used to construct a speech generation model based on the mel-spectrogram generation model and the semantic discrete feature generation model.

[0144] In an optional embodiment of the present invention, the speech generation model construction module includes:

[0145] A model-to-be-processed construction submodule is used to generate a model using the semantic discrete features to construct a model to be processed;

[0146] A submodule for obtaining training text to be processed, used for obtaining training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed;

[0147] A hybrid expert model obtaining submodule is used to fine-tune the model to be processed by using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model;

[0148] The speech generation model construction submodule is used to construct the speech generation model based on the hybrid expert model and the Mel-spectrogram generation model.

[0149] In an optional embodiment of the present invention, the hybrid expert model includes at least one expert model.

[0150] In an optional embodiment of the present invention, the hybrid expert model further includes a language router; the device includes:

[0151] A probability distribution obtaining module, used for inputting a preset text into the hybrid expert model to obtain a probability distribution of the at least one expert model output by the language router;

[0152] An extraction module, used for extracting semantic discrete features of the text using the expert model;

[0153] The fusion module is used to fuse the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features.

[0154] In an optional embodiment of the present invention, the device comprises:

[0155] A mel frequency map obtaining module, used for inputting the target semantic discrete features into the mel frequency map generation model to obtain the mel frequency map of the text;

[0156] An audio waveform acquisition module, used to acquire the audio waveform of the text using a preset vocoder based on the Mel frequency map;

[0157] The speech acquisition module is used to acquire the speech corresponding to the text based on the audio waveform and the text.

[0158] In an optional embodiment of the present invention, the autoregressive speech model training module includes:

[0159] The inference output is used as a submodule to use the training semantic discrete features as the inference output of the autoregressive speech model during the training process of the autoregressive speech model.

[0160] In an optional embodiment of the present invention, the fusion module includes:

[0161] A weight determination submodule, configured to determine a weight of the at least one expert model based on the probability distribution;

[0162] The fusion submodule is used to fuse the semantic discrete features of the text based on the weights to obtain the target semantic discrete features.

[0163] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0165] Memory 603, used for storing computer programs;

[0166] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0167] Inputting a preset training speech into a preset vector quantizer to obtain training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech;

[0168] Acquire a training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model;

[0169] Obtaining a training Mel-spectrogram corresponding to the training semantic discrete feature;

[0170] The preset optimal transmission condition stream matching model is trained by using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model;

[0171] A speech generation model is constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model.

[0172] In an optional embodiment of the present invention, the step of constructing a speech generation model based on the mel-spectrogram generation model and the semantic discrete feature generation model includes:

[0173] Using the semantic discrete feature generation model to construct a model to be processed;

[0174] Acquire a training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed;

[0175] Fine-tuning the model to be processed using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model;

[0176] The speech generation model is constructed based on the hybrid expert model and the mel-spectrogram generation model.

[0177] In an optional embodiment of the present invention, the hybrid expert model includes at least one expert model.

[0178] In an optional embodiment of the present invention, the hybrid expert model further includes a language router; the method includes:

[0179] Inputting a preset text into the hybrid expert model to obtain a probability distribution of the at least one expert model output by the language router;

[0180] Extracting semantic discrete features of the text using the expert model;

[0181] Based on the probability distribution, the semantic discrete features of the text are fused to obtain target semantic discrete features.

[0182] In an optional embodiment of the present invention, the method includes:

[0183] Inputting the target semantic discrete features into the Mel-spectrogram generation model to obtain a Mel-frequency graph of the text;

[0184] Based on the Mel frequency map, using a preset vocoder to obtain an audio waveform of the text;

[0185] Based on the audio waveform and the text, the speech corresponding to the text is obtained.

[0186] In an optional embodiment of the present invention, the method of training a preset autoregressive speech model using the training text and the training semantic discrete features to obtain a semantic discrete feature generation model includes:

[0187] During the training process of the autoregressive speech model, the training semantic discrete features are used as the inference output of the autoregressive speech model.

[0188] In an optional embodiment of the present invention, the step of fusing the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features includes:

[0189] Based on the probability distribution, determining a weight of the at least one expert model;

[0190] Based on the weights, the semantic discrete features of the text are fused to obtain the target semantic discrete features.

[0191] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0192] The communication interface is used for communication between the above terminal and other devices.

[0193] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0194] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0195] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes a method for constructing a speech generation model described in the above embodiment.

[0196] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute a method for constructing a speech generation model described in the above embodiment.

[0197] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0198] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0199] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0200] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for constructing a speech generation model, characterized in that: The method comprises: Inputting a preset training speech into a preset vector quantizer to obtain training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech; Acquire a training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model; Obtaining a training Mel-spectrogram corresponding to the training semantic discrete feature; The preset optimal transmission condition stream matching model is trained by using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model; A speech generation model is constructed based on the mel-spectrogram generation model and the semantic discrete feature generation model.

2. The method according to claim 1, characterized in that The method of constructing a speech generation model based on the Mel-spectrogram generation model and the semantic discrete feature generation model includes: Using the semantic discrete feature generation model to construct a model to be processed; Acquire a training text to be processed in a target language style and semantic discrete features of speech corresponding to the training text to be processed; Fine-tuning the model to be processed using the training text to be processed and the semantic discrete features of the speech corresponding to the training text to be processed to obtain a hybrid expert model; The speech generation model is constructed based on the hybrid expert model and the mel-spectrogram generation model.

3. The method according to claim 2, characterized in that The hybrid expert model includes at least one expert model.

4. The method according to claim 3, characterized in that The hybrid expert model also includes a language router; the method includes: Inputting a preset text into the hybrid expert model to obtain a probability distribution of the at least one expert model output by the language router; Extracting semantic discrete features of the text using the expert model; Based on the probability distribution, the semantic discrete features of the text are fused to obtain target semantic discrete features.

5. The method according to claim 4, characterized in that The method comprises: Inputting the target semantic discrete features into the Mel-spectrogram generation model to obtain a Mel-frequency graph of the text; Based on the Mel frequency map, using a preset vocoder to obtain an audio waveform of the text; Based on the audio waveform and the text, the speech corresponding to the text is obtained.

6. The method according to claim 1, characterized in that The method of using the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model includes: During the training process of the autoregressive speech model, the training semantic discrete features are used as the inference output of the autoregressive speech model.

7. The method according to claim 4, characterized in that The step of fusing the semantic discrete features of the text based on the probability distribution to obtain target semantic discrete features includes: Based on the probability distribution, determining a weight of the at least one expert model; Based on the weights, the semantic discrete features of the text are fused to obtain the target semantic discrete features.

8. A device for constructing a speech generation model, characterized in that: The device comprises: A training semantic discrete feature obtaining module is used to input a preset training speech into a preset vector quantizer to obtain the training semantic discrete features of the training speech; the training semantic discrete features include the language style of the training speech; An autoregressive speech model training module is used to obtain a training text corresponding to the training speech, and use the training text and the training semantic discrete features to train a preset autoregressive speech model to obtain a semantic discrete feature generation model; A training Mel-spectrogram acquisition module is used to acquire a training Mel-spectrogram corresponding to the training semantic discrete feature; An optimal transmission condition stream matching model training module is used to train a preset optimal transmission condition stream matching model using the training semantic discrete features and the training mel-spectrogram to obtain a mel-spectrogram generation model; The speech generation model construction module is used to construct a speech generation model based on the Mel-spectrogram generation model and the semantic discrete feature generation model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which when executed by one or more processors cause the processors to perform the method according to any one of claims 1 to 7.

Images