Voice generation method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a voice generation method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining voice sample data which comprises unique voice sample data and dialogue voice sample data; constructing a voice generation network framework, wherein the voice generation network framework comprises a text-to-semantic model, an acoustic model and a vocoder; training the voice generation network framework based on the voice sample data to obtain a dialogue voice generation model; and obtaining an initial text of to-be-generated dialogue voice, inputting the initial text into the dialogue voice generation model, and generating corresponding multi-round dialogue voice. According to the method, multi-stream semantic feature prediction can be carried out at the same time, each semantic feature sequence corresponds to one speaker, the generated dialogue voice is similar to real human voice in naturalness and coherence, multiple rounds of dialogues of multiple speakers are involved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech generation method, device, computer equipment and storage medium. Background Art

[0002] With the development of technology, people often use online customer service functions when using banking application software. Today's online customer service can answer corresponding questions based on user inquiries. The existing customer service platform uses customer service robots to complete customer service work. In order to improve the work efficiency of customer service robots, zero-sample voice generation is generally used to generate robot voice when the target voice is not used, so that the robot speaks closer to humans.

[0003] The goal of zero-shot speech generation is to generate speech without including the target speech during the training process, given only the target transcription and a brief reference of the target speech. Current zero-shot speech generation models have made many advances in generating high-fidelity and diverse speech. However, the generation of conversational speech and achieving human-like naturalness in the process of speech generation remain challenging tasks. Currently, there are few studies on multi-speaker, multi-dialogue turn speech generation models. Among the existing conversational speech generation models, some models simply connect the various segments of speech. When the same speaker has multiple rounds of dialogue, this may lead to inconsistencies in the speaker's previous and subsequent features; limiting its ability to generate speech, and sometimes making the model output difficult to understand, reducing the user experience. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a speech generation method, apparatus, computer device and storage medium, aiming to solve the problem that there is little research on multi-speaker and multi-dialogue turn speech generation models in the prior art. In the existing conversational speech generation models, some models simply connect each segment of speech. When the same speaker conducts multiple rounds of conversations, this may lead to inconsistency in the speaker's previous and subsequent features; it limits its ability to generate speech, and sometimes makes the model output difficult to understand, thereby reducing the user experience.

[0005] The technical solution of the present invention is as follows:

[0006] A first embodiment of the present invention provides a speech generation method, the method comprising:

[0007] Acquire voice sample data, wherein the voice sample data includes monologue voice sample data and dialogue voice sample data;

[0008] Constructing a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder;

[0009] Training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model;

[0010] The initial text of the dialogue speech to be generated is obtained, and the initial text is input into the dialogue speech generation model to generate corresponding multi-round dialogue speech.

[0011] Another embodiment of the present invention provides a speech generation device, the device comprising:

[0012] Acquire voice sample data, wherein the voice sample data includes monologue voice sample data and dialogue voice sample data;

[0013] Constructing a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder;

[0014] Training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model;

[0015] The initial text of the dialogue speech to be generated is obtained, and the initial text is input into the dialogue speech generation model to generate corresponding multi-round dialogue speech.

[0016] Another embodiment of the present invention provides a computer device, the computer device comprising at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above-mentioned speech generation method.

[0019] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can perform the steps of the above-mentioned speech generation method.

[0020] Beneficial effects: The speech generation method of the embodiment of the present invention can obtain speech sample data, the speech sample data includes monologue speech sample data and dialogue speech sample data; construct a speech generation network framework, the speech generation network framework includes a text-to-semantic model, an acoustic model and a vocoder; train the speech generation network framework based on the speech sample data to obtain a dialogue speech generation model; obtain the initial text of the dialogue speech to be generated, input the initial text into the dialogue speech generation model, and generate corresponding multi-round dialogue speech. The present invention can simultaneously perform multi-stream semantic feature prediction, each semantic feature sequence corresponds to a speaker, and the generated conversational speech is not only similar to real human speech in naturalness and coherence, but also involves multi-round dialogues of multiple speakers. The speech of one speaker can be seamlessly mixed with the speech of another speaker, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of an application environment of an embodiment of a speech generation method of the present invention;

[0023] Figure 2 A flowchart of a preferred embodiment of a speech generation method of the present invention;

[0024] Figure 3 for Figure 2 A flowchart of a specific embodiment of step S100;

[0025] Figure 4 for Figure 3 A flowchart of a specific embodiment of step S101;

[0026] Figure 5 for Figure 3 A flowchart of a specific embodiment of step S102;

[0027] Figure 6 for Figure 2 A flowchart of a specific embodiment of step S200;

[0028] Figure 7 for Figure 2 A flowchart of a specific embodiment of step S400;

[0029] Figure 8 for Figure 7 A flowchart of a specific embodiment of step S401;

[0030] Fig. 9 A schematic diagram of functional modules of a preferred embodiment of a speech generating device of the present invention;

[0031] Fig.10 A structural schematic diagram of a preferred embodiment of a computer device of the present invention;

[0032] Fig.11 Another structural schematic diagram of a preferred embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0035] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. Here, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0036] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural speech processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0037] The speech generation method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client obtains speech sample data, the speech sample data includes monologue speech sample data and dialogue speech sample data; constructs a speech generation network framework, the speech generation network framework includes a text-to-semantic model, an acoustic model and a vocoder; trains the speech generation network framework based on the speech sample data to obtain a dialogue speech generation model; obtains the initial text of the dialogue speech to be generated, inputs the initial text into the dialogue speech generation model, and generates corresponding multi-round dialogue speech. In the present invention, multi-stream semantic feature prediction can be performed simultaneously, each semantic feature sequence corresponds to a speaker, and multi-round dialogue speech containing multiple speakers can be generated simultaneously without generating speech for each speaker and then splicing. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] In view of the above problems, the present invention provides a method for generating speech. Figure 2 , Figure 2 FIG. 1 is a flow chart of a preferred embodiment of a speech generation method of the present invention. Figure 2 As shown, it includes:

[0039] Step S100: Acquire voice sample data, where the voice sample data includes monologue voice sample data and dialogue voice sample data.

[0040] The speech generation method in the embodiment of the present invention is applied to a server that provides an intelligent voice reply function in an intelligent customer service chat scenario, and the server can generate corresponding multi-round dialogue speech based on the text.

[0041] Intelligent customer service refers to the use of artificial intelligence technology, especially natural language processing, machine learning and deep learning, to provide automated customer service solutions for enterprises and organizations. Intelligent customer service can quickly respond to users' common problems and provide accurate information through preset question and answer libraries or intelligent search functions. Intelligent customer service can guide users to solve problems through self-service, such as operation guides, account information inquiries, etc. Intelligent customer service provides personalized services and product recommendations by analyzing user data and behavior. Intelligent customer service can also judge the user's emotional state, provide more considerate services, and enhance user experience. Intelligent customer service can support multiple languages ​​and serve users in different countries and regions. Intelligent customer service can also identify abnormal situations in the service and respond in time. Through machine learning, the conversation ability is continuously optimized to improve the problem solving rate. In addition, customer service robots can also simulate human conversation methods, interact with users through text or voice, and provide 24 / 7 service without human intervention.

[0042] The embodiment of the present invention is mainly used for intelligent customer service to interact with users through voice. Because the existing intelligent customer service voice supports monologue and cannot generate conversational voice, the user experience is poor. In the present invention, after learning the voice of the customer service staff, the corresponding conversational voice is generated. When the user queries, the generated conversational voice can be played to improve the user experience.

[0043] In the present invention, speech sample data for a conversational speech generation model is obtained, and the speech sample data includes two types of speech data, one is monologue speech data and the other is conversational speech data. Through these two types of sample data, the conversational speech model can generate both monologue speech and conversational speech, thereby improving the generalization of the model.

[0044] Among them, Figure 3 As shown, step S100, i.e. obtaining voice sample data, includes:

[0045] Step S101: Acquire initial monologue voice data, and pre-process the initial voice data to obtain monologue voice sample data.

[0046] In the voice sample data, there is initial monologue voice data. The initial monologue voice data is personal voice data and has a certain duration. Therefore, it is necessary to cut the initial monologue voice data as required, cut it into voice data of shorter duration, and facilitate algorithm processing, and obtain monologue voice sample data after processing.

[0047] Among them, Figure 4 As shown, in step S101, initial monologue voice data is obtained, and the initial voice data is preprocessed to obtain monologue voice sample data, including:

[0048] Step S1011, obtaining initial monologue voice data, and cutting the initial monologue voice data into monophonic samples that meet a predetermined duration;

[0049] Step S1012: extracting mel-spectrograms and corresponding semantic feature sequences from all monophonic samples to obtain monologue speech sample data.

[0050] When training the model, two types of speech data are mainly used, monologue speech data and dialogue speech data. For the initial monologue speech data, long segments need to be cut into smaller mono samples, while also meeting the minimum duration requirement, and the Mel spectrogram and semantic feature sequence of each sample need to be extracted. The minimum duration can be set as needed.

[0051] Mel spectrogram is a method of converting audio signals into two-dimensional images. It represents the frequency spectrum of audio signals based on the Mel frequency scale. This representation has been widely used in speech recognition, speech synthesis, audio classification and other fields.

[0052] The Mel spectrum generation process is as follows:

[0053] Signal preprocessing: Preprocess the audio signal, including pre-emphasis, framing, windowing, etc., in preparation for short-time Fourier transform (STFT).

[0054] STFT transform: Perform short-time Fourier transform on the preprocessed audio signal, converting it from the time domain to the frequency domain to obtain a spectrum.

[0055] Mel filter bank: The spectrum is filtered through a set of equal-height triangular filters (i.e., Mel filter bank). These filters are linearly distributed on the Mel frequency scale and can simulate the human ear's perception of different frequencies.

[0056] Generate Mel spectrogram: Visualize the filtered spectrum to obtain a Mel spectrogram. In the Mel spectrogram, the y-axis represents the Mel frequency, the x-axis represents the time, and the color represents the energy of the audio signal at that frequency and time point.

[0057] The semantic feature sequence is a specific sequence generated based on the text after the speech is recognized.

[0058] Step S102: Acquire initial conversation voice data, and preprocess the initial conversation voice data to obtain conversation voice sample data.

[0059] After the initial conversation voice data is obtained, since some conversations in the initial conversation voice data are longer and some are shorter, during processing, some of the initial conversation voice data with longer conversations need to be cut into shorter data to obtain conversation voice sample data.

[0060] Among them, Figure 5 As shown, step S102, i.e., obtaining initial conversation voice data, preprocessing the initial conversation voice data to obtain conversation voice sample data, includes:

[0061] Step S1021, obtaining first dialogue voice data whose duration is greater than a first predetermined duration, and cutting the first dialogue voice data into multi-channel voices whose duration is less than or equal to a second predetermined duration, wherein the multi-channel voices include at least two voices from two speakers;

[0062] Step S1022: extracting mel-spectrogram and semantic feature sequence from the multi-channel speech to obtain dialogue speech sample data.

[0063] For conversational speech, it is necessary to cut a conversation longer than the first predetermined time into multi-channel speech less than or equal to the second predetermined time, and at least two speech from two speakers must be included, and it is also necessary to ensure that the first sentence and the last sentence of each processed conversation do not overlap with other conversations. The first predetermined time is greater than the second predetermined time, and the first predetermined time and the second predetermined time can be set as needed.

[0064] After obtaining multi-channel speech, it is necessary to extract the mel-spectrogram and semantic feature sequence to obtain conversational speech sample data.

[0065] Step S103: Generate voice sample data based on the monologue voice sample data and the dialogue voice sample data.

[0066] After obtaining the monologue voice sample data and the dialogue voice sample data, they are mixed to obtain the final voice sample data.

[0067] Step S200: construct a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder.

[0068] Constructing a speech generation network framework. The speech generation network framework model of the embodiment of the present invention mainly includes a text-to-semantic model, an acoustic model and a voice encoder. The text-to-semantic model is used to convert the text corresponding to the sample speech into a semantic feature sequence.

[0069] The acoustic model is used to convert the semantic feature sequence into a mel-spectrogram, and the vocoder generates conversational speech based on the mel-spectrogram.

[0070] Among them, Figure 6 As shown, step S200, i.e. building a speech generation network framework, includes:

[0071] Step S201: construct a text-to-semantics model, wherein the text-to-semantics model is a sequence-to-sequence model based on an encoder-decoder structure;

[0072] Step S202: construct an acoustic model, where the acoustic model is an encoder based on a conditional stream matching model;

[0073] Step S203: construct a vocoder, wherein the vocoder is used to synthesize multi-round conversation speech according to the mel-spectrogram.

[0074] Construct the speech generation network framework. When constructing specifically, the text-to-semantic model: The text-to-semantic model is a sequence-to-sequence model based on an encoder-decoder structure. Its input is a text tag sequence obtained after the text is processed by a text tagger. Special tags are used during the processing to represent speaker transitions and interjections, and the output is a multi-stream semantic feature sequence. Taking two speakers as an example, when finally generating a speech feature sequence, a strategy is adopted to divide the generated semantic feature sequence into two different paragraphs, each corresponding to a different speaker participating in the conversation. The core idea of ​​the sequence-to-sequence model is to accept an input sequence, map it to a fixed-length representation through encoding (Encoder), and then map this representation back to the output sequence through decoding (Decoder). This makes the Seq2Seq model suitable for processing tasks with variable-length input and output.

[0075] Acoustic model: The acoustic model is an encoder based on a conditional stream matching model, whose input is a sequence of semantic features and multiple speaker prompts, in the form of Mel spectrograms corresponding to a short speech of each speaker. The conditional stream matching model is a generative model that converts input data into the desired output through conditional generation. In the encoder-decoder architecture, the encoder based on conditional stream matching is responsible for converting the input data (such as text or speech signals) into a series of feature representations, which will be used as input to the decoder to generate the target output.

[0076] A vocoder is a codec that analyzes and synthesizes speech, also known as a speech analysis and synthesis system or a speech band compression system. It is a powerful tool for compressing communication bands and conducting confidential communications.

[0077] The vocoder analyzes the speech signal at the transmitting end, extracts the characteristic parameters of the speech signal, such as the judgment of clear and voiced sounds, the fundamental frequency of voiced sounds, and the spectrum envelope, and then encodes and encrypts these characteristic parameters to match the channel. After being transmitted to the receiving end through the information channel, the original speech waveform is restored based on the received characteristic parameters. In this process, the analysis can be performed in the frequency domain, performing spectrum analysis on the speech signal; it can also be performed in the time domain, using its periodicity to extract some parameters for linear prediction, or performing correlation analysis on the speech signal.

[0078] According to different working principles and parameter extraction methods, vocoders can be divided into many types, such as channel vocoders, formant vocoders, homomorphic vocoders, linear prediction vocoders, etc. These types of vocoders differ in terms of synthesized voice quality, bit rate and complexity.

[0079] Channel vocoder: By analyzing the amplitude spectrum of the speech signal, it is divided into multiple adjacent frequency bands or channels, and then the envelope of each frequency band is encoded and transmitted. At the receiving end, the same filter bank is used to restore the original speech signal. Channel vocoder has the advantages of high voice quality, strong resistance to background noise, and good stability.

[0080] Formant vocoder: mainly transmits formant frequency and bandwidth as parameters. Since only a few formants need to be transmitted to achieve a lower data rate, the formant vocoder may exceed the channel vocoder in speech quality, but it is more difficult to correctly track the formant frequency when implementing it.

[0081] Homomorphic vocoder: Also known as cepstrum vocoder, the model parameter it transmits is the cepstrum of speech. Although in theory it is possible to achieve an ideal separation of the parameters of the excitation source and the vocal tract response in the speech model, it requires a lot of computational effort to implement, has a high data rate, and has poor resistance to speech background noise.

[0082] Linear Prediction Vocoder: Linear prediction is used to model the response of the vocal tract and the linear prediction coefficients are transmitted as parameters. Linear Prediction Vocoder is the most widely used type of vocoder and its speech quality ranks among the best among vocoders.

[0083] Step S300: training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model.

[0084] When training the model, two types of speech data are mainly used, monologue speech sample data and dialogue sample speech data. During the training process, the sample data is divided into a training set and a test set, the speech generation network framework is trained based on the training set, the model parameters are adjusted through a predefined loss function, and the model is tested based on the test set. The training process first converts the sample into a semantic feature sequence through a text-to-semantic model, then converts the semantic feature sequence into a mel-spectrogram through an acoustic model, and finally converts the mel-spectrogram into dialogue speech through a vocoder.

[0085] Step S400: obtaining an initial text of the dialogue speech to be generated, inputting the initial text into the dialogue speech generation model, and generating corresponding multi-round dialogue speech.

[0086] When it is necessary to generate multiple rounds of conversational speech, the initial text of the conversational speech to be generated is obtained, the initial text is input into the conversational speech generation model, the conversational speech generation model is obtained, and the final multiple rounds of conversational speech are generated. The initial text can be obtained by uploading a document or by obtaining input from a staff member.

[0087] Among them, Figure 7 As shown, in step S400, the initial text of the dialogue speech to be generated is obtained, the initial text is input into the dialogue speech generation model, and the corresponding multi-round dialogue speech is generated, including:

[0088] Step S401: obtaining an initial text of the dialogue speech to be generated, inputting the initial text into the text-to-semantic model, and obtaining a semantic feature sequence corresponding to each speaker.

[0089] Among them, Figure 8 As shown, in step S401, the initial text of the dialogue speech to be generated is obtained, and the initial text is input into the text-to-semantic model to obtain the semantic feature sequence corresponding to each speaker, including:

[0090] Step S4011, obtaining an initial text of the dialogue speech to be generated, inputting the initial text into a text marker to obtain a text marker sequence, wherein the text marker uses specific markers to indicate speaker transitions and interjections;

[0091] Step S4012: input the text tag sequence into the text-to-semantic model to obtain a multi-stream semantic feature sequence.

[0092] The text-to-semantic model is a sequence-to-sequence model based on an encoder-decoder structure. Its input is a text tag sequence obtained after the text is processed by a text tagger. Special tags are used to represent speaker transitions and interjections during the processing, and the output is a multi-stream semantic feature sequence. Taking two speakers as an example, a strategy is adopted when finally generating a speech feature sequence to divide the generated semantic feature sequence into two different paragraphs, each corresponding to a different speaker participating in the conversation.

[0093] The main function of a text tagger is to convert text data into a format that the model can understand and process. A text tagger is a tool or method that can split the input text data into smaller units, which are called tokens. These tokens can be words, parts of words, punctuation marks, or single characters. Tokenization is the first step in text processing, which provides the basis for subsequent tasks such as text analysis, information extraction, and model training.

[0094] The working principle of a text tokenizer usually includes: Text preprocessing: Clean and normalize the input text, such as removing irrelevant characters, converting to lowercase, etc. Split text: Split text into smaller units, i.e. tokens, according to preset rules or algorithms. These rules can be based on whitespace characters, punctuation marks, word boundaries, etc. Tokenization: Assign a unique identifier (such as an index or ID) to each token so that it can be referenced in subsequent processing.

[0095] According to different segmentation rules and application scenarios, text tokenizers can be divided into many types, such as whitespace-based tokenizers, character-based tokenizers, word-based tokenizers, and more complex algorithms such as WordPiece, Byte-Pa ir Encoding (BPE), etc. These types of tokenizers have their own characteristics:

[0096] Whitespace-based tokenizer: Simply splits the text into words based on whitespace characters (such as spaces, newlines, etc.). This approach works well for most English texts, but may not be accurate enough for other languages ​​or texts containing special symbols.

[0097] Character-based tokenizers: Segment the text into individual characters or character groups. This approach is particularly effective for processing text with unknown vocabulary or spelling errors, but may result in an excessive number of generated tokens.

[0098] Word-based tokenizers: Segment the text into words using a predefined vocabulary. This approach requires maintaining a large vocabulary and may require special handling for words that are not in the vocabulary (such as using unknown vocabulary tokens).

[0099] Complex algorithms: such as WordPiece and BPE, which balance vocabulary size and representation accuracy by iteratively merging characters or subwords to build tokens.

[0100] The multi-stream semantic feature sequence refers to converting the conversation text into multiple discrete semantic feature streams when dealing with multiple speakers or multi-round conversations, each of which represents the semantic information of an individual speaker. This method can more effectively capture complex features in the conversation, such as seamless transitions, overlapping speech, and paralinguistic behavior.

[0101] Text-to-semantics model: This is the core model for multi-stream semantic feature sequence generation. It is based on an encoder-decoder architecture, inputs a text token sequence generated by a text tagger (such as BERT), and outputs a multi-stream semantic token sequence. This model specifically focuses on conversations between two or more speakers, using a pre-trained speech tagger (such as HuBERT) to extract discrete hidden units as a sequence of semantic tokens.

[0102] Tokenization: In the text-to-semantics model, the conversation text is first divided into tokens, and special tags are added to these tokens to indicate speaker transitions, interjections, etc. Then, the encoder part receives these text tokens and encodes them into a continuous vector representation. The decoder part converts the encoded vector into a sequence of semantic tokens. For conversations with multiple speakers, the decoder generates multiple parallel sequences of semantic tokens.

[0103] Acoustic model: The stream matching-based acoustic model converts the semantic tag sequence generated by the text-to-semantic model into a frequency domain representation of the speech signal (such as a me l spectrogram). The model uses continuous normalized flow (CNF) and flow matching (FM) techniques to define the relationship between the vector field and the flow through ordinary differential equations (ODE), thereby efficiently generating high-quality speech.

[0104] Step S402: Obtain prompts from multiple speakers, input the prompts from the multiple speakers and the semantic feature sequence into the acoustic model, and generate a mixed mel-spectrogram corresponding to the semantic feature sequence.

[0105] The acoustic model is an encoder based on the conditional stream matching model. Its input is a sequence of semantic features and multiple speaker prompts. The specific form of the speaker prompt is the Mel spectrogram corresponding to a short speech segment of each speaker.

[0106] The training of acoustic models relies on a large amount of feature data and corresponding annotations to learn the connection between acoustic features and speech units. The training process is usually divided into two stages: training and alignment.

[0107] Training phase: Use a large amount of annotated speech data to extract the acoustic features of speech signals. Commonly used acoustic features include MFCC (Mel Frequency Cepstrum Coefficient), Mel Spectrum, etc. These features are usually extracted from speech signals by feature extraction modules. They are multi-dimensional vectors, and the values ​​can be discrete or continuous.

[0108] Alignment phase: Use the annotated speech data and the existing initial model to align the features of the speech signal and the corresponding speech units. The alignment process calculates the alignment relationship between the features and the speech units and updates the parameters of the acoustic model. This process is repeated many times until the model converges.

[0109] In terms of modeling methods, acoustic models can adopt a variety of algorithms and model structures, such as hidden Markov model (HMM), Gaussian mixture model (GMM), deep neural network (DNN), etc. Among them, HMM is the mainstream method of early acoustic models, which describes the temporal characteristics of speech signals by defining the transition probability and output probability between states. With the development of deep learning, DNN and its variants (such as convolutional neural network CNN, recurrent neural network RNN, etc.) have gradually become the mainstream methods of acoustic models. They can automatically learn complex features in speech signals and improve recognition accuracy.

[0110] Step S403: input the mixed mel-spectrogram into the vocoder, and synthesize corresponding multi-round dialogue speech according to the output of the vocoder.

[0111] The text corresponding to the generated speech is input, and the text is sequentially encoded and decoded to generate a semantic feature sequence for each speaker. The acoustic model combines multiple speaker prompts to convert the semantic feature sequence output by the semantic model into a mixed Mel spectrogram. Finally, the vocoder synthesizes multiple rounds of conversational speech from the Mel spectrogram.

[0112] Compared with the prior art, the speech generation method of the embodiment of the present invention has the following technical advantages:

[0113] It is capable of simultaneously predicting multi-stream semantic features, with each semantic feature sequence corresponding to a speaker. The conditional stream matching acoustic model based on multiple speakers can generate mixed Mel spectrograms under multiple context conditions, thereby simultaneously generating multi-round conversational speech containing multiple speakers, without the need to generate speech for each speaker and then splice them.

[0114] Conversational speech is not only similar to real human speech in terms of naturalness and coherence, but also involves multiple rounds of dialogue with multiple speakers, where the words of one speaker can be seamlessly mixed with the words, interjections or laughter of another speaker.

[0115] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0116] Another embodiment of the present invention provides a speech generation device, which corresponds one-to-one to the speech generation method of the above embodiment. Fig. 9 As shown, the device 1 comprises:

[0117] The sample data acquisition module 100 is used to acquire speech sample data, wherein the speech sample data includes monologue speech sample data and dialogue speech sample data;

[0118] A network framework construction module 200 is used to construct a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder;

[0119] A model training module 300 is used to train the speech generation network framework based on the speech sample data to obtain a conversational speech generation model;

[0120] The conversational speech generation module 400 is used to obtain the initial text of the conversational speech to be generated, input the initial text into the conversational speech generation model, and generate corresponding multi-round conversational speech.

[0121] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0122] In one embodiment, the sample data acquisition module 100 is specifically used for:

[0123] Acquire initial monologue voice data, and preprocess the initial voice data to obtain monologue voice sample data;

[0124] Acquiring initial conversation voice data, and preprocessing the initial conversation voice data to obtain conversation voice sample data;

[0125] Based on the monologue voice sample data and the dialogue voice sample data, voice sample data is generated.

[0126] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0127] In one embodiment, the sample data acquisition module 100 is further used for:

[0128] Acquire initial monologue voice data, and cut the initial monologue voice data into monophonic samples satisfying a predetermined duration;

[0129] Mel-spectrograms and corresponding semantic feature sequences are extracted for all monophonic samples to obtain monologue speech sample data.

[0130] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0131] In one embodiment, the sample data acquisition module 100 is further used for:

[0132] Acquire first conversation voice data whose duration of initial conversation voice data is greater than a first predetermined duration, and cut the first conversation voice data into multi-channel voices whose duration is less than or equal to a second predetermined duration, wherein the multi-channel voices include at least two voices from two speakers;

[0133] Mel-spectrograms and semantic feature sequences are extracted from the multi-channel speech to obtain dialogue speech sample data.

[0134] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0135] In one embodiment, the network framework construction module 200 is specifically used for:

[0136] Constructing a text-to-semantics model, wherein the text-to-semantics model is a sequence-to-sequence model based on an encoder-decoder structure;

[0137] Constructing an acoustic model, wherein the acoustic model is an encoder based on a conditional stream matching model;

[0138] Construct a vocoder for synthesizing multi-turn conversation speech from mel-spectrograms.

[0139] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0140] In one embodiment, the conversational speech generation module 400 is specifically used for:

[0141] Acquire an initial text of the dialogue speech to be generated, input the initial text into the text-to-semantic model, and obtain a semantic feature sequence corresponding to each speaker;

[0142] Obtaining prompts from multiple speakers, inputting the prompts from the multiple speakers and the semantic feature sequence into the acoustic model, and generating a mixed mel-spectrogram corresponding to the semantic feature sequence;

[0143] The mixed mel-spectrogram is input into the vocoder, and corresponding multi-round conversation speech is synthesized according to the output of the vocoder.

[0144] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0145] In one embodiment, the conversational speech generation module 400 is further configured to:

[0146] Obtaining an initial text of the dialogue speech to be generated, inputting the initial text into a text marker to obtain a text marker sequence, wherein the text marker uses specific markers to indicate speaker transitions and interjections;

[0147] The text tag sequence is input into the text-to-semantic model to obtain a multi-stream semantic feature sequence.

[0148] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0149] The present invention provides a speech generation device that can simultaneously perform multi-stream semantic feature prediction, where each semantic feature sequence corresponds to a speaker. The generated conversational speech is not only similar to real human speech in terms of naturalness and coherence, but also involves multiple rounds of conversations among multiple speakers, thereby improving the user experience.

[0150] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as shown in Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a speech generation method.

[0151] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a speech generation method are implemented.

[0152] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0153] Acquire voice sample data, wherein the voice sample data includes monologue voice sample data and dialogue voice sample data;

[0154] Constructing a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder;

[0155] Training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model;

[0156] The initial text of the dialogue speech to be generated is obtained, and the initial text is input into the dialogue speech generation model to generate corresponding multi-round dialogue speech.

[0157] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0158] Acquire voice sample data, wherein the voice sample data includes monologue voice sample data and dialogue voice sample data;

[0159] Constructing a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder;

[0160] Training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model;

[0161] The initial text of the dialogue speech to be generated is obtained, and the initial text is input into the dialogue speech generation model to generate corresponding multi-round dialogue speech.

[0162] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0163] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0164] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0166] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0167] Conditional language such as "can," "can," "might," or "may," among other things, unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey that particular embodiments can include (while other embodiments do not) particular features, elements, and / or operations. Thus, such conditional language is also generally intended to imply that features, elements, and / or operations are anyway required for one or more embodiments or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0168] What has been described herein in this specification and the accompanying drawings includes examples that can provide speech generation methods and devices. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is apparent that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in an alternative, other embodiments of the present disclosure may be apparent from consideration of this specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples proposed in this specification and the accompanying drawings are considered to be illustrative and non-restrictive in all respects. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for the purpose of limitation.

Claims

1. A speech generation method, characterized in that , the method comprises: Acquire voice sample data, wherein the voice sample data includes monologue voice sample data and dialogue voice sample data; Constructing a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder; Training the speech generation network framework based on the speech sample data to obtain a conversational speech generation model; The initial text of the dialogue speech to be generated is obtained, and the initial text is input into the dialogue speech generation model to generate corresponding multi-round dialogue speech.

2. The speech generation method according to claim 1, characterized in that: The acquiring of voice sample data comprises: Acquire initial monologue voice data, and preprocess the initial voice data to obtain monologue voice sample data; Acquiring initial conversation voice data, and preprocessing the initial conversation voice data to obtain conversation voice sample data; Based on the monologue voice sample data and the dialogue voice sample data, voice sample data is generated.

3. The speech generation method according to claim 2, characterized in that: The step of obtaining initial monologue voice data and preprocessing the initial voice data to obtain monologue voice sample data includes: Acquire initial monologue voice data, and cut the initial monologue voice data into monophonic samples satisfying a predetermined duration; Mel-spectrograms and corresponding semantic feature sequences are extracted for all monophonic samples to obtain monologue speech sample data.

4. The speech generation method according to claim 2, characterized in that: The step of obtaining initial conversation voice data and preprocessing the initial conversation voice data to obtain conversation voice sample data includes: Acquire first conversation voice data whose duration of initial conversation voice data is greater than a first predetermined duration, and cut the first conversation voice data into multi-channel voices whose duration is less than or equal to a second predetermined duration, wherein the multi-channel voices include at least two voices from two speakers; Mel-spectrograms and semantic feature sequences are extracted from the multi-channel speech to obtain dialogue speech sample data.

5. The speech generation method according to claim 1, characterized in that: The construction of the speech generation network framework includes: Constructing a text-to-semantics model, wherein the text-to-semantics model is a sequence-to-sequence model based on an encoder-decoder structure; Constructing an acoustic model, wherein the acoustic model is an encoder based on a conditional stream matching model; Construct a vocoder for synthesizing multi-turn conversation speech from mel-spectrograms.

6. The speech generation method according to claim 5, characterized in that: The obtaining of the initial text of the to-be-generated dialogue speech, inputting the initial text into the dialogue speech generation model, and generating corresponding multi-round dialogue speech includes: Acquire an initial text of the dialogue speech to be generated, input the initial text into the text-to-semantic model, and obtain a semantic feature sequence corresponding to each speaker; Obtaining prompts from multiple speakers, inputting the prompts from the multiple speakers and the semantic feature sequence into the acoustic model, and generating a mixed mel-spectrogram corresponding to the semantic feature sequence; The mixed mel-spectrogram is input into the vocoder, and corresponding multi-round conversation speech is synthesized according to the output of the vocoder.

7. The speech generation method according to claim 6, characterized in that: The step of obtaining the initial text of the dialogue speech to be generated, inputting the initial text into the text-to-semantic model, and obtaining a semantic feature sequence corresponding to each speaker includes: Obtaining an initial text of the dialogue speech to be generated, inputting the initial text into a text marker to obtain a text marker sequence, wherein the text marker uses specific markers to indicate speaker transitions and interjections; The text tag sequence is input into the text-to-semantic model to obtain a multi-stream semantic feature sequence.

8. A speech generating device, characterized in that: The device comprises: A sample data acquisition module, used to acquire speech sample data, wherein the speech sample data includes monologue speech sample data and dialogue speech sample data; A network framework construction module, used to construct a speech generation network framework, wherein the speech generation network framework includes a text-to-semantic model, an acoustic model, and a vocoder; A model training module, used to train the speech generation network framework based on the speech sample data to obtain a conversational speech generation model; The conversational speech generation module is used to obtain the initial text of the conversational speech to be generated, input the initial text into the conversational speech generation model, and generate corresponding multi-round conversational speech.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the speech generation method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the one or more processors can execute the steps of the speech generation method described in any one of claims 1-7.

Images