Speech synthesis method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-synthesized text and a reference audio; dividing the to-be-synthesized text into a plurality of text segments by adopting a sliding window algorithm; generating an audio waveform corresponding to each text segment based on the reference audio and a pre-trained speech synthesis model; and performing speech synthesis on the audio waveform corresponding to each text segment to obtain a target speech. Through the mode, the input text is segmented by adopting the sliding window algorithm to form a plurality of smaller and easier-to-manage text segments, and the plurality of text segments and the parameter audio are input into the speech synthesis model, so that the speech synthesis model generates the audio sentence by sentence based on the shorter text segments; therefore, the model can focus on accurate voice and intonation details of each segment, the accuracy and natural fluency of audio output are ensured, the degradation problem caused by the fact that the text length exceeds a typical length parameter is solved, and the stability of voice generation is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, device, equipment and medium. Background Art

[0002] Speech synthesis is a technology that converts text information into speech information, that is, converts text information into any audible speech. In recent years, with the rapid development of deep learning, a variety of deep learning-based speech synthesis models have emerged. Speech synthesis models need to generate corresponding speech signals based on the input text. However, due to the limitations of the model's architecture and computational complexity, when the model processes text that exceeds typical length parameters, information loss or misunderstanding may occur, which may cause the model to become unstable when generating speech. Summary of the invention

[0003] The present invention provides a speech synthesis method, apparatus, computer equipment and medium to solve the technical problem that a speech synthesis model is limited by the architecture and computational complexity of the model, and information loss or misunderstanding may occur when processing text exceeding typical length parameters, which may cause the model to become unstable when generating speech.

[0004] In a first aspect, a speech synthesis method is provided, comprising:

[0005] Obtain the text to be synthesized and the reference audio;

[0006] The sliding window algorithm is used to divide the text to be synthesized into multiple text segments;

[0007] Generate the audio waveform corresponding to each text segment based on the reference audio and pre-trained speech synthesis model;

[0008] Speech synthesis is performed on the audio waveform corresponding to each text segment to obtain the target speech.

[0009] In a second aspect, a speech synthesis device is provided, comprising:

[0010] An acquisition module, used to acquire the text to be synthesized and the reference audio;

[0011] A division module, used for dividing the text to be synthesized into multiple text segments by using a sliding window algorithm;

[0012] A first generation module, used to generate an audio waveform corresponding to each text segment based on reference audio and a pre-trained speech synthesis model;

[0013] The second generation module is used to perform speech synthesis on the audio waveform corresponding to each text segment to obtain the target speech.

[0014] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method are implemented.

[0016] In the scheme implemented by the above-mentioned speech synthesis method, device, computer equipment and storage medium, first, a sliding window algorithm is used to segment the input text to form several smaller and more manageable text segments. Thereafter, several text segments and parameter audio are input into a speech synthesis model, so that the speech synthesis model generates audio sentence by sentence based on shorter text segments, so that the model can focus more on the precise voice and intonation details of each segment, ensuring the accuracy and natural fluency of the audio output, solving the degradation problem caused by the text length exceeding the typical length parameter, and improving the stability and effectiveness of speech generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is a schematic diagram of an application environment of a speech synthesis method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a speech synthesis method according to an embodiment of the present invention;

[0020] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0021] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0022] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S31;

[0023] Figure 6 yes Figure 4 A schematic flow chart of a specific implementation of step S32;

[0024] Figure 7 is a schematic diagram of the structure of a speech synthesizer in a speech synthesis module in one embodiment of the present invention;

[0025] Figure 8 is a schematic block diagram of performing speech synthesis using a speech synthesis model in one embodiment of the present invention;

[0026] Fig. 9 is a structural schematic diagram of a speech synthesis device in one embodiment of the present invention;

[0027] Fig.10 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Fig.11 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the client communicates with the server through the network. The user inputs the text to be synthesized and the reference speech on the client. After the server receives the text to be synthesized and the reference speech sent by the client, it uses the sliding window algorithm to segment the text to be synthesized in units of sentences to obtain a shorter text segment sequence, and uses the pre-trained speech synthesis model to extract features from the text segment sequence and the reference speech, extracts the acoustic features for speech synthesis, and obtains the target speech.

[0031] It should be noted that the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The client and the server can be connected via Bluetooth, USB (Universal Serial Bus) or other communication connection methods, which are not limited in the embodiments of the present invention. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 As shown, Figure 2 A flowchart of a speech synthesis method provided by an embodiment of the present invention includes the following steps:

[0033] S10: Obtain the text to be synthesized and the reference audio;

[0034] First, the text to be synthesized and the reference audio are obtained. The text to be synthesized refers to the long text content that needs to be converted into speech. The text to be synthesized can be determined according to actual conditions, and the embodiment of the present invention does not make specific restrictions on this.

[0035] Reference audio refers to an audio sample used as a reference, which usually contains information such as the speaker's voice characteristics, intonation, emotion, speaking speed, etc. The reference audio can be a recording, a music clip, a voice signal, etc., which is not specifically limited in the embodiment of the present invention.

[0036] For example, in financial application scenarios, the input text can be news text, and the reference audio can be the voice of a news anchor, thereby generating a professional news report; or the input text can be a conversation text, and the reference audio can be the voice of a customer service representative, thereby generating a friendly, customer-oriented voice response. In medical application scenarios, a doctor's diagnosis text is input, and a reference audio of a professional doctor is provided, thereby generating a serious and steady diagnosis voice.

[0037] S20: using a sliding window algorithm to divide the text to be synthesized into multiple text segments;

[0038] In the related art, the acquired text to be synthesized is usually directly input into the speech synthesis module for text feature extraction. However, due to the limitations of the model architecture and computational complexity, if the text information of the text to be synthesized is long, the model will be overloaded during the reasoning process, which may cause the model to pause, repeat or unnatural transition when generating speech. Based on the above problems, the present application proposes to use a sliding window algorithm to segment the input text after acquiring the text to be synthesized to generate several shorter text segments.

[0039] In this way, the model can process each short sentence more finely and reduce the distortion and noise caused by processing long texts.

[0040] It should be noted that the sliding window algorithm is intended to capture and analyze data segments or subsequences of different lengths. A fixed-size window is defined and the window is slid on the data sequence to extract and analyze the subsequence. Each time the window is slid, the data in the window is processed, and then the window position moves to the right by a predefined step size. This process is repeated until the entire sequence is processed. In one embodiment of the present application, Figure 3 As shown, a text segmentation scheme to be synthesized is provided. In S20, the text to be synthesized is divided into multiple text segments using a sliding window algorithm, which specifically includes the following steps S21-S22:

[0041] S21: Mark the starting position and the ending position of the text to be synthesized;

[0042] S22: Place the window at the starting position, extract a text segment of a preset window size as the initial text segment, slide the window to the right according to a preset sliding step, extract the next text segment of a preset window size, repeat the above process until sliding to the end position of the text to be synthesized, and obtain multiple text segments.

[0043] For steps S21-S22, after obtaining the text to be synthesized, in order to convert it into a vector representation, the text information needs to be divided into several sentences. In the related art, the text information is usually divided into multiple whole sentences according to conventional separators such as ".!?", and then feature extraction is performed on the divided whole sentences. However, this division method is too rough. Affected by the complexity of the text, some of the divided text sentences are longer and may exceed the model length limit, which may cause rhythm (such as pauses, intonation) distortion and affect the naturalness and emotional expression of the voice. In addition, when the sentence is too long, the model may make mistakes when processing certain complex structures, resulting in inaccurate pronunciation. Based on the above problems, the present application proposes to predefine a fixed sliding window size (i.e., the number of words in the text segment) and a sliding step size (i.e., the distance moved each time the slide is performed). Mark the starting and ending positions of the text to be synthesized, place the window at the starting position of the text to be synthesized, extract a text segment of a preset window size as the initial text segment, slide the window to the right according to the preset sliding step size, extract the next text segment of the preset window size, and repeat this process until it slides to the end position of the text to be synthesized.

[0044] Afterwards, the completeness and coherence of the multiple text segments are checked to ensure that each text segment is semantically reasonable and coherent.

[0045] Optionally, the preset window size and the preset sliding step size are set according to the requirements of the task and the complexity of the text. The preset window size range generally selects a few words, such as three to five words. By setting an appropriate window size, the sliding window algorithm can maintain certain contextual information between paragraphs, thereby reducing the information loss caused by paragraph segmentation. This can improve the naturalness of the synthesized text. Furthermore, the preset sliding step size can be equal to the window size, so that the words in each window maintain the contextual relationship of the original text in order, and there is no overlap between windows.

[0046] In the above manner, the sliding window algorithm can be used to effectively divide the text to be synthesized into multiple shorter text segments, ensuring that the length of each text segment does not exceed the maximum processing limit of the speech model, avoiding truncation and processing errors, and ensuring the consistency and accuracy of the synthesis effect. At the same time, only small segments of text are processed each time, which helps to improve computing efficiency.

[0047] S30: Generate an audio waveform corresponding to each text segment based on the reference audio and the pre-trained speech synthesis model.

[0048] After the text to be synthesized is divided, several divided text segments and reference audio are input into a pre-trained speech synthesis model to convert the text into a vector representation to obtain text features, and acoustic features are extracted from the parameter audio. The obtained text features and acoustic features are combined to predict the target acoustic features of each text segment. Subsequently, each target acoustic feature is inversely transformed through a vocoder to obtain the audio waveform of each text segment.

[0049] In one embodiment of the present application, Figure 4 As shown, a specific target speech synthesis scheme is provided. In S30, based on the reference audio and the pre-trained speech synthesis model, an audio waveform corresponding to each text segment is generated, which specifically includes the following steps S31-S32:

[0050] S31: Perform phoneme conversion on each text segment to obtain the phoneme text corresponding to each text segment;

[0051] In practical applications, the text to be synthesized may be Chinese text or English text, or a mixed text containing Chinese and English sounds, for example, English words or phrases are interspersed in Chinese text. Among them, Chinese is a tonal speech, and tones can not only be used to express semantic differences, but also play an important role in tone and emotional expression. In order to ensure the quality of the output audio and reduce the probability of tone errors in the speech synthesis process, this application proposes to perform phoneme conversion on the text in each text segment. If it is Chinese, each word is converted into pinyin with tones. If it is English, the English words are converted into corresponding phonetic symbols. Afterwards, the converted phoneme results are spliced ​​into the final phoneme text.

[0052] By converting text into phoneme text in the above manner, fine control over pronunciation is provided, and ambiguity between text and speech is eliminated, so that speech synthesis can focus more on the characteristics of phonemes rather than the characteristics of the entire word, thereby improving the accuracy of speech recognition.

[0053] In one embodiment of the present application, Figure 5 As shown, a specific scheme for converting text into phoneme text is provided. In S31, each text segment is converted into phonemes to obtain the phoneme text corresponding to each text segment, which specifically includes the following steps S311-S315:

[0054] S311: for any text segment, using a regular expression to distinguish the language of characters in the text segment, and marking the Chinese characters and / or English characters in the text segment;

[0055] S312: Using a word segmentation tool, performing word segmentation processing on the marked text characters in the text segment;

[0056] For steps S311-S312, since the pronunciation rules of Chinese and English are very different, there are many differences between Chinese phonemes and English phonemes. By distinguishing Chinese and English, it can be ensured that each character in the text can be pronounced correctly to avoid clarity problems caused by phoneme confusion. Specifically, for any text segment, all Chinese characters in the text segment are found and marked by regular expressions, and all continuous English characters are found and marked to achieve language distinction of the text segment. It should be noted that for Chinese, the same Chinese character may have different pronunciations in different words. In order to ensure the accuracy of the pronunciation of the synthesized language, a word segmentation tool (such as the jieba library in Python) is used to segment the Chinese characters marked in the text segment to cut the Chinese text into independent words.

[0057] Optionally, English characters in a text segment are usually separated into English words by spaces. To ensure the accuracy of subsequent phonetic notation, the English text may be normalized to remove illegal characters, convert all letters in the English text to lowercase, and expand and restore English abbreviations.

[0058] S313: performing pinyin annotation on the Chinese characters after word segmentation in the text segment to obtain the pinyin characters corresponding to the Chinese characters;

[0059] S314: annotate the English characters in the text segment with phonetic symbols to obtain phonetic characters corresponding to the English characters;

[0060] S315: Generate a phonetic text of the text segment based on the pinyin characters and the phonetic characters.

[0061] For steps S313-S315, a suitable tool (such as a Python library) is selected to determine the pinyin and tone of each Chinese character according to the Chinese phrase formed by word division to obtain the pinyin character corresponding to each Chinese character. At the same time, a suitable pronunciation dictionary (such as the Carnegie Mellon University (CMU) pronunciation dictionary) is selected to annotate the English words in the text segment with phonetic symbols. Finally, the annotated pinyin characters and phonetic characters are merged to form a phoneme text corresponding to the text segment.

[0062] For example, the Chinese characters are "现在去银行" (Now go to the bank), which are divided into multiple phrases "现在" (Now), "去" (go), and "银行" (bank). Using a Python library, phonetic annotation is performed on them to obtain the phonetic characters "xiàn zài qù yín háng". The Chinese characters are "行走在路上" (Walking on the road), which are divided into multiple phrases "行走" (walking), "在" (on), and "路上" (road). Among them, the tone of the character "行" in the phrase "行走" is different from that in "银行". After phonetic annotation, the phonetic characters "xíng zǒu zài lù shàng" are obtained. In addition, if the English characters are "Hello world", phonetic symbol annotation is performed on them through a pronunciation dictionary to obtain the pronunciation characters

[0063] S32: Co-input multiple phoneme texts and reference audio into a pre-trained speech synthesis model to obtain the audio waveform corresponding to each text segment.

[0064] Among them, the converted multiple phoneme texts and the reference audio are co-input into the pre-trained speech synthesis model. The text encoder and audio encoder of the speech synthesis model are used to extract text feature vectors and audio feature vectors from each phoneme text and reference audio respectively. Subsequently, the text features and audio features are combined and input into the speech synthesizer to obtain the predicted target acoustic features. Finally, a vocoder is used to convert the predicted target acoustic features into the audio waveform corresponding to each text segment.

[0065] In an embodiment of the present application, as Figure 6 shown, a specific speech synthesis scheme is provided. In S32, that is, multiple phoneme texts and reference audio are co-input into the pre-trained speech synthesis model to obtain the audio waveform corresponding to each text segment, which specifically includes the following steps S321 - S324:

[0066] S321: Input each phoneme text into the text encoder to generate text feature vectors;

[0067] S322: Input the reference audio into the audio encoder to generate audio feature vectors;

[0068] S323: Input the text feature vectors and audio feature vectors into the speech synthesizer to obtain the target acoustic features corresponding to each text segment;

[0069] S324: Use a vocoder to convert the target acoustic features into audio waveforms.

[0070] For steps S321-S324, the speech synthesis model includes a text encoder, an audio encoder, a speech synthesizer, and a vocoder. The phoneme text corresponding to each text segment is input into the text encoder to capture the semantic information and pronunciation information of the text and obtain a text feature vector. At the same time, the reference audio is input into the audio encoder to capture the acoustic features of the audio (such as pitch, timbre, etc.) from the audio data and obtain an audio feature vector.

[0071] Afterwards, the text feature vector and the audio feature vector are input into a speech synthesizer to combine the text feature vector and the audio feature vector to generate a mel-spectrogram corresponding to each text segment, which is the target acoustic feature. Finally, a vocoder is used to convert the target acoustic feature into an audio waveform.

[0072] S40: Perform speech synthesis on the audio waveform corresponding to each text segment to obtain the target speech.

[0073] After the audio waveform of each text segment is generated, the generated audio waveforms are concatenated one by one in the order of several text segments to form the target speech corresponding to the input text.

[0074] Optionally, after the splicing is completed, the target voice is checked to ensure that the start and end time of each audio segment are aligned to avoid time errors during splicing and to ensure that the audio at the splicing location is smooth and without abruptness.

[0075] In one embodiment of the present application, a specific training scheme for a speech synthesis model is provided. Before S10, that is, before generating an audio waveform corresponding to each text segment based on the reference audio and the pre-trained speech synthesis model, the following steps S11-S12 are also included:

[0076] S11: Based on the first voice collection parameter, obtain a plurality of first Chinese voice data and a plurality of first pinyin phoneme data corresponding thereto;

[0077] S12: Using the plurality of first Chinese speech data and the plurality of first pinyin phoneme data as training data to train the speech synthesis model, and obtaining a trained speech synthesis model.

[0078] For steps S11-S12, since the speech synthesis model is modified based on the LauraGPT model, in the related art, the LauraGPT model is usually pre-trained based on English samples and is only applicable to English text-to-speech synthesis, and the model applicability is poor. Based on the above problems, this application proposes to collect a large amount of Chinese speech as sample data to train the model, so that the model can perform mixed language text-to-speech synthesis, effectively improving the flexibility and adaptability of the model.

[0079] Specifically, the first voice acquisition parameter refers to the set voice acquisition duration and voice type based on the task requirements of model training. According to the first voice acquisition parameter, a large amount of first Chinese voice data is collected, and text conversion is performed for each first Chinese voice data, providing corresponding text content, and using a suitable pinyin conversion tool to convert all the content in the text into first Chinese pinyin phonemes represented by pinyin and tones. Afterwards, a group of training sample pairs is formed based on each first Chinese voice data and its corresponding first pinyin phoneme data, and all training sample pairs are divided into training sets, verification sets and test sets to train the speech synthesis model, and finally obtain the trained speech synthesis model.

[0080] Optionally, the first voice collection parameter includes the voice collection time and the voice collection type. In this application, in order to train a high-quality speech synthesis model, the Chinese voice data to be collected needs to have high clarity and grammatical standardization. The voice collection type is broadcast voice (such as news, weather forecast, etc.) and reading voice (such as reading articles). This type of voice is clear, the pronunciation is accurate, and the speech speed is stable. The range of voice collection time can be set to 100,000 hours to 150,000 hours.

[0081] Optionally, in a text-to-speech (TTS) model, the speech synthesizer is a key component for converting input text features into acoustic features so that these acoustic features can then be converted by a vocoder into a final audio waveform. The related art adopts a TTS (Text-to-Speech Synthesis) model based on LLMs (Large Language Models), which shows remarkable capabilities in zero-sample generalization, allowing the speech synthesis model to generate speech for unseen speakers without additional training. Among them, VALL-E uses the encoder of Encodec to process waveform features, and uses the speech synthesis component to predict output tokens, and then the decoder of Encodec generates an audio waveform. However, the use of large-scale speech models like these is often limited by limited computing resources and financial costs, making them unsuitable for personal devices with low computing power and small GPU memory. Based on the above problems, the present application proposes to form a compact and lightweight speech synthesis component by reducing the size of the decoder Transformer component only. As Figure 7The figure shows a schematic diagram of the structure of the speech synthesizer, in which the lightweight LLM (Large Language Model) as the backbone of the speech synthesis model is composed of multiple decoder modules, each of which contains a forward propagation neural network and masked self-attention. The preset architecture parameters of the Transformer component are 6 layers. By reducing the number of layers of the Transformer component from 12 to 6, shrinking the Transformer component with only the decoder module, and forming a lightweight LLM as the backbone of the speech synthesis model, the number of parameters of the component is reduced from 40 million to 20 million, achieving a significant reduction. This size reduction ensures that the model is effective and efficient even under resource-constrained conditions.

[0082] In the actual application scenario, an open source pre-trained codec from the Funcodec toolkit is used as a speech synthesizer. This codec uses 32 codebooks, downsamples the original audio by 640 times, and supports Chinese and English speech at a sampling rate of 16KHz. The codec is a variational autoencoder (VAE) that uses residual vector quantization (RVQ) to generate multiple parallel discrete potential representation sequences. Trained on a large-scale dataset, it performs very robustly in a variety of scenarios. Compared with Encodec and SoundStream, the codec uses amplitude spectrum loss to enhance mid- and high-frequency signals. The codec also integrates structured dropout to smooth the code space and adapt to different bandwidths within one model. Unlike random initialization, it uses k-means clustering to initialize the codebook. In addition, the codec maintains high utilization of the codebook through exponential moving average and has a dead code elimination mechanism to ensure efficient utilization of the codebook.

[0083] Through the above method, the phonemes and tones of Chinese speech are used as training data for the speech synthesis model to obtain a speech synthesis model that meets the needs of Chinese speech synthesis prediction, so that the trained speech synthesis model can perform mixed language text-to-speech synthesis.

[0084] In actual application scenarios, such as Figure 8As shown, it is a schematic block diagram of speech synthesis using a speech synthesis model. In the inference phase, the reference audio is input into the audio encoder to generate a series of audio tokens. At the same time, any accompanying text data is segmented into multiple text blocks by a sliding window, converted into phoneme text, and then processed by a text segmenter to generate a text feature vector. The lightweight LLM backbone (i.e., speech synthesizer) uses the audio tokens to predict the next audio token in the sequence. Once the LLM completes its prediction, the encoder vocoder converts the predicted audio tokens into continuous audio waveforms (i.e., target acoustic features), and then aggregates the audio waveforms of multiple text blocks to form the target speech. The method of synthesizing speech using a lightweight LLM and combining the method of segmenting text with a sliding window is implemented. By decomposing the text and generating audio one by one in the order of short sentences, the model focuses on the precise language and intonation details of each segment, and effectively processes long input texts with minimal training data and computing resources, especially for texts that exceed typical length parameters, solves the degradation problem of LLM processing long sequences, and improves the stability and effectiveness of speech generation.

[0085] It should be noted that since the first Chinese voice data are all standard voice data, in actual applications, the speech synthesis model may be used in a dialogue scenario, and the spoken voice data in the dialogue contains more details and changes in natural dialogue, as well as voices such as laughter and coughing. Therefore, the speech synthesis model trained only with standardized voice has limitations. Therefore, in one embodiment of the present application, an optimization scheme for a speech synthesis model is provided. After S12, that is, multiple first Chinese voice data and multiple first pinyin phoneme data are used as training data to train the speech synthesis model. After obtaining the trained speech synthesis model, the following steps S13-S14 are also included:

[0086] S13: Based on the second voice collection parameter, obtain a plurality of second Chinese voice data and a plurality of second pinyin phoneme data corresponding thereto;

[0087] S14: Optimize the pre-trained speech synthesis model using the plurality of second Chinese speech data and the plurality of second pinyin phoneme data as training data to obtain an optimized speech synthesis model.

[0088] For steps S13-S14, the second voice collection parameters are parameters set based on the task requirements of spoken voice training, including voice collection type and voice collection duration. The voice collection type is conversational voice collected in a variety of natural scenes (such as shopping, travel, work, study, etc.), and the voice data includes non-language sounds such as laughter and coughing. Secondly, after the model has been trained with standard voice, spoken voice training is only a quick adaptation and adjustment on this basis, without the need for a comprehensive training process again. Therefore, the voice collection duration can range from 500 hours to 600 hours. Through short-term targeted training, the model can learn and master spoken features faster, such as intonation changes, speech speed adjustment, etc., while saving computing resources.

[0089] According to the second voice acquisition parameters, a large amount of second Chinese voice data is collected, and text conversion is performed for each second Chinese voice data, corresponding text content is provided, and a suitable pinyin conversion tool is used to convert all the content in the text into second Chinese pinyin phonemes represented by pinyin and tone. Thereafter, a set of training sample pairs is formed based on each second Chinese voice data and its corresponding second pinyin phoneme data, and all training sample pairs are divided into a training set, a validation set, and a test set to optimize the speech synthesis model, and finally obtain an optimized speech synthesis model.

[0090] Through the above method, the speech synthesis model can combine the advantages of standardization and colloquialism, making the speech synthesis model more robust in various complex speech environments and reducing errors and distortions.

[0091] It can be seen that in the above scheme, first, the sliding window algorithm is used to segment the input text to form several smaller and more manageable text segments. Thereafter, the several text segments and parameter audio are input into the speech synthesis model, so that the speech synthesis model generates audio sentence by sentence based on shorter text segments, so that the model can focus more on the precise voice and intonation details of each segment, ensuring the accuracy and natural fluency of the audio output, solving the degradation problem caused by the text length exceeding the typical length parameter, and improving the stability and effectiveness of speech generation.

[0092] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0093] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one to the speech synthesis method in the above embodiment. Fig. 9 As shown, the speech synthesis device includes: an acquisition module 101, a division module 102, a first generation module 103 and a second generation module 104. The functional modules are described in detail as follows:

[0094] An acquisition module 101 is used to acquire the text to be synthesized and the reference audio;

[0095] A division module 102, used for dividing the text to be synthesized into a plurality of text segments by using a sliding window algorithm;

[0096] A first generating module 103, configured to generate an audio waveform corresponding to each text segment based on a reference audio and a pre-trained speech synthesis model;

[0097] The second generation module 104 is used to perform speech synthesis on the audio waveform corresponding to each text segment to obtain the target speech.

[0098] In one embodiment, the division module 102 is specifically configured to:

[0099] Mark the starting and ending positions of the text to be synthesized;

[0100] Place the window at the starting position, extract a text segment of a preset window size as the initial text segment, slide the window to the right according to the preset sliding step, extract the next text segment of the preset window size, repeat the above process until sliding to the end position of the text to be synthesized, and obtain multiple text segments.

[0101] In one embodiment, the first generating module 103 is specifically configured to:

[0102] Perform phoneme conversion on each text segment to obtain the phoneme text corresponding to each text segment;

[0103] Multiple phoneme texts and reference audio are input into the pre-trained speech synthesis model to obtain the audio waveform corresponding to each text segment.

[0104] In one embodiment, the first generating module 103 is further configured to:

[0105] For any text segment, regular expressions are used to distinguish the characters in the text segment by language, and Chinese characters and / or English characters in the text segment are marked;

[0106] Use the word segmentation tool to segment the marked Chinese characters in the text segment;

[0107] Perform pinyin annotation on the Chinese characters after word segmentation in the text segment to obtain the pinyin characters corresponding to the Chinese characters;

[0108] Perform phonetic annotation on the English characters in the text segment to obtain the phonetic characters corresponding to the English characters;

[0109] Generate phonetic text of a text segment based on pinyin characters and phonetic characters.

[0110] In one embodiment, the first generating module 103 is further configured to:

[0111] Input each phoneme text into the text encoder to generate a text feature vector;

[0112] Input the reference audio into the audio encoder to generate an audio feature vector;

[0113] Input the text feature vector and the audio feature vector into the speech synthesizer to obtain the target acoustic features corresponding to each text segment;

[0114] Use a vocoder to convert the target acoustic features into an audio waveform.

[0115] In one embodiment, the acquisition module 101 is further used for:

[0116] Based on the first voice collection parameter, a plurality of first Chinese voice data and a plurality of first pinyin phoneme data corresponding thereto are obtained.

[0117] In one embodiment, the device further comprises:

[0118] The training module is used to train the speech synthesis model using a plurality of first Chinese speech data and a plurality of first pinyin phoneme data as training data to obtain a trained speech synthesis model.

[0119] In one embodiment, the acquisition module 101 is further used to acquire a plurality of second Chinese voice data and a plurality of second pinyin phoneme data corresponding thereto based on the second voice collection parameters.

[0120] In one embodiment, the training module is further used to optimize the pre-trained speech synthesis model using the plurality of second Chinese speech data and the plurality of second pinyin phoneme data as training data to obtain an optimized speech synthesis model.

[0121] The present invention provides a speech synthesis device. First, a sliding window algorithm is used to segment input text to form a number of smaller and more manageable text segments. Thereafter, the several text segments and parameter audio are input into a speech synthesis model, so that the speech synthesis model generates audio sentence by sentence based on shorter text segments, so that the model can focus more on the precise voice and intonation details of each segment, ensure the accuracy and natural fluency of the audio output, solve the degradation problem caused by the text length exceeding the typical length parameter, and improve the stability and effectiveness of speech generation.

[0122] For the specific definition of the speech synthesis device, please refer to the definition of the speech synthesis method above, which will not be repeated here. Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a speech synthesis method.

[0124] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a speech synthesis method.

[0125] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0126] Obtain the text to be synthesized and the reference audio;

[0127] The sliding window algorithm is used to divide the text to be synthesized into multiple text segments;

[0128] Generate the audio waveform corresponding to each text segment based on the reference audio and pre-trained speech synthesis model;

[0129] Speech synthesis is performed on the audio waveform corresponding to each text segment to obtain the target speech.

[0130] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0131] Obtain the text to be synthesized and the reference audio;

[0132] The sliding window algorithm is used to divide the text to be synthesized into multiple text segments;

[0133] Generate the audio waveform corresponding to each text segment based on the reference audio and pre-trained speech synthesis model;

[0134] Speech synthesis is performed on the audio waveform corresponding to each text segment to obtain the target speech.

[0135] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0137] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0138] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtain the text to be synthesized and the reference audio; Using a sliding window algorithm to divide the text to be synthesized into multiple text segments; Based on the reference audio and the pre-trained speech synthesis model, generate an audio waveform corresponding to each of the text segments; Speech synthesis is performed on the audio waveform corresponding to each of the text segments to obtain the target speech.

2. The method according to claim 1, characterized in that: The step of dividing the text to be synthesized into a plurality of text segments by using a sliding window algorithm specifically comprises: Marking the starting position and the ending position of the text to be synthesized; The window is placed at the starting position, a text segment of a preset window size is extracted as the initial text segment, the window is slid to the right according to a preset sliding step, and the next text segment of a preset window size is extracted. The above process is repeated until it slides to the end position of the text to be synthesized, thereby obtaining the multiple text segments.

3. The method according to claim 1, characterized in that The step of generating an audio waveform corresponding to each of the text segments based on the reference audio and the pre-trained speech synthesis model specifically includes: Perform phoneme conversion on each text segment to obtain the phoneme text corresponding to each text segment; The plurality of phoneme texts and the reference audio are input into the pre-trained speech synthesis model to obtain an audio waveform corresponding to each text segment.

4. The method according to claim 3, characterized in that The step of performing phoneme conversion on each text segment to obtain the phoneme text corresponding to each text segment specifically includes: For any text segment, regular expressions are used to distinguish the characters in the text segment by language, and Chinese characters and / or English characters in the text segment are marked; Using a word segmentation tool, performing word segmentation processing on the Chinese characters marked in the text segment; Perform pinyin annotation on the Chinese characters after word segmentation in the text segment to obtain the pinyin characters corresponding to the Chinese characters; Perform phonetic annotation on the English characters in the text segment to obtain the phonetic characters corresponding to the English characters; A phonetic text of the text segment is generated based on the pinyin characters and the phonetic characters.

5. The method according to claim 3, characterized in that: The speech synthesis model includes a text encoder, an audio encoder, a speech synthesizer and a vocoder. The step of inputting the plurality of phoneme texts and the reference audio into the pre-trained speech synthesis model to obtain the audio waveform corresponding to each text segment specifically includes: Input each phoneme text into the text encoder to generate a text feature vector; Inputting the reference audio into the audio encoder to generate an audio feature vector; Inputting the text feature vector and the audio feature vector into the speech synthesizer to obtain target acoustic features corresponding to each text segment; The target acoustic feature is converted into the audio waveform using the vocoder.

6. The method according to claim 1, characterized in that Before generating the audio waveform corresponding to each of the text segments based on the reference audio and the pre-trained speech synthesis model, the method further includes: Based on the first voice collection parameter, obtaining a plurality of first Chinese voice data and a plurality of first pinyin phoneme data corresponding thereto; The speech synthesis model is trained using the multiple first Chinese speech data and the multiple first pinyin phoneme data as training data to obtain a trained speech synthesis model.

7. The method according to claim 6, characterized in that After the speech synthesis model is trained by using the plurality of first Chinese speech data and the plurality of first pinyin phoneme data as training data to obtain the trained speech synthesis model, the method further includes: Based on the second voice collection parameter, obtaining a plurality of second Chinese voice data and a plurality of second pinyin phoneme data corresponding thereto; The plurality of second Chinese speech data and the plurality of second pinyin phoneme data are used as training data to optimize the pre-trained speech synthesis model to obtain an optimized speech synthesis model.

8. A speech synthesis device, characterized in that: include: An acquisition module, used to acquire the text to be synthesized and the reference audio; A division module, used for dividing the text to be synthesized into a plurality of text segments by using a sliding window algorithm; A first generating module, used for generating an audio waveform corresponding to each of the text segments based on the reference audio and a pre-trained speech synthesis model; The second generation module is used to perform speech synthesis on the audio waveform corresponding to each of the text segments to obtain the target speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images