Speech synthesis method, speech synthesis device, electronic device, storage medium

Abstract

The embodiment of the application provides a speech synthesis method, a speech synthesis device, an electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and financial technology. The method comprises the following steps: obtaining text data and speaking embedding features; performing phoneme conversion processing on the text data to obtain phoneme data; performing word-level processing on the text data to obtain preliminary word-level labels; performing phrase prediction on the preliminary word-level labels through a phrase classification model to obtain target phrase labels; inputting the phoneme data, the target phrase labels and the speaking embedding features into a preset duration prediction model to perform duration prediction, and obtaining speech duration; and performing speech synthesis according to the speech duration and preset speech phoneme embedding features to obtain target speech data. The embodiment of the application can predict the speech duration in combination with the pause between phrases and the speaking object features, the predicted speech duration is closer to the actual situation, and the synthesized target speech has better naturalness.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and financial technology, and in particular to a speech synthesis method, speech synthesis device, electronic device, and storage medium. Background Technology

[0002] Speech synthesis is a technology that converts text information into speech information. Currently, speech synthesis typically focuses primarily on the content of the text itself. However, real-world applications are complex and contain overlooked details, such as pause duration, leading to suboptimal speech synthesis results that differ from real-world scenarios. For example, in a financial scenario, when an insured person consults an intelligent agent about insurance matters, the agent provides a personalized response based on pre-set text information. Similarly, in a medical scenario, when a patient seeks medical advice, the system synthesizes speech based on the patient's description to provide a consultation response. Summary of the Invention

[0003] The main objective of this application is to propose a speech synthesis method, speech synthesis device, electronic device, and storage medium that predicts speech duration by combining pauses between phrases and features of the speaker, and then performs speech synthesis based on the duration, thereby improving the naturalness of the synthesized speech.

[0004] To achieve the above objectives, a first aspect of this application provides a speech synthesis method, the speech synthesis method comprising: Acquire text data and speech embedding features; wherein the speech embedding features are used to characterize the speech features of a preset reference speaking object; The text data is subjected to phoneme conversion processing to obtain phoneme data; The text data is processed at the word level to obtain preliminary word-level tags; The target phrase label is obtained by predicting the initial word-level label using a pre-trained phrase classification model. The phoneme data, the target phrase tag, and the speech embedding features are input into a preset duration prediction model to predict the duration of speech. Speech synthesis is performed based on the speech duration and preset speech phoneme embedding features to obtain target speech data.

[0005] In some embodiments, the phrase classification model includes: multiple fully connected layers and activation functions; the step of predicting the target phrase label by using the pre-trained phrase classification model to perform phrase prediction on the initial word-level labels includes: The initial word-level labels are input into the phrase classification model; The initial word-level labels are processed by the multi-layer fully connected layer to obtain label prediction features; The target phrase label is obtained by normalizing the label prediction features using the activation function.

[0006] In some embodiments, the step of performing word-level processing on the text data to obtain preliminary word-level tags includes: The text data is separated by a preset delimiter to obtain text tag data; The text tag data is embedded and transformed using a pre-defined language representation model to obtain representation vector data; The language representation model is used to perform sub-word-level prediction on the representation vector data to obtain sub-word-level embedding features; The sub-word-level embedding features are subjected to label conversion processing to obtain the preliminary word-level labels.

[0007] In some embodiments, the step of performing label conversion processing on the sub-word-level embedding features to obtain the preliminary word-level labels includes: The sub-word-level embedding features are aggregated and calculated to obtain candidate embedding features; The candidate embedding features are aligned to obtain the preliminary word-level labels.

[0008] In some embodiments, before performing speech synthesis based on the speech duration and preset speech phoneme embedding features to obtain target speech data, the method further includes: Feature extraction is performed on the phoneme data to obtain the original phoneme embedding features; The original phoneme embedding features and the speech embedding features are subjected to feature concatenation processing to obtain fused feature pairs; Align the original phoneme embedding features and the speech embedding features in the fused feature pair to obtain an aligned feature pair; The original phoneme embedding features in the alignment feature pair are obtained as the speech phoneme embedding features.

[0009] In some embodiments, the step of synthesizing speech data based on the speech duration and preset speech phoneme embedding features to obtain target speech data includes: The speech phoneme embedding features are upsampled based on the speech duration to obtain the phoneme features to be synthesized. The phoneme features to be synthesized are decoded to obtain the Mel spectrum of the text data; Speech synthesis is performed based on the Mel spectrum to obtain the target speech data.

[0010] In some embodiments, the phoneme conversion processing of the text data to obtain phoneme data includes: The text data is normalized to obtain normalized text data. The normalized data is segmented to obtain the dictionary data of the text data; The dictionary data is subjected to phoneme transcription processing to obtain the phoneme data.

[0011] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the speech synthesis apparatus comprising: The acquisition module is used to acquire text data and speech embedding features; The processing module is used to perform phoneme conversion processing on the text data to obtain phoneme data; The language representation module is used to perform word-level processing on the text data to obtain preliminary word-level labels; The phrase prediction module is used to predict the target phrase label by using a pre-trained phrase classification model to perform phrase prediction on the preliminary word-level label. Duration prediction module: used to input the phoneme data, the target phrase tag and the speech embedding features into a preset duration prediction model to predict the duration and obtain the speech duration; The speech synthesis module is used to synthesize speech based on the duration of the speech and preset speech phoneme embedding features to obtain target speech data.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method described in the first aspect.

[0014] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application, through adding a phrase classification model to the duration prediction model during the speech synthesis process, predicts target phrase tags in the text data. The target phrase tags represent the probability of pauses between phrases. The duration prediction model combines phonemes, preset speech embedding features, and the target phrase tags predicted by the phrase classification model to predict the duration, thus obtaining the speech duration. Speech synthesis is then performed based on the predicted speech duration and phoneme embedding features to obtain target speech data. By combining the probability of pauses between phrases and the referenced speech embedding features for duration prediction, the predicted speech duration is closer to reality, and the synthesized target speech using the predicted speech duration has better naturalness. Attached Figure Description

[0015] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 1 The flowchart of step S103 in the process; Figure 4 yes Figure 3 The flowchart of step S304 in the process; Figure 5 yes Figure 1 The flowchart of step S104 in the process; Figure 6 This is a flowchart of a speech synthesis method provided in another embodiment of this application; Figure 7 yes Figure 1 The flowchart of step S106 in the process; Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0020] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0021] Text synthesis To Speech (TTS): TTS is a text-to-speech technology. TTS generally consists of two steps: the first step is text processing, which mainly converts the text into phonemes and obtains a phoneme sequence marked with the start and end times, frequency changes, and other information for each phoneme; the second step is speech synthesis, which mainly generates speech based on the phoneme sequence, the marked start and end times, frequency changes, and other information.

[0022] Embedding: Embedding is a vector representation that uses a low-dimensional vector to represent an object, such as a word, a product, or a movie. The property of this embedding vector is that vectors that are close in distance correspond to objects with similar meanings. Essentially, embedding is a mapping from semantic space to vector space, while preserving as much of the original sample's semantic relationship as possible in the vector space; for example, two semantically similar words are also likely to be located close to each other in the vector space.

[0023] Encoder: An encoder is used to transform an input sequence into a fixed-length vector.

[0024] Decoding: Decoding is used to convert a fixed vector generated by encoding back into an output sequence; where the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0025] Phoneme: The smallest unit that makes up a syllable, defined by the natural properties of speech sounds, and analyzed based on the articulation within the syllable. Phonemes differ from language to language. For example, Chinese phonemes are based on the pronunciation rules of Pinyin, including initials and finals; while English phonemes consist of vowels, consonants, and phonetic symbols.

[0026] Sub-word level: Sub-word level is one of the common word segmentation algorithms. Common word segmentation algorithms also include character-level and word-level word segmentation algorithms. The general principle of character-level word segmentation is to segment text based on each individual Chinese character or English letter. The general principle of word-level word segmentation is to segment text based on punctuation or spaces. Sub-word level word segmentation is between character-level and word-level. Its general principle is to continuously segment and merge text data according to a preset vocabulary size, ultimately obtaining the smallest granularity of text data without losing specific semantic information.

[0027] In related technologies, most speech synthesis methods based on duration prediction rely on predicting the duration of complete sentences and then embedding the predicted duration into the speech synthesis model. This leads to discrepancies between the occurrence of pauses, the prediction of duration, and the naturalness of the synthesized speech and reality. From a practical application perspective, improving the universality and accuracy of the duration predictor has a close impact on the quality and naturalness of speech synthesis.

[0028] The study of text data found that pauses in sentences mostly occur at punctuation marks, but some pauses still occur in the middle of the sentence. These pauses do not occur between individual words or phrases, and these pauses have a certain impact on duration prediction.

[0029] Based on this, embodiments of this application provide a speech synthesis method, speech synthesis device, electronic device, and storage medium that predicts speech duration by combining pauses between phrases and features of the speaker. The predicted speech duration is closer to the actual situation, and the target speech synthesized by combining the predicted speech duration has better naturalness.

[0030] The speech synthesis method, speech synthesis device, electronic device, and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0031] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0034] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0035] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user text data, voice data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0036] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0037] Step S101: Obtain text data and speech embedding features; wherein, speech embedding features are used to characterize the speech features of a preset reference speaking object; Step S102: Perform phoneme conversion processing on the text data to obtain phoneme data; Step S103: Perform word-level processing on the text data to obtain preliminary word-level tags; Step S104: Predict the target phrase label by using a pre-trained phrase classification model to perform phrase classification on the initial word-level labels; Step S105: Input the phoneme data, target phrase labels and speech embedding features into the preset duration prediction model to predict the duration and obtain the speech duration. Step S106: Speech synthesis is performed based on speech duration and preset speech phoneme embedding features to obtain target speech data.

[0038] Steps S101 to S106 of this embodiment involve acquiring text data and speech embedding features, where the speech embedding features characterize the speech features of a preset reference speaker, such as timbre and pitch features; performing phoneme conversion on the text data to obtain phoneme data; adding a phrase classification model to the duration prediction model during speech synthesis, predicting the text data using the phrase classification model to obtain target phrase labels, which represent the probability of pauses between phrases, indicating pauses within a sentence; combining the phonemes, speech embedding features, and the target phrase labels predicted by the phrase classification model to predict the duration of the speech; and performing speech synthesis based on the predicted speech duration and speech phoneme embedding features to obtain target speech data. By combining the probability of pauses between phrases and the reference speech embedding features for duration prediction, the predicted speech duration is closer to reality, and the synthesized target speech based on the predicted speech duration has better naturalness.

[0039] It should be noted that the text data in step S101 can be Chinese characters, but it can also be other languages ​​or a combination of Chinese and other languages; the content of the text data is preset text content, and the sources of the text data include, but are not limited to: text entered by the user on the user terminal; the content of readable documents, such as e-books; the response statements obtained by the robot through deep learning algorithms when communicating with the user; product review text; and web page text content. This application does not restrict the content and source of the text data. Taking a financial scenario as an example, when an insured person consults with an intelligent agent customer service regarding insurance-related issues, the intelligent agent customer service will provide an intelligent response based on the insured person's inquiry. The content of this intelligent response can be pre-set relevant text information, which is text data. For example, the text data could be something like: "Based on your economic status and occupation, we recommend that you choose insurance package A." Taking a medical scenario as an example, when a patient undergoes an intelligent consultation, the intelligent consultation system can provide a consultation response based on the patient's description. The content of this consultation response can be pre-set relevant diagnostic information, which is text data. For example, the text data could be something like: "Based on your current description, it can be preliminarily determined that you have a lung infection."

[0040] It should be noted that a speech embedding feature is a feature extracted from the speech of a specific speaker and used to embed into a speech synthesis system, representing the linguistic features of that specific speaker. This speech embedding feature can be obtained from a speaker recognition system based on a d-vector framework. This is achieved by mapping the Mel-spectral features extracted from the speech to a fixed-dimensional hidden layer, and then extracting the features of the specific speaker from the hidden layer to obtain a d-vector vector. When performing a speech synthesis task, this speech embedding feature is the average speech embedding feature calculated from all training utterances of a given specific speaker. For example, in a financial scenario, the speech embedding feature could be the feature of a junior intelligent customer service representative, a mid-level intelligent customer service representative, or even an intelligent customer service manager. In a medical scenario, the speech embedding feature could be the feature of a junior female nurse, a mid-level male nurse, an attending physician (male), or a chief physician (female).

[0041] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S203: Step S201: Normalize the text data to obtain normalized text data; Step S202: Perform word segmentation on the normalized data to obtain dictionary data for the text data; In step S203, phoneme transcription processing is performed on the dictionary data to obtain phoneme data.

[0042] In the steps S201 to S203 illustrated in the embodiments of the present application, standardized text data is obtained by sequentially performing normalization processing and word segmentation processing on text data; then phoneme transcription processing is performed on the standardized text data to obtain phoneme data. The text data after normalization processing can reduce noise in the data, and when input to the encoder for encoding processing, the obtained original phoneme embedding features are more accurate. In subsequent operations, the accuracy of the phrase classification model and the duration prediction model can be improved, so that the predicted speech duration is closer to the actual situation, and further the target speech synthesized in combination with the predicted speech duration has better naturalness.

[0043] In step S201 of some embodiments, the normalization processing includes regularization processing, which converts text data into a text sequence in a unified format. For example, the number "666" is normalized to "liu liu liu"; the date "2022-01-01" is normalized to "er ling er er nian yi yue yi ri"; the currency "￥100" is normalized to "yi bai yuan".

[0044] In step S203 of some embodiments, performing phoneme transcription processing on the dictionary data to obtain phoneme data may be performed by transcribing the text data according to a pre-constructed phoneme comparison table. For example, the phoneme corresponding to the word "speech" in the phoneme comparison table is "yu3 yin1". When the input dictionary data contains the word "speech", the word "speech" is converted into the phoneme "yu3 yin1".

[0045] Please refer to Figure 3 , in some embodiments, step S103 may include, but is not limited to, steps S301 to S304: In step S301, the text data is segmented by a preset delimiter to obtain text token data; In step S302, embedding feature conversion is performed on the text token data through a preset language representation model to obtain representation vector data; In step S303, subword-level prediction is performed on the representation vector data through the language representation model to obtain subword-level embedding features; In step S304, label conversion processing is performed on the subword-level embedding features to obtain preliminary word-level labels.

[0046] Steps S301 to S304, as shown in this embodiment, involve segmenting text data using a preset delimiter to obtain text-tagged data. The segmentation process involves marking the beginning and end of sentences in the text data, enabling the language representation model to identify the beginning and end of sentences and different sentences based on the delimiter. Next, the text-tagged data is transformed using the preset language representation model to obtain representation vector data; that is, the marked text data is transformed to obtain embedded features, i.e., vector features. Subsequently, the representation vector data is input into the language representation model for sub-word-level prediction, i.e., the data model is segmented at the sub-word level to obtain sub-word-level embedding features. Finally, the sub-word-level embedding features are subjected to label transformation processing to obtain the preliminary word-level labels.

[0047] It should be noted that the language representation model is a modified BERT model. BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model based on the Transformers architecture and encoder, which can further enhance the generalization ability of word vector models and fully describe character-level, word-level, sentence-level, and even inter-sentence relationship features. BERT has three embedding layers: TokenEmbedding, Segment Embedding, and Position Embedding. Specifically, this language representation model has only one embedding layer, Token Embedding, which represents the tokenization of words and sentences, resulting in a vector representation of the word or sentence across the entire corpus. Tokenization is the basic unit for indexing, representing each character indexed. Since the content related to a field is reduced to only the core elements, the size of the index, as text content related to a certain field, is also reduced. Specifically, in step S302, the text tag data is tokenized and transformed into embedded features, ultimately obtaining the representation vector data of all words and sentences in the text data.

[0048] It should be noted that the preset delimiters are [CLS] and [SEP]. The [CLS] flag is placed at the beginning of the first sentence, and the representation vector C obtained by BERT can be used for subsequent classification tasks. The [SEP] flag is used to separate two input sentences. For example, if the input sentences are A and B, the [SEP] flag should be added after sentences A and B. For example, the sentence "The text data is separated by the preset delimiter to obtain text-tagged data" will be marked as "[CLS] The text data is separated by the preset delimiter [SEP] to obtain text-tagged data [SEP]".

[0049] Please see Figure 4 In some embodiments, step S304 may include, but is not limited to, steps S401 to S402: Step S401: Aggregate and calculate the sub-word-level embedding features to obtain candidate embedding features; Step S402: Align the candidate embedding features to obtain preliminary word-level labels.

[0050] Steps S401 to S402, as shown in the embodiments of this application, involve aggregating and calculating the sub-word-level embedding features, such as by averaging, to obtain features that meet the requirements, namely candidate embedding features. Subsequently, the candidate embedding features are aligned to obtain preliminary word-level labels. After feature alignment, the input into the phrase classification model can improve the noise reduction effect of the algorithm, thereby improving the prediction accuracy of the phrase classification model and making the predicted speech duration closer to the real situation.

[0051] In step S401 of some embodiments, the aggregation calculation is an average calculation. Since the predicted sub-word embeddings may have certain errors, taking the average of the predicted sub-word embeddings can reduce the error to a certain extent and reduce the impact on the prediction accuracy of the subsequent input phrase classification model.

[0052] Please see Figure 5 In some embodiments, the phrase classification model includes multiple fully connected layers and activation functions, and step S104 may include, but is not limited to, steps S501 to S503: Step S501: Input the initial word-level labels into the phrase classification model; Step S502: Perform feature extraction processing on the initial word-level labels through multiple fully connected layers to obtain label prediction features; Step S503: Normalize the predicted label features using an activation function to obtain the target phrase label.

[0053] Steps S501 to S503, as illustrated in this embodiment, involve inputting preliminary word-level labels into a pre-trained phrase classification model for phrase prediction. The preliminary word-level labels are binary phrase labels obtained after word-level processing using a language representation model. Next, feature extraction is performed on the preliminary word-level labels through multiple fully connected layers to obtain a categorical output, namely, the label prediction feature. Finally, the label prediction feature is normalized using an activation function, mapping it to a fixed interval, such as [0,1], [-1,1], to obtain the target phrase label, thus achieving phrase label classification. This target phrase label represents the probability of pauses between phrases. For example, in the sentence "I want to eat barbecue today," there can be a pause between the phrases "today" and "want to eat." Therefore, during prediction, the probability of a pause after the phrase "today" will be higher than the probability of a pause after other phrases.

[0054] It should be noted that the pre-trained phrase classification model is an unsupervised learning neural network model, containing multiple fully connected layers and activation functions, and this phrase classification model is a pre-trained model. The data used during the training of the phrase classification model is the target binary phrase labels. The training process of the phrase classification model is evaluated through a loss function, and the model is adjusted accordingly based on the loss function value to ensure the training effect of the model.

[0055] The activation function is an activation function that can perform normalization processing. It can be the Softmax activation function, the Sigmoid function, or other activation functions. In this embodiment, no restrictions are imposed.

[0056] It should be noted that, specifically, the phrase classification model has three fully connected layers, and the activation function is the Softmax activation function. During training, the phrase classification model is trained on the binary phrase labels of the training set, and the cross-entropy loss function is used during the training process. To ensure the training effect of the phrase classification model, a larger loss function value indicates greater uncertainty in the model; a smaller loss function value indicates greater determinism. The target binary phrase label is the forced-aligned phrase label. Cross-entropy loss function. It can be expressed as shown in the following formula (1): (1) Where i is the phoneme number in the phoneme data. The duration corresponding to the i-th phoneme in the phoneme embedding feature. Let be the duration of the i-th phoneme after prediction.

[0057] It should be noted that the duration prediction model in step S105 is a neural network model used to predict the duration of phonemes. It has multiple convolutional neural network layers, each followed by a normalization layer, a dropout layer, and a linear layer. Furthermore, the duration prediction model can be stacked on top of Fourier blocks on the phoneme side and trained together with a spatial fast fuzzy network model to predict the duration of each phoneme, with mean squared error (MSE) loss.

[0058] In step S105 of some embodiments, phoneme data, target phrase labels, and speech embedding features are used together as input to the duration prediction model. During the prediction process, the phoneme data is trained with the target phrase labels as constraints, and the speech embedding features are combined to predict the speech duration. The obtained speech duration takes into account the pauses between phrases and the linguistic features of the given speaker, and the predicted speech duration is closer to the reality.

[0059] Please see Figure 6 Before step S106 in some embodiments, the speech synthesis method may include, but is not limited to, steps S601 to S604. Step S601: Extract features from the phoneme data to obtain the original phoneme embedding features; Step S602: Perform feature concatenation processing on the original phoneme embedding features and speech embedding features to obtain fused feature pairs; Step S603: Align the original phoneme embedding features and speech embedding features in the fused feature pair to obtain an aligned feature pair; Step S604: Obtain the original phoneme embedding features in the alignment feature pair as speech phoneme embedding features.

[0060] In some embodiments, step S601 can be performed at any step after step S102 and before step S106, and this application embodiment does not limit this. After performing phoneme conversion processing on the text data to obtain phoneme data, feature extraction is performed on the phoneme data to obtain the original phoneme embedding features; the obtained original phoneme embedding features facilitate speech synthesis in subsequent related operations to obtain the target speech data.

[0061] Steps S601 to S604, as illustrated in this embodiment, involve performing feature mapping on the phoneme data to obtain original phoneme embedding data. This phoneme embedding feature includes temporal and frequency features of the phonemes. Next, the original phoneme embedding feature is concatenated with the speech embedding feature, and the resulting fused feature pair is input into the alignment module. The original phoneme embedding feature and the speech embedding feature in the fused feature pair are aligned to obtain an aligned feature pair. Finally, the aligned original phoneme embedding feature is extracted from the aligned feature pair as the speech phoneme embedding feature. Through alignment processing, the original phoneme embedding feature possesses the linguistic features of a preset reference speaker, resulting in speech synthesized using the speech phoneme embedding feature having a better natural feel.

[0062] It should be noted that feature extraction of phoneme data is performed through an encoder. This encoder extracts the time and frequency information of each phoneme and maps the features of each phoneme in the phoneme sequence accordingly, such as the start time, stop time, and frequency. The result is a representation vector, i.e., the phoneme embedding feature. Specifically, the phoneme data is input into a pre-defined encoder, which performs feature extraction and outputs the phoneme embedding data.

[0063] It should be noted that feature concatenation processing of the original phoneme embedding features and speech embedding features refers to concatenating the original phoneme embedding features and speech embedding features into vectors. For example, given two vectors, vector A[0,1,2,3,4] and vector B[5,6,7,8,9], concatenating them results in vector C[0,1,2,3,4,5,6,7,8,9].

[0064] It should be noted that aligning the original phoneme embedding features and the speech embedding features in the fusion feature pair means aligning and extending the original phoneme embedding features with reference to the speech embedding features. This extends the linguistic features in the referenced speech embedding features to the original phoneme embedding features, so that the original phoneme embedding features have the linguistic features of a specific speaking object. The speech synthesized subsequently using the aligned original phoneme embedding features will have a better sense of naturalness.

[0065] Please see Figure 7 In some embodiments, step S106 includes, but is not limited to, steps S701 to S703: Step S701: Upsample the speech phoneme embedding features according to the speech duration to obtain the phoneme features to be synthesized; Step S702: Decode the features of the phonemes to be synthesized to obtain the Mel spectrum of the text data; Step S703: Speech synthesis is performed based on the Mel spectrum to obtain the target speech data.

[0066] Steps S701 to S703, as illustrated in this embodiment, involve upsampling the speech phoneme embedding features based on the speech duration predicted by the duration prediction model. This upsampling process adjusts the temporal features of each phoneme in the embedded features to obtain the phoneme features to be synthesized. Specifically, the phonemes to be synthesized include at least: the speech features of a specific speaker, such as timbre, volume, and the start and end times of the phonemes. Next, the phoneme features to be synthesized are input to a decoder for decoding to generate the corresponding Mel spectrum. Finally, a vocoder performs speech synthesis based on the Mel spectrum to obtain the target speech data. The speech synthesis process references the predicted speech duration. When the speech duration is closer to the real-world situation, the synthesized speech achieves better naturalness and is closer to the actual speech of the speaker.

[0067] The speech phoneme embedding features are upsampled based on the predicted speech duration. Specifically, the start and end times of the phonemes are shifted forward or backward according to the predicted duration, thereby adjusting the duration of the phonemes. For example, if the duration of phoneme y is 0.05 seconds, and the predicted duration of phoneme y is 0.07 seconds, the duration of phoneme y is extended to 0.07 seconds after upsampling.

[0068] This application proposes a speech synthesis method that combines a duration predictor based on the probability of pauses between phrases with the characteristics of the speaker. The method incorporates pauses between phrases when predicting duration, ensuring that the predicted speech duration closely resembles the intonation and rhythm of a real speaker. The synthesized speech, after acoustic modeling, will also be closer to real-world speech, exhibiting better naturalness. This application can be applied to scenarios such as intelligent robots, intelligent customer service, and voice broadcasting. By considering pauses between phrases and incorporating the linguistic features of a specific speaker, the synthesized speech closely approximates the real speech of that speaker, improving the user experience.

[0069] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The speech synthesis apparatus includes: The acquisition module 801 is used to acquire text data and speech embedding features; Processing module 802 is used to perform phoneme conversion processing on text data to obtain phoneme data; The language representation module 803 is used to perform word-level processing on text data to obtain preliminary word-level labels; The phrase prediction module 804 is used to predict the target phrase label by using a pre-trained phrase classification model to perform phrase prediction on the initial word-level labels. Duration prediction module 805: Used to input phoneme data, target phrase labels and speech embedding features into a preset duration prediction model to predict the duration of speech and obtain the speech duration; The speech synthesis module 806 is used to synthesize speech based on the speech duration and preset speech phoneme embedding features to obtain target speech data.

[0070] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0071] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0072] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0073] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] The speech synthesis method, speech synthesis device, electronic device, and storage medium provided in this application acquire text data and speech embedding features, wherein the speech embedding features are used to characterize the speech features of a preset reference speaker, such as timbre features and pitch features; the text data is processed by phoneme conversion to obtain phoneme data; during the speech synthesis process, a phrase classification model is added to the duration prediction model, and the text data is predicted by the phrase classification model to obtain target phrase labels, which are the probability of pauses between phrases, representing pauses in a sentence; the duration prediction model combines phonemes, speech embedding features, and the target phrase labels predicted by the phrase classification model to predict the duration, obtaining the speech duration; speech synthesis is performed based on the predicted speech duration and speech phoneme embedding features to obtain target speech data. By combining the probability of pauses between phrases and the reference speech embedding features for duration prediction, the predicted speech duration is closer to reality, and the target speech synthesized based on the predicted speech duration has better naturalness.

[0076] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0077] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0080] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0081] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The speech synthesis method includes: Acquire text data and speech embedding features; wherein, the speech embedding features are used to characterize the speech features of a preset reference speaking object, and the speech embedding features are the average speech embedding features calculated from all training utterances of the reference speaking object; The text data is subjected to phoneme conversion processing to obtain phoneme data; The text data is processed at the word level to obtain preliminary word-level tags; The initial word-level labels are predicted using a pre-trained phrase classification model to obtain target phrase labels; wherein, the target phrase labels are the probability of pauses between phrases and are used to characterize pauses in a sentence. The phoneme data, the target phrase tag, and the speech embedding features are input into a preset duration prediction model to predict the duration of speech. Speech synthesis is performed based on the speech duration and preset speech phoneme embedding features to obtain target speech data; wherein, the speech phoneme embedding features are obtained after processing the phoneme data; The phrase classification model is trained based on target binary phrase labels, which are forced-aligned phrase labels. The phrase classification model includes: multiple fully connected layers and activation functions; the step of predicting the target phrase labels by using the pre-trained phrase classification model on the initial word-level labels includes: The initial word-level labels are input into the phrase classification model; The initial word-level labels are processed by the multi-layer fully connected layer to obtain label prediction features; The target phrase label is obtained by normalizing the label prediction features using the activation function. The step of synthesizing speech data based on the speech duration and preset speech phoneme embedding features to obtain target speech data includes: The speech phoneme embedding features are upsampled based on the speech duration to obtain the phoneme features to be synthesized. The phoneme features to be synthesized are decoded to obtain the Mel spectrum of the text data; Speech synthesis is performed based on the Mel spectrum to obtain the target speech data.

2. The speech synthesis method according to claim 1, characterized in that, The step of performing word-level processing on the text data to obtain preliminary word-level tags includes: The text data is separated by a preset delimiter to obtain text tag data; The text tag data is embedded and transformed using a pre-defined language representation model to obtain representation vector data; The language representation model is used to perform sub-word-level prediction on the representation vector data to obtain sub-word-level embedding features; The sub-word-level embedding features are subjected to label conversion processing to obtain the preliminary word-level labels.

3. The speech synthesis method according to claim 2, characterized in that, The step of performing label conversion processing on the sub-word-level embedded features to obtain the preliminary word-level labels includes: The sub-word-level embedding features are aggregated and calculated to obtain candidate embedding features; The candidate embedding features are aligned to obtain the preliminary word-level labels.

4. The speech synthesis method according to claim 1, characterized in that, Before performing speech synthesis based on the speech duration and preset speech phoneme embedding features to obtain target speech data, the method further includes: Feature extraction is performed on the phoneme data to obtain the original phoneme embedding features; The original phoneme embedding features and the speech embedding features are subjected to feature concatenation processing to obtain fused feature pairs; Align the original phoneme embedding features and the speech embedding features in the fused feature pair to obtain an aligned feature pair; The original phoneme embedding features in the alignment feature pair are obtained as the speech phoneme embedding features.

5. The speech synthesis method according to any one of claims 1 to 4, characterized in that, The process of performing phoneme conversion on the text data to obtain phoneme data includes: The text data is normalized to obtain normalized text data. The normalized data is segmented to obtain the dictionary data of the text data; The dictionary data is subjected to phoneme transcription processing to obtain the phoneme data.

6. A speech synthesis device, characterized in that, The speech synthesis device includes: The acquisition module is used to acquire text data and speech embedding features; wherein, the speech embedding features are used to characterize the speech features of a preset reference speaking object, and the speech embedding features are the average speech embedding features calculated from all training utterances of the reference speaking object; The processing module is used to perform phoneme conversion processing on the text data to obtain phoneme data; The language representation module is used to perform word-level processing on the text data to obtain preliminary word-level labels; The phrase prediction module is used to predict the initial word-level labels using a pre-trained phrase classification model to obtain target phrase labels; wherein, the target phrase label is the probability of a pause between phrases, used to characterize pauses in a sentence; Duration prediction module: used to input the phoneme data, the target phrase tag and the speech embedding features into a preset duration prediction model to predict the duration and obtain the speech duration; A speech synthesis module is used to synthesize speech based on the speech duration and preset speech phoneme embedding features to obtain target speech data; wherein, the speech phoneme embedding features are obtained after processing the phoneme data; The phrase classification model is trained based on target binary phrase labels, which are forced-aligned phrase labels. The phrase classification model includes: multiple fully connected layers and activation functions; the step of predicting the target phrase labels by using the pre-trained phrase classification model on the initial word-level labels includes: The initial word-level labels are input into the phrase classification model; The initial word-level labels are processed by the multi-layer fully connected layer to obtain label prediction features; The target phrase label is obtained by normalizing the label prediction features using the activation function. The step of synthesizing speech data based on the speech duration and preset speech phoneme embedding features to obtain target speech data includes: The speech phoneme embedding features are upsampled based on the speech duration to obtain the phoneme features to be synthesized. The phoneme features to be synthesized are decoded to obtain the Mel spectrum of the text data; Speech synthesis is performed based on the Mel spectrum to obtain the target speech data.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 5.

Images