A low-delay speech synthesis method, device, equipment and medium

Abstract

The present invention discloses a low-latency speech synthesis method, apparatus, medium, and device, belonging to the field of speech processing technology. The method comprises: S10: normalizing input text to obtain normalized text; S20: performing word segmentation, part-of-speech tagging, and prosodic information analysis on the normalized text to obtain a language-structured text; S30: extracting features from the language-structured text using a deep learning algorithm to obtain a text feature vector; and S40: converting the text feature vector into a speech signal using a transformer-based non-autoregressive speech synthesis model. The present invention reduces the delay time of speech synthesis and improves the naturalness and expressiveness of speech.

Description

Technical Field

[0001] The present application relates to the field of speech processing, and in particular to a low-latency speech synthesis method, apparatus, device and medium. Background Art

[0002] While traditional speech synthesis technology can convert text into speech, it often suffers from high latency in practical applications. This latency stems from multiple factors, including the complexity of model training, processing time during the speech synthesis process, and data transmission bottlenecks. High latency not only impacts the smoothness of the user experience but also limits the application of speech synthesis technology in certain scenarios requiring high real-time performance.

[0003] Furthermore, existing technologies focus on improving the quality and efficiency of speech synthesis through effective model training methods. However, in practical applications, the generalization ability of the models remains a challenge. Due to the complexity and diversity of speech data, models exhibit significant performance degradation when faced with new speech features not previously seen in the training data.

[0004] Furthermore, the collection and processing of high-quality speech data, along with the investment in high-performance computing resources, makes model training extremely time-consuming. In some application scenarios, users expect instant speech feedback. However, current speech synthesis methods cannot meet this high real-time requirement, especially when processing long sentences or complex text. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a low-latency speech synthesis method, apparatus, device, and medium to solve at least one of the above technical problems.

[0006] To achieve the above objectives, in a first aspect, a low-latency speech synthesis method is provided, comprising:

[0007] Normalize the input text to obtain normalized text;

[0008] Performing word segmentation, part-of-speech tagging, and prosodic information analysis on the standardized text to obtain a language-structured text;

[0009] Using a deep learning algorithm to extract features from the language structured text to obtain a text feature vector;

[0010] The text feature vector is converted into a speech signal using a transformer-based non-autoregressive speech synthesis model.

[0011] In a second aspect, a low-latency speech synthesis device is provided, comprising:

[0012] A text normalization processing module is used to normalize the input text to obtain normalized text;

[0013] A language structured processing module is used to perform part-of-speech tagging, word segmentation and prosodic information analysis on the standardized text to obtain a language structured text;

[0014] A feature extraction module, which uses a deep learning algorithm to extract features from the language structured text to obtain a text feature vector;

[0015] The speech synthesis module converts the text feature vector into a speech signal using a transformer-based non-autoregressive speech synthesis model.

[0016] According to a third aspect, an electronic device is provided, comprising:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the low-delay speech synthesis method as described in any one of the first aspects.

[0020] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the low-delay speech synthesis method as described in any one of the first aspects is implemented.

[0021] The above technical solution has the following beneficial technical effects:

[0022] The low-latency speech synthesis method proposed in the present invention improves the performance and efficiency of speech synthesis through innovative text processing and speech synthesis technologies. The method first performs fine normalization processing on the input text to ensure the standardization and consistency of the text; then, through word segmentation, part-of-speech tagging and prosodic information analysis, the text is converted into a highly structured language expression. By using deep learning algorithms, especially BERT-based semantic encoding and multi-task transformers, rich, multi-dimensional semantic and prosodic features can be extracted from language-structured text to generate high-quality text feature vectors. Finally, a transformer-based non-autoregressive speech synthesis model is adopted to achieve fast and natural speech signal generation through parallel decoding and dynamic adjustment of the time step. Compared with traditional methods, the present invention reduces the delay time of speech synthesis and improves the naturalness and expressiveness of speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0024] Figure 1 This is an overall flow chart of a low-latency speech synthesis method according to an embodiment of the present invention;

[0025] Figure 2 is a specific flow chart of step S10 in an embodiment of the present invention;

[0026] Figure 3 is a specific flow chart of step S20 in an embodiment of the present invention;

[0027] Figure 4 is a specific flow chart of step S30 of an embodiment of the present invention;

[0028] Figure 5 is a specific flow chart of step S40 in an embodiment of the present invention;

[0029] Figure 6 is a specific flow chart of step S50 of an embodiment of the present invention;

[0030] Figure 7 is a functional block diagram of a low-latency speech synthesis device according to an embodiment of the present invention;

[0031] Figure 8 is a schematic structural diagram of a computer system according to an embodiment of the present invention;

[0032] Figure 9 This is a detailed flow chart of a low-latency speech synthesis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Embodiments of the present invention provide a low-latency speech synthesis method, apparatus, device, and medium to achieve low-latency speech synthesis. These embodiments achieve low-latency speech synthesis by optimizing the training and generation methods of the speech synthesis model. This optimization includes improvements to the model structure, optimized processing of training data, and parallelization of the synthesis process. Through these measures, the embodiments of the present invention effectively shorten speech synthesis time and improve the system's real-time responsiveness.

[0035] The present invention also proposes a real-time speech synthesis model and device, enhancing their applicability in practical applications. This real-time speech synthesis model not only rapidly converts text into speech but also makes fine-grained adjustments to speech based on multiple dimensions, such as context, speaker characteristics, and emotional expression, generating more natural and fluent speech output. This allows speech synthesis technology to better adapt to the needs of diverse scenarios.

[0036] Traditional speech synthesis technology often suffers from significant latency during text-to-speech conversion, which not only impacts the smoothness of the user experience but also limits its application in more real-time interactive scenarios. Against this backdrop, the present invention aims to address latency issues in existing technologies and advance speech synthesis technology to a higher level.

[0037] The purpose of the embodiments of the present invention is to provide a better user experience. In real-time interactive scenarios, such as online games, remote meetings, and instant messaging, users expect immediate and natural voice feedback. Traditional speech synthesis technologies, due to their complex processing and high computational load, often struggle to meet this demand. Therefore, the embodiments of the present invention achieve low-latency speech synthesis by optimizing algorithms and streamlining model structures, thereby improving the smoothness and naturalness of the user experience.

[0038] The embodiments of the present invention help broaden the application scenarios of speech synthesis technology. With the development of science and technology, speech synthesis technology has been widely used in a variety of fields, such as intelligent customer service, voice assistants, and audiobooks. However, due to the existence of latency issues, some scenarios with high real-time requirements, such as online education and telemedicine, still find it difficult to fully utilize the advantages of speech synthesis technology. By achieving low-latency speech synthesis, the embodiments of the present invention enable speech synthesis technology to be more widely applied in various real-time interaction scenarios, providing more users with convenient and efficient voice services.

[0039] Embodiments of the present invention also aim to improve the naturalness and expressiveness of speech synthesis. While traditional speech synthesis technology can achieve basic text-to-speech conversion, the naturalness and expressiveness of speech still need to be improved. By introducing deep learning algorithms and neural network models, such as WaveNet and Tacotron, embodiments of the present invention achieve more refined modeling and generation of speech signals, making synthesized speech closer to natural speech in terms of sound quality, intonation, and speaking speed, thereby improving the naturalness and expressiveness of speech synthesis. Example 1

[0040] like Figure 1 and Figure 9 As shown, an embodiment of the present invention provides a low-latency speech synthesis method, which includes the following steps:

[0041] S10: Normalize the input text to obtain normalized text.

[0042] In this embodiment, input text normalization refers to the systematic cleaning and standardization of raw text. For example, for the input text "The weather is nice today!", redundant exclamation marks will be removed, full-width punctuation will be converted to half-width punctuation, and uppercase and lowercase characters will be standardized. For text containing internet slang, abbreviations, or special symbols, the system will convert them to standard language expressions. Specific implementations can ensure the standardization and consistency of input text through predefined regular expression rules and text cleaning algorithms, providing a standardized text foundation for subsequent processing.

[0043] S20: performing word segmentation processing, part-of-speech tagging processing, and prosodic information analysis on the standardized text to obtain a language-structured text.

[0044] In this embodiment, during the language structuring phase, the system performs multi-level language analysis on the normalized text. For example, for the phrase "The weather is nice today," the system first performs word segmentation, yielding ["today," "weather," "nice"]. A pre-trained part-of-speech tagging model then annotates each word with its grammatical attributes, such as ["today / time word," "weather / noun," "nice / adjective"]. Prosodic information analysis extracts prosodic features such as intonation, speech rate, pauses, and stress. A multi-task transformer model predicts the prosodic features of each word, providing rich language structure information for subsequent speech synthesis.

[0045] In an alternative embodiment, an alternative to the pre-trained part-of-speech tagging model may be a machine learning method. Machine learning methods still play an important role in the field of part-of-speech tagging. The Hidden Markov Model (HMM) can capture the statistical laws of part-of-speech sequences through probability transfer and emission probability. The Conditional Random Field (CRF) model can consider the global optimal tagging sequence and has good modeling capabilities for complex language structures. The maximum entropy model performs well in processing sparse features and complex contexts through feature functions and probability distributions. These methods have low computational overhead and are suitable for scenarios with limited resources.

[0046] In an alternative embodiment, an alternative to the pre-trained part-of-speech tagging model may be a combined rule-based and statistical approach. This combined rule-based and statistical approach can provide a more flexible part-of-speech tagging strategy. Corpus-based statistical methods can learn part-of-speech distribution characteristics in large-scale language data. A hybrid rule-matching and machine learning model can leverage linguistic knowledge and data-driven learning methods to improve tagging accuracy.

[0047] In an alternative embodiment, an alternative to the pre-trained part-of-speech tagging model may be a multilingual universal part-of-speech tagging model. The multilingual universal part-of-speech tagging model facilitates cross-language applications. The spaCy part-of-speech tagger supports multiple languages, has efficient processing capabilities and good scalability. The Stanford part-of-speech tagger developed by Stanford University provides a rich set of language models and tagging tools, and is widely used in academic research and industrial applications. The universal dependency analysis tool UDPipe multilingual model is based on a universal dependency grammar and can achieve cross-language part-of-speech tagging and syntactic analysis.

[0048] S30: Using a deep learning algorithm to extract features from the language structured text to obtain a text feature vector.

[0049] In this embodiment, a deep learning algorithm is used in the feature extraction stage, and the BERT (Bidirectional Encoder Representations from Transformers) model is used for semantic encoding. Taking the language structured text from the previous step as input, the BERT model will generate a deep semantic representation to capture the contextual associations between words. Through multi-dimensional feature extraction, not only grammatical and semantic association features are obtained, but also part-of-speech tagging information and prosodic association features, such as intonation changes and speaking speed features, are extracted. These multi-dimensional features are ultimately integrated and converted into a high-dimensional text feature vector, providing rich and accurate language semantic information for subsequent speech synthesis.

[0050] S40: Using a transformer-based non-autoregressive speech synthesis model, convert the text feature vector into a speech signal.

[0051] In this embodiment, during the speech synthesis stage, a transformer-based non-autoregressive model converts text feature vectors into speech signals. The speech synthesis model first normalizes the input text feature vectors and generates high-dimensional semantic feature vectors using the encoder's multi-head self-attention module. A length adjuster time-aligns the semantic feature vectors based on the predicted speech signal time step. A non-autoregressive decoder uses a parallel decoding method to rapidly generate speech signal time series. Finally, the output layer performs post-processing on the generated speech signal, including sound quality enhancement and format conversion, ultimately generating natural and fluent speech output.

[0052] This approach not only reduces speech synthesis latency, but also improves the naturalness and expressiveness of speech through deep learning technology.

[0053] like Figure 2 As shown, step S10 specifically includes:

[0054] S101: Clean the input text by removing invalid characters, redundant information, and non-printable characters to obtain clean text.

[0055] In the text cleaning stage, a multi-level text filtering strategy is designed to accurately remove invalid information in the input text. The specific implementation is to comprehensively purify the input text through predefined regular expression rules and text filtering algorithms. For example, for complex text like "Today the weather is great! @Xiaoming666", special symbols (@), numbers, line breaks ( ), and tab characters (\t) will be deleted in sequence according to the preset rules, and extra exclamation marks will be removed. Finally, the key text content "Today the weather is great" will be extracted. This process not only includes basic character filtering but also extends to handling HTML tags, web links, special encoding characters, etc., to ensure the purity and subsequent processability of the text.

[0056] S102: Perform format unification and standardization processing on the clean text to obtain normalized text.

[0057] In the format unification and standardization processing stage, comprehensive normalization conversion is implemented on the clean text. Through a series of fine text processing rules, multi-dimensional standardization of the text is carried out. First, full-width or half-width is unified, converting full-width punctuation marks, numbers, and letters to half-width form; secondly, case is unified; for Chinese text, traditional and simplified Chinese conversion can also be performed; at the same time, special words and abbreviations are normalized. For example, the colloquial expression "today" is standardized to "today". Finally, by deleting extra spaces at the beginning and end and standardizing internal spaces, highly standardized text is generated. This multi-level text normalization method not only cleans text noise but also improves the consistency and standardization of the text.

[0058] As Figure 3 shown, step S20 specifically includes:

[0059] S201: Perform word segmentation on the normalized text and obtain the word segmentation result using a word segmentation algorithm.

[0060] Specifically, in the word segmentation processing stage of step S201, a deep learning-based word segmentation algorithm is adopted to accurately segment the input normalized text. Taking the Chinese text "Today the weather is great" as an example, using a neural network word segmentation model that combines character-level and word-level features, the text can be accurately segmented into "Today / weather / great". The word segmentation algorithm uses a pre-trained language model, comprehensively considering the context, word collocations, and grammar rules to improve the accuracy of word segmentation. For ambiguous words and proper nouns, a statistical and rule-based post-processing mechanism is also introduced to further optimize the word segmentation result.

[0061] S202: Perform part-of-speech tagging on the word segmentation results, using a pre-trained part-of-speech tagging model to obtain a text sequence with part-of-speech tags.

[0062] Specifically, during the part-of-speech tagging stage, a pre-trained deep learning part-of-speech tagging model is used to accurately tag the word segmentation results. Continuing with the word segmentation result "Today / The weather / It's really good" from the previous step, each word is annotated with the corresponding part-of-speech tag, such as "Today / Time word," "Weather / Noun," and "Really good / Adjective." The part-of-speech tagging model is trained on a large-scale corpus and utilizes technologies such as Bidirectional Long Short-Term Memory (Bi-LSTM) and Conditional Random Field (CRF). This model accurately captures the grammatical features of words in different contexts, improving the accuracy and robustness of part-of-speech tagging.

[0063] S203: Extracting grammatical structure features based on the text sequence with part-of-speech tags.

[0064] During the grammatical structure feature extraction phase, the system uses a deep grammatical parsing algorithm to extract the grammatical structure features of text sequences with part-of-speech tags. For the text sequence "The weather is really nice today," the system first constructs a dependency syntactic analysis tree to identify the grammatical relationships between words. Specifically, through the Graph Neural Network (GNN) and attention mechanism, the system is able to capture dependency relationships between words, such as subject-verb-object relationships, modification relationships, etc. The extracted grammatical features include not only direct grammatical connections between words, but also higher-level grammatical abstract features.

[0065] S204: A multi-task transformer deep learning model is used, with the grammatical structure features as input, a high-dimensional feature representation containing grammatical information and contextual information is extracted through a shared coding layer, and the high-dimensional feature representation is passed as input to multiple output layers for different prosody prediction tasks, respectively, to generate a tone feature sequence, a speech rate change sequence, a pause position sequence and a stress sequence, and the tone feature sequence, the speech rate change sequence, the pause position sequence and the stress sequence are integrated into prosody information.

[0066] In this step, a multi-task learning framework with a shared encoding layer is designed. It takes grammatical structure features as input and extracts high-dimensional feature representations through the Transformer encoder. For the sentence "The weather is really nice today", the multi-task transformer deep learning model first captures the complex interactions between words through the self-attention mechanism, generating high-dimensional features that integrate grammatical and contextual information. Subsequently, this feature is passed to four specialized output layers to predict intonation, speaking rate, pauses, and stress sequences respectively. For example, the intonation prediction output layer can generate a sequence representing intonation changes, the speaking rate change output layer predicts the speaking rate transitions between words, the pause position output layer identifies natural pause points, and the stress output layer marks the stress information of keywords. Ultimately, these four sequences are integrated into comprehensive prosodic information.

[0067] S205: Combining the word segmentation result, the text sequence with part-of-speech tags, and the prosody information to generate a language-structured text.

[0068] In the final stage of language-structured text generation, word segmentation results, part-of-speech tagging sequences, and prosodic information are integrated across multiple dimensions. For the sentence "The weather is great today," the system not only retains the original word segmentation result ("today / weather / great") and the part-of-speech tags ("today / time word, weather / noun, great / adjective"), but also incorporates previously predicted prosodic information, such as intonation, speech rate sequences, pause locations, and stress markers. Through an information fusion algorithm, these multi-layered features are organically combined to generate a multi-dimensional, structured language representation. This representation not only preserves the text's grammatical and semantic information but also incorporates rich prosodic features, providing high-quality input for subsequent speech synthesis.

[0069] In the process of generating linguistically structured text, information fusion algorithms can be implemented in a variety of ways. First, simple fusion methods based on feature concatenation can directly combine word segmentation results, part-of-speech tagging, and prosodic information into a multidimensional feature vector in sequence, achieving basic integration. Second, weighted fusion linear models can be used to assign weights to different features and optimize the fusion performance by quantifying their importance. Furthermore, rule-based fusion methods logically adjust and optimize features for specific scenarios based on linguistic or phonetic rules. Furthermore, machine learning models such as decision trees or random forests can be used to learn the correlations between word segmentation, part-of-speech, and prosodic information, achieving more accurate fusion. For more complex applications, deep learning techniques (such as LSTM and Transformer) can capture the complex relationships between multi-level features through nonlinear modeling, generating higher-quality comprehensive representations. Furthermore, a multimodal information fusion framework can be adopted, treating word segmentation, part-of-speech, and prosody as multimodal data and leveraging cross-modal learning mechanisms to further optimize fusion results.

[0070] like Figure 4As shown, step S30 specifically includes the following steps:

[0071] S301: Encode the language structured text using the BERT representation model based on the bidirectional Transformer encoder to generate a text context semantic representation.

[0072] Specifically, BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. Its advantage lies in its ability to understand text semantics through bidirectional context. During the semantic encoding phase, the BERT model first performs word embedding conversion, mapping discrete text words into a continuous, high-dimensional vector space. Through a multi-head self-attention mechanism, the model is able to capture the complex contextual relationships between words and assign differentiated semantic weights to different words. The specific encoding process includes: word embedding layer, positional encoding, multi-head self-attention layer, and feedforward neural network layer. This multi-layer encoding enables the model to generate deep semantic representations, in which the vector representation of each word incorporates global context information.

[0073] Taking the phrase "Technology changes life" as an example, the BERT model first performs word embedding conversion on each word. For example, "technology," "change," and "life" are mapped into high-dimensional vectors. Using a multi-head self-attention mechanism, the model captures the semantic connections between "technology" and "change," and between "change" and "life," generating semantic representations that incorporate contextual information.

[0074] S302: Extracting multi-dimensional semantic features according to the text context semantic representation, wherein the multi-dimensional semantic features include: grammatical semantic association features and part-of-speech tagging information.

[0075] In an optional embodiment, this step may use the Tacotron encoder. Tacotron is the first truly end-to-end Text-to-Speech Synthesis (TTS) deep neural network model. The Tacotron Encoder is an end-to-end text-to-speech system based on deep learning, which can directly convert the input character text sequence into a speech signal.

[0076] In step S302, the process of extracting multi-dimensional semantic features using the Tacotron encoder is to further mine the grammatical and semantic information of the text based on the BERT semantic encoding. The role of the Tacotron encoder is to extract the grammatical and semantic association features and part-of-speech tagging information of the text, so as to help the model deeply understand the structure and content of the text. Specifically, the Tacotron encoder analyzes the dependencies and semantic roles between words in the text and identifies the grammatical structure of the text. For example, for the sentence "Technology changes life", the Tacotron encoder will extract grammatical information such as "technology" as the subject, "change" as the predicate verb, and "life" as the object, and provide part-of-speech tagging, such as "technology / noun", "change / verb", and "life / noun".

[0077] There are several advantages to using the Tacotron encoder. First, Tacotron is an end-to-end text-to-speech (TTS) model that allows the model to generate speech directly from text without relying on traditional feature engineering. Therefore, it can not only perform speech synthesis, but also effectively handle language understanding tasks. Second, the Tacotron encoder can extract rich grammatical and semantic features, and by analyzing the complex relationships between words, it can deeply explore the deep meaning of the text, thereby providing more accurate feature representations for speech synthesis and other tasks. In addition, Tacotron can efficiently integrate multi-dimensional information such as grammar, semantics, and prosody, which makes it more expressive and more accurate when processing text.

[0078] In summary, the advantage of using the Tacotron encoder lies not only in its end-to-end model design, which can eliminate the complexity of traditional feature engineering, but also in its ability to efficiently extract grammatical and semantic information through deep learning methods, providing more accurate text feature expression.

[0079] In another alternative embodiment, the BERT model and related grammatical analysis methods can be used to extract grammatical semantic association features and part-of-speech tagging information, that is, multi-dimensional semantic feature extraction is to further mine the grammatical and semantic information of the text on the basis of BERT semantic encoding. This process mainly includes two key features: grammatical semantic association features and part-of-speech tagging information. The grammatical semantic association features identify the grammatical structure and semantic connection in the text by analyzing the dependency relationship and semantic role between words. The part-of-speech tagging information provides a detailed classification of words at the grammatical level, such as nouns, verbs, adjectives, etc. These features not only stay at the surface of grammatical markers, but also deeply explore the semantic relationship and grammatical function between words.

[0080] S303: Extracting prosody-related features using a long short-term memory model based on the grammatical-semantic association features and the part-of-speech tagging information. The prosody-related features include intonation features, speech rate features, pause features, and stress features.

[0081] Specifically, prosodic feature extraction employs a Long Short-Term Memory (LSTM) model, leveraging the grammatical and semantic features and part-of-speech tagging information extracted in the previous step to deeply explore the prosodic characteristics of the text. The LSTM model selectively retains and forgets semantic information through gating mechanisms (forget gate, input gate, and output gate), effectively capturing long-range dependencies within a sequence. Prosodic features encompass four dimensions: intonation, speaking rate, pauses, and stress. These features are based not only on grammar and semantics but also take into account the dynamics of language expression and prosodic characteristics.

[0082] S304: Integrate the multi-dimensional semantic features and the prosody-related features into a text feature vector.

[0083] This embodiment can use Tacotron, and use Tacotron's decoder to combine the text features generated by Tacotron's encoder with the prosody information to gradually generate the final text feature vector. Specifically, the integration of text feature vectors is a complex process of fusing multi-dimensional semantic features and prosody-related features. The system designs a multi-layer feature fusion network, and assigns weights to different types of features through the attention mechanism, such as grammatical features, semantic features, intonation features, etc. Heterogeneous features are mapped to a unified high-dimensional feature space using fully connected layers and nonlinear activation functions. The fusion process is not a simple splicing or averaging, but a complex interaction between learning features.

[0084] For example, for the phrase "Technology Changes Life," the system uses an attention mechanism to assign different weights to features such as grammar, semantics, and intonation, and then uses a fully connected layer to map these features into a unified vector space. The resulting text feature vector is a high-dimensional, dense representation that contains the text's multidimensional information, providing feature representation for subsequent natural language processing tasks.

[0085] Specifically, in step S304, the process of using Tacotron to integrate and convert multi-dimensional semantic features and prosody-related features into text feature vectors mainly relies on Tacotron's decoder. Specifically, Tacotron's decoder is used to combine the text features generated by the Tacotron encoder with the prosody information extracted from the text to generate the final text feature vector. This process uses a multi-layer feature fusion network to assign weights to different types of features (such as grammatical features, semantic features, intonation features, etc.) through an attention mechanism to ensure that the importance of each feature in the synthesis process is reasonably reflected. The fused features are processed through a fully connected layer and a nonlinear activation function to map different types of features to a unified high-dimensional feature space. Ultimately, the generated text feature vector contains multi-dimensional information of the text and can provide effective feature support for subsequent natural language processing tasks (such as speech synthesis, text classification, etc.).

[0086] In summary, step S304 requires the use of Tacotron, especially its decoder part, to fuse multi-dimensional semantic features and prosody-related features into a text feature vector; while step S302 does not need to use Tacotron, and can use the BERT model or other natural language processing methods to extract grammatical and semantic features.

[0087] The embodiments of the present invention improve the depth and accuracy of text semantic understanding and representation by combining BERT semantic encoding, multi-dimensional semantic feature extraction, LSTM rhythmic feature analysis and multi-layer feature fusion technology. This method realizes the multi-level and multi-dimensional semantic analysis of language text for the first time. It not only breaks through the limitations of traditional single-dimensional feature extraction, but also accurately captures complex language features such as grammar, semantics, intonation, and speaking speed in the text through the synergy of deep learning models. Compared with traditional methods, the present invention can more comprehensively understand the intrinsic semantics of the text, improve the feature representation ability in natural language processing tasks, and has a stronger depth of semantic understanding and feature expression accuracy, providing more intelligent and accurate technical support for subsequent applications such as speech synthesis, machine translation, and text classification.

[0088] like Figure 5 As shown, step S40 specifically includes the following steps:

[0089] S401: receiving the text feature vector through the input layer of the speech synthesis model, normalizing the text feature vector, and outputting the normalized text feature vector.

[0090] Normalizing text feature vectors helps eliminate scale differences between different dimensions of feature vectors, ensuring the stability and effectiveness of subsequent neural network processing. Normalization uses Z-score or min-max normalization to map high-dimensional text feature vectors to a uniform numerical range. Normalization calculates the mean and standard deviation of each feature dimension, transforming the feature vector into a standard normal distribution with a mean of 0 and a variance of 1. This normalization not only eliminates scale differences between features, but also preserves the relative size and distribution characteristics of the original features.

[0091] For example, suppose a text feature vector V = [10, 50, 200, 500], where the values ​​of the dimensions vary significantly. After normalization, the mean μ and standard deviation σ are calculated, and then normalized using the formula (x - μ) / σ. The normalized vector is mapped to the range [-1, 1] or [0, 1], making the numerical scales of the features in each dimension closer, providing more stable input for subsequent neural network processing.

[0092] S402: Input the normalized text feature vector into the encoder of the speech synthesis model, and use the multi-head self-attention module inside the encoder to encode the normalized text feature vector to generate a high-dimensional semantic feature vector.

[0093] The encoder's multi-head self-attention module is a key component in speech synthesis models for capturing complex semantic relationships. By using multiple attention heads in parallel, this module can simultaneously focus on the semantic information in different subspaces of the input feature vector. Each attention head includes three projection matrices: query, key, and value. By calculating attention weights across different feature dimensions, it achieves nonlinear transformation and semantic enhancement of the input vector. The multi-head self-attention module can capture long-range dependencies and complex semantic interactions, and more accurately model the intrinsic connections between feature vectors than traditional fully connected layers.

[0094] For the normalized text feature vector, the multi-head self-attention module (assuming eight attention heads are used) concurrently calculates the attention distribution across different subspaces. For example, when synthesizing the text "Technology changes life," different attention heads focus on features at different levels, such as grammatical relationships between words, semantic associations, and intonation variations. Through weighted summation and nonlinear transformation, a richer and more semantically enhanced high-dimensional feature vector is generated.

[0095] S403: Input the high-dimensional semantic feature vector into the length adjuster of the speech synthesis model, predict the time step information of the speech signal, and perform time alignment processing on the high-dimensional semantic feature vector based on the time step information of the speech signal to generate a time-aligned high-dimensional semantic feature vector.

[0096] The length adjuster is used in speech synthesis models to achieve precise timing control. Its function is to predict the time steps of the speech signal and perform time alignment on the high-dimensional semantic feature vectors. The length adjuster employs an attention-based prediction network and uses regression or classification methods to estimate the speech duration corresponding to each text unit (e.g., syllable, word). Time alignment involves not only temporal mapping but also considers the prosodic characteristics, intonation, and natural fluency of the speech. The alignment algorithm interpolates, repeats, or deletes feature vectors to ensure that the generated speech signal accurately matches the semantic features of the input text in the temporal dimension.

[0097] For example, when synthesizing the text "Technology changes life," the length adjuster first predicts the speech duration of each word. For example, "technology" requires 0.5 seconds, "change" requires 0.3 seconds, and "life" requires 0.4 seconds. Based on these predicted time steps, the length adjuster performs time alignment on the high-dimensional semantic feature vectors. By interpolating or repeating the feature vectors, the generated speech signal ensures that the time dimension of the generated speech signal accurately corresponds to the semantic features of the text.

[0098] S404: Inputting the time-aligned high-dimensional semantic feature vector into the non-autoregressive decoder of the speech synthesis model, and generating a time series of speech signals using a parallel decoding method of the non-autoregressive decoder.

[0099] Unlike traditional autoregressive decoders, which generate speech signals incrementally, non-autoregressive decoders generate the entire speech signal time series in parallel. They utilize a convolutional neural network (CNN) or Transformer architecture, employing multiple layers of nonlinear transformations and skip connections to achieve rapid, parallel generation of speech signals. The innovation of non-autoregressive decoders lies in the introduction of techniques such as temporal masking, length prediction, and adversarial training, effectively alleviating the information loss and quality degradation associated with parallel generation.

[0100] For example, when synthesizing the speech signal for the text "Technology Changes Life," the non-autoregressive decoder feeds time-aligned high-dimensional semantic feature vectors into a multi-layer convolutional neural network. Through parallel convolution operations, the speech signal segments for each time step are generated simultaneously. For example, for a 0.5-second speech clip about "technology," the decoder can generate speech features for 50 time steps in parallel, each containing multi-dimensional feature information such as pitch, timbre, and energy.

[0101] S405: Post-processing the time series of the speech signal through the output layer of the speech synthesis model, wherein the post-processing includes sound quality enhancement and format conversion of the speech signal to generate a final playable speech signal.

[0102] In this step, WaveNet can be used to receive the time series of speech signals and generate audio waveforms one by one. Specifically, post-processing of the output layer is conducive to improving the quality and adaptability of speech signals. Sound quality enhancement technologies mainly include spectrum enhancement, noise suppression, resonance peak adjustment, etc. Through deep generative models, such as Generative Adversarial Network (GAN) or Variational Autoencoder (VAE), the spectrum, timbre, energy and other characteristics of the speech signal are carefully optimized. Format conversion processing involves converting the original speech signal into different audio coding formats, such as WAV, MP3, AAC, etc., and adjusting parameters such as bit rate and sampling rate to adapt to different playback devices and application scenarios.

[0103] For example, for the speech signal generated by "Technology Changes Life," post-processing at the output layer first uses a generative adversarial network to enhance the speech spectrum, removing artifacts and improving the naturalness and clarity of the speech. Subsequently, format conversion is performed, converting the original high-sampled speech signal into a low-bitrate MP3 format suitable for mobile devices while maintaining clarity and natural fluency. The resulting speech signal not only meets the requirements for semantic accuracy but also achieves high standards in sound quality and applicability.

[0104] Specifically, in step S405, the time series of the speech signal is post-processed by the output layer of the speech synthesis model to generate a final playable speech signal. This includes multiple sub-steps corresponding to the entire process from receiving the time series signal, generating the audio waveform, sound quality enhancement, format conversion to outputting the final speech signal:

[0105] S405-1: The output layer of the speech synthesis model receives the speech signal time series generated in the previous step. This time series contains speech features at each time step, such as pitch, timbre, and volume. By analyzing and processing these features, subsequent processing will further optimize and generate the final audio waveform.

[0106] S405-2: This step uses WaveNet to receive the time series of speech signals and generate audio waveforms one by one. As a generative model, WaveNet samples the speech signal at each time step and generates the corresponding audio waveform. Leveraging its deep neural network architecture, it generates high-quality audio waveforms and refines the details of each audio signal, ensuring the output sound is highly natural and realistic.

[0107] S405-3: After the audio waveform is generated, the next step is to optimize the speech signal using sound quality enhancement techniques to improve its quality and naturalness. These techniques include spectral enhancement, noise suppression, and formant adjustment. Deep learning models, such as generative adversarial networks (GANs) or variational autoencoders (VAEs), perform detailed spectral optimization on the speech signal, eliminating any artifacts and improving speech clarity and naturalness.

[0108] S405-4: After the sound quality enhancement process, the generated audio signal undergoes format conversion and adaptation to ensure optimal playback on different devices and playback scenarios. This includes converting the high-sampling-rate voice signal into formats suitable for mobile devices or different audio platforms, such as WAV, MP3, and AAC. Furthermore, parameters such as the audio file's bitrate and sampling rate are adjusted to suit the needs of different playback devices, ensuring sound quality and playback stability.

[0109] S405-5: Finally, the enhanced and format-converted audio signal is output, generating and outputting a playable voice signal. At this point, the voice signal is optimized in terms of sound quality, timbre, clarity, and playback compatibility, ensuring playback quality across a wide range of devices and application scenarios.

[0110] The embodiment of the present invention improves the intelligence and accuracy of speech synthesis by designing the processing flow of the speech synthesis model. First, the normalization of the text feature vector ensures the stability and effectiveness of the subsequent neural network module; second, the multi-head self-attention encoder can deeply capture the complex semantic associations and improve the depth and accuracy of the semantic representation; the introduction of the length regulator realizes the accurate prediction and alignment of the speech time step, solving the difficulty of time control in traditional speech synthesis; the parallel generation technology of the non-autoregressive decoder improves the computational efficiency of speech synthesis; finally, the sound quality enhancement and format conversion processing of the output layer further optimize the naturalness and applicability of the speech signal. Compared with traditional methods, the present invention not only makes progress in semantic understanding, time control and speech generation, but also realizes end-to-end intelligent optimization of the speech synthesis process, providing a more intelligent, efficient and natural solution for voice interaction technology.

[0111] like Figure 6As shown, before step S10, step S50 is also included: training the speech synthesis model to obtain a trained speech synthesis model;

[0112] The training process of the speech synthesis model includes the following steps:

[0113] S501: Obtain a bimodal dataset containing text data and corresponding voice data, and perform text data processing and voice data processing on the bimodal dataset; the text data processing is used to perform denoising, word segmentation, part-of-speech tagging, and prosodic information analysis on the text data to obtain structured text training data; the voice data processing is used to perform noise processing and format conversion on the voice data to obtain standardized voice training data; the structured text training data, the standardized voice training data, and the time step information extracted from the standardized voice training data constitute a standardized training dataset.

[0114] During the initial training phase of a speech synthesis model, dataset quality and preprocessing are crucial. Bimodal datasets, which contain both text and corresponding speech signals, provide rich training information for the model. Text data processing is a complex, multi-layered process that converts raw text into a more structured and standardized form. For example, for Chinese speech synthesis datasets, public datasets such as THCHS-30 or MSpider can be used. Both are databases related to speech recognition. During processing, the text is first denoised to remove irrelevant punctuation and special characters. Next, word segmentation is performed, using Chinese word segmentation tools such as Jieba to segment the text into its smallest semantic units. Part-of-speech tagging is then performed to identify the grammatical attributes of each word, such as noun, verb, or adjective. Finally, prosodic analysis is performed to extract intonation, stress, and rhythmic features from the text.

[0115] In speech data processing, noise processing helps improve the signal-to-noise ratio of speech signals and eliminate background noise and interference during the recording process. Format conversion ensures that all speech data can be stored and processed in a unified format (e.g., WAV, 16-bit, 16kHz). Time step information extraction facilitates subsequent non-autoregressive speech synthesis, as it records the detailed characteristics of the speech signal in the time dimension. For example, for a piece of speech data, techniques such as spectral subtraction or wavelet transform can be used for noise suppression. The original audio is converted to a standardized WAV format with a unified sampling rate of 16kHz and a quantization bit size of 16 bits. Time step information can be obtained by extracting the duration of each phoneme or syllable, which provides precise time alignment information for subsequent speech synthesis.

[0116] S502: Divide the standardized training dataset into a training subset, a validation subset, and a test subset. Stratified random sampling can be used to ensure that the data distribution of each subset is consistent. The training subset is used for model parameter learning, the validation subset is used for model tuning and preventing overfitting, and the test subset is used for final model performance evaluation. The division ratio can be training subset: validation subset: test subset = 7:2:1. For example, for a dataset containing 10,000 speech-text pairs, it will be divided into 7,000 training data, 2,000 validation data, and 1,000 test data.

[0117] S503: Perform feature extraction on the structured text training data in the training subset using a deep learning algorithm to obtain a text feature vector.

[0118] Specifically, text feature extraction is a crucial step in natural language processing, aiming to convert discrete textual information into a continuous, high-dimensional vector representation. Deep learning algorithms, particularly models based on attention mechanisms, can effectively capture the semantic and grammatical features of text. The key to feature extraction lies in mapping raw text data into a semantically rich, moderately dimensional vector space, achieving a structured representation of textual information through sophisticated mathematical transformations.

[0119] The feature extraction process includes key steps such as text preprocessing, word embedding, feature encoding, and semantic representation learning. These steps form a systematic feature conversion process that can gradually transform raw text into vector representations with deep semantic information.

[0120] Text preprocessing is the foundation of feature extraction. This stage requires standardization of the original text, including tokenization, stop word removal, and case normalization. Tokenization is the process of breaking continuous text into its smallest semantic units, with different processing strategies for different languages. For example, for the Chinese text "Earth is the only planet in the solar system known to harbor life," tokenization is first performed to obtain ["Earth", "is", "solar system", "in", "only", "known", "exists", "life", "of", "planet"].

[0121] Word embedding technology is a key method for mapping discrete words into a continuous vector space. Mainstream methods include Word2Vec (Word to Vector), GloVe (Global Vectors for Word Representation), and pre-trained language models such as BERT. These methods capture the semantic similarity between words, mapping semantically similar words to adjacent vector spaces. By using technologies like Word2Vec, each word after tokenization can be converted into a dense vector of fixed dimension (e.g., 300 dimensions). For example, "Earth" is represented as [0.2, -0.3, 0.5, ..., 0.1], and "Planet" is represented as [0.25, -0.28, 0.52, ..., 0.15].

[0122] Feature encoding is the process of converting word embeddings into higher-level semantic representations. Encoding methods include recurrent neural networks (RNNs), long short-term memory (LSTMs), and attention-based encoders. The multi-head self-attention mechanism can capture complex dependencies between words. In practice, a sequence of word embeddings is input. The self-attention mechanism then calculates the degree of association between each word and other words, ultimately outputting a context-sensitive feature vector.

[0123] Semantic representation learning is used to generate fixed-length vectors that capture overall semantic information. Available methods include average pooling, max pooling, and attention pooling. For a sentence like "Earth is the only planet in the solar system known to harbor life," attention pooling can generate a comprehensive semantic vector. The input is the word embedding and the encoded feature vector, and the output is a fixed-length vector that represents the semantics of the entire sentence.

[0124] To improve the expressiveness of features, feature vectors are compressed and normalized. This includes using principal component analysis (PCA) for dimensionality reduction, applying batch normalization to adjust vector distribution, and using L2 regularization to prevent overfitting. These techniques further optimize feature vectors, making them more compact and meaningful.

[0125] Ultimately, the original text "Earth is the only planet in the solar system known to harbor life" is converted into a semantically rich, dimensionally controlled feature vector. This vector not only retains the core semantic information of the original text but also provides high-quality input for the subsequent speech synthesis module.

[0126] S504: Construct a transformer-based non-autoregressive speech synthesis model. The transformer is a Transformer model, which is a deep learning architecture based on a self-attention mechanism. The speech synthesis model includes the following components:

[0127] An input layer, configured to receive the text feature vector, perform normalization processing on the text feature vector, and output the normalized text feature vector;

[0128] The main purpose of the input layer is to convert the text feature vectors obtained in the previous stage into a standardized input format. Normalization is an important technical means to ensure the consistency and stability of model input data. By normalizing the input text feature vectors, we can eliminate dimensional differences between different feature vectors, making subsequent encoding and processing more stable and efficient.

[0129] Specific methods for normalization include various techniques. Z-score normalization and min-max scaling are two possible approaches. Taking Z-score normalization as an example, for an input text feature vector X, the normalization formula is: X_normalized = (X - μ) / σ, where μ is the vector's mean and σ is the standard deviation. For example, for a feature vector containing the text "Earth is the only planet in the solar system known to harbor life," Z-score normalization can map the original high-dimensional vector to a standard normal distribution, ensuring numerical stability for subsequent processing.

[0130] The encoder utilizes a multi-head self-attention module within the encoder to encode the normalized text feature vector to generate a high-dimensional semantic feature vector. The encoder's primary function is to perform deep semantic encoding on the input normalized text feature vector using the multi-head self-attention mechanism. The multi-head self-attention module allows the model to simultaneously focus on information at different positions in the input sequence, thereby capturing complex contextual dependencies. This encoding method effectively transforms discrete text features into continuous, rich semantic representations.

[0131] The length adjuster is used to predict the time step information of the speech signal based on the high-dimensional semantic feature vector and the time step information in the training subset, wherein the training subset includes the structured text training data, the standardized speech training data and the time step information extracted from the standardized speech training data; and based on the predicted time step information, the high-dimensional semantic feature vector is time-aligned to generate a time-aligned high-dimensional semantic feature vector. The task of the length adjuster is to solve the length matching problem in text-to-speech conversion. Traditional autoregressive models usually need to generate speech step by step, while non-autoregressive models require the ability to predict the entire time step of speech at one time. The length adjuster achieves accurate time step prediction and feature vector alignment by learning the mapping relationship between text feature vectors and speech time steps.

[0132] A non-autoregressive decoder is used to generate a speech signal time series by processing the time-aligned high-dimensional semantic feature vectors in parallel using a parallel decoding method. This decoder generates the entire speech signal time series in parallel, rather than in a traditional step-by-step manner. This approach improves the computational efficiency of speech synthesis and is particularly suitable for applications with high real-time requirements. By processing the time-aligned high-dimensional semantic feature vectors in parallel, the decoder converts abstract semantic representations into concrete acoustic features.

[0133] The output layer is used to post-process the speech signal time series to generate a final playable speech signal. The output layer is the final processing stage of the speech synthesis model. Its primary task is to convert the raw speech signal time series generated by the decoder into high-quality, natural, playable speech. Post-processing techniques, such as waveform smoothing, spectral enhancement, and noise suppression, enhance the naturalness and clarity of speech.

[0134] S505: Train the speech synthesis model using the training subset, detect the gradient values ​​generated when the parameters are updated during the training process, and clip the gradient values ​​that exceed the predetermined range to within the predetermined range; and dynamically adjust the learning rate according to the loss of the validation subset in each round of training.

[0135] During neural network training, due to the complex network structure and nonlinear activation functions, gradient values ​​can become abnormally large. To address this issue, a gradient clipping mechanism was designed. This mechanism sets a predetermined gradient threshold. When the L2 norm of the gradient exceeds this threshold, the gradient is scaled proportionally. For example, if the predetermined threshold is 1.0 and the current gradient norm is 5.0, the gradient is scaled proportionally to 1 / 5 of its original value, or 1.0. This method effectively prevents gradient explosion, maintains numerical stability during model training, and ensures smooth convergence of neural networks for complex speech synthesis tasks.

[0136] Dynamic adjustment of the learning rate is a key strategy to improve model training results. Traditional fixed learning rates are difficult to adapt to the convergence characteristics of the model at different training stages. Therefore, this embodiment provides an adaptive learning rate adjustment mechanism based on the validation subset loss. When the validation subset loss does not decrease significantly over several consecutive training cycles, the model will automatically reduce the learning rate to help the model escape the local optimal solution and find a better parameter space. This dynamic adjustment can be achieved through a variety of methods, such as exponential decay, loss reduction strategy, or cyclic learning rate. By monitoring the learning progress of the model in real time, dynamic learning rate adjustment can improve the generalization ability and final performance of the model.

[0137] In a further embodiment, step S506 is also included: performance verification of the trained speech synthesis model, which specifically includes: evaluating the speech quality, synthesis accuracy and delay time of the speech synthesis model on the verification subset and the test subset to obtain performance verification results; and optimizing the parameters of the speech synthesis model according to the performance verification results.

[0138] In a further embodiment, the method further includes step S60: performing denoising and volume adjustment on the voice signal to generate a processed voice signal, and playing or saving the processed voice signal through an audio output device. This step improves the overall quality of the voice signal, eliminates background noise interference, and achieves precise volume control through a signal processing algorithm.

[0139] Modern denoising technologies mainly include traditional denoising methods based on frequency domain analysis and intelligent denoising algorithms based on deep learning. Frequency domain denoising technology converts the time domain signal into the frequency domain through Fourier transform, identifies and suppresses the noise frequency band, and then reconstructs the clear speech signal through inverse transform. Deep learning denoising methods use convolutional neural networks (CNN) or generative adversarial networks (GAN) to learn complex noise features and denoising patterns through massive training data. For example, for speech recorded in an urban environment, there are background interferences such as traffic noise and air conditioning noise. The denoising algorithm needs to accurately identify and suppress these frequency bands while preserving the original timbre and clarity of the speech.

[0140] Volume adjustment is a critical step in ensuring optimal audio quality for voice signals in various scenarios. Volume adjustment isn't simply amplification or reduction; it involves a complex process of dynamic range compression and normalization. Professional volume adjustment algorithms consider multiple dimensions, including the dynamic range of speech, instantaneous volume fluctuations, and psychoacoustic characteristics. Adaptive algorithms dynamically adjust the volume based on the characteristics of the speech content and the target application scenario. For example, navigation voice requires a relatively constant volume to ensure clarity and audibility; emotional voice assistants, on the other hand, require more nuanced volume modulation based on context and emotional expression.

[0141] Audio output is the final step in voice signal processing, involving multiple output methods and device adaptation. Output methods primarily fall into two categories: real-time playback and file storage. Real-time playback requires low latency, high-quality, instant output, and is commonly found in voice interaction systems. File storage, on the other hand, requires support for multiple audio formats to meet the needs of diverse application scenarios. For example, smartphone applications require simultaneous support for multiple output methods, including speakers, headphones, and Bluetooth audio devices, and the ability to intelligently adjust audio parameters based on device characteristics.

[0142] Choosing an output format is a crucial step in audio processing. Common audio formats include lossless formats (such as WAV) and lossy compressed formats (such as MP3 and AAC). Lossless formats preserve complete audio details and are suitable for professional audio production; lossy formats strike a balance between compression ratio and sound quality, making them more suitable for everyday applications and network transmission. Choosing an output format requires a balance between audio quality, file size, compatibility, and other factors.

[0143] Through the above-mentioned comprehensive speech signal processing process, the overall quality of speech synthesis and user experience can be improved, and multi-dimensional optimization from the technical to the perception level can be achieved. Example 2

[0144] like Figure 7 As shown, an embodiment of the present invention provides a low-latency speech synthesis device, which includes:

[0145] A text normalization processing module is used to normalize the input text to obtain normalized text;

[0146] A language structured processing module is used to perform part-of-speech tagging, word segmentation and prosodic information analysis on the standardized text to obtain a language structured text;

[0147] A feature extraction module, which uses a deep learning algorithm to extract features from the language structured text to obtain a text feature vector;

[0148] The speech synthesis module converts the text feature vector into a speech signal using a transformer-based non-autoregressive speech synthesis model.

[0149] In some embodiments, the text normalization processing module specifically includes:

[0150] The text cleaning submodule is used to clean the input text, remove invalid characters, redundant information and non-printing characters, and obtain clean text;

[0151] The text standardization submodule is used to perform format unification and standardization processing on the clean text to obtain a normalized text.

[0152] In some embodiments, the language structure processing module specifically includes:

[0153] A word segmentation submodule is used to perform word segmentation processing on the normalized text and obtain a word segmentation result using a word segmentation algorithm;

[0154] A part-of-speech tagging submodule is used to perform part-of-speech tagging on the word segmentation results, using a pre-trained part-of-speech tagging model to obtain a text sequence with part-of-speech tags;

[0155] A grammatical feature extraction submodule, configured to extract grammatical structure features based on the text sequence with part-of-speech tags;

[0156] A prosodic information generation submodule employs a multi-task transformer deep learning model, takes the grammatical structure features as input, extracts high-dimensional feature representations containing grammatical information and contextual information through a shared encoding layer, passes the high-dimensional feature representations as input to multiple output layers for different prosodic prediction tasks, generates intonation feature sequences, speech rate variation sequences, pause position sequences, and stress sequences, and integrates the intonation feature sequences, speech rate variation sequences, pause position sequences, and stress sequences into prosodic information;

[0157] The text combination submodule is used to combine the word segmentation result, the text sequence with part-of-speech tags and the prosody information to generate a language structured text.

[0158] In some embodiments, the feature extraction module specifically includes:

[0159] A text encoding submodule, configured to encode the language structured text using a BERT representation model based on a bidirectional Transformer encoder to generate a text context semantic representation;

[0160] A semantic feature extraction submodule is used to extract multi-dimensional semantic features based on the text context semantic representation, wherein the multi-dimensional semantic features include: grammatical semantic association features and part-of-speech tagging information;

[0161] a prosodic feature extraction submodule, configured to extract prosodic features using a long short-term memory model based on the grammatical semantic association features and the part-of-speech tagging information, wherein the prosodic features include intonation features, speech rate features, pause features, and stress features;

[0162] The feature vector generation submodule is used to integrate the multi-dimensional semantic features and the prosody-related features into a text feature vector.

[0163] In some embodiments, the speech synthesis module specifically includes:

[0164] An input normalization submodule receives the text feature vector through the input layer of the speech synthesis model, normalizes the text feature vector, and outputs the normalized text feature vector;

[0165] An encoder submodule inputs the normalized text feature vector into the encoder of the speech synthesis model, and uses the multi-head self-attention module within the encoder to encode the normalized text feature vector to generate a high-dimensional semantic feature vector;

[0166] a length adjustment submodule, which inputs the high-dimensional semantic feature vector into the length adjuster of the speech synthesis model, predicts the time step information of the speech signal, and performs time alignment processing on the high-dimensional semantic feature vector based on the time step information of the speech signal to generate a time-aligned high-dimensional semantic feature vector;

[0167] a non-autoregressive decoding submodule, inputting the time-aligned high-dimensional semantic feature vector into the non-autoregressive decoder of the speech synthesis model, and generating a time series of speech signals using a parallel decoding method of the non-autoregressive decoder;

[0168] The output post-processing submodule performs post-processing on the time series of the speech signal through the output layer of the speech synthesis model, wherein the post-processing includes sound quality enhancement and format conversion of the speech signal to generate a final playable speech signal.

[0169] In some embodiments, the apparatus further comprises:

[0170] The model training module is used to train the speech synthesis model to obtain a trained speech synthesis model;

[0171] The model training module is specifically used to perform the following steps:

[0172] Acquire a bimodal data set comprising text data and corresponding voice data, and perform text data processing and voice data processing on the bimodal data set;

[0173] Dividing the processed data set into a training subset, a validation subset, and a test subset;

[0174] Use deep learning algorithms to extract features from the structured text training data in the training subset to obtain text feature vectors;

[0175] Construct a transformer-based non-autoregressive speech synthesis model;

[0176] Training the speech synthesis model using the training subset, and performing gradient clipping and dynamic adjustment of the learning rate during the training process;

[0177] Verify the performance of the trained speech synthesis model and optimize the model parameters based on the verification results.

[0178] In some embodiments, the apparatus further comprises:

[0179] The speech post-processing module is used to perform noise removal and volume adjustment on the generated speech signal, generate a processed speech signal, and play or save it through an audio output device.

[0180] The beneficial technical effects of the embodiments of the present invention are as follows:

[0181] The embodiments of this invention optimize traditional speech synthesis algorithms by proposing a new low-latency speech synthesis method. This optimization not only reduces speech synthesis latency but also improves speech synthesis quality and efficiency. This optimized algorithm performance makes speech synthesis technology more stable and reliable in practical applications.

[0182] The real-time speech synthesis model, designed with extensive training data and advanced algorithms (such as LSTM, BERT, and Transformer), possesses strong generalization capabilities. Even with diverse voice inputs and complex speech environments, it can accurately generate high-quality speech output. This enhanced generalization capability makes speech synthesis technology more flexible and adaptable in practical applications.

[0183] The embodiments of this invention fully consider the efficiency of speech processing. By adopting an efficient architecture and optimized algorithms, they can quickly process large amounts of speech data and generate high-quality speech output. This improved processing efficiency makes speech synthesis technology more efficient and real-time in practical applications.

[0184] By implementing low-latency speech synthesis, the embodiments of this invention enhance the user experience when using voice interaction functions. Whether in smart homes, in-vehicle systems, or online education, users can enjoy a smoother and more natural voice interaction process. This interactive method with instant feedback allows users to experience more efficient and convenient service, thereby enhancing user satisfaction and loyalty to the product.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Example 3

[0186] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the low-delay speech synthesis methods described above.

[0187] If the integrated modules / units are implemented as software functional units and sold or used as standalone products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. Of course, there are other types of readable storage media, such as quantum memory and graphene memory. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals. Example 4

[0188] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a low-latency speech synthesis method provided by the present invention.

[0189] To reduce speech synthesis latency, the model structure was streamlined and optimized. A lightweight neural network architecture was adopted to reduce the model's computational workload and storage requirements. Furthermore, technologies such as parallel computing and asynchronous updates were introduced to further improve the model's speed and real-time performance. These optimizations enable the speech synthesis model to achieve low-latency speech synthesis while maintaining high performance.

[0190] By optimizing algorithms, streamlining model structures, and leveraging efficient computing resources, the present invention achieves near-instant speech synthesis, enabling users to hear clear, fluent speech output while simultaneously inputting text. This low-latency feature is crucial for voice interaction scenarios requiring rapid response, enhancing user experience and interactivity.

[0191] The optimization algorithm includes gradient clipping and dynamic learning rate adjustment. In step S505, gradient values ​​are detected and clipped in real time during training to ensure they do not exceed a predetermined range, preventing exploding gradients. Furthermore, the learning rate is dynamically adjusted based on the validation set loss during each round of training, enabling more accurate model convergence.

[0192] This embodiment optimizes the feature extraction algorithm in step S303, using a long short-term memory (LSTM) network to extract prosody-related features. This algorithm can capture subtle features in speech signals, such as intonation, speaking rate, pauses, and stress, enhancing the naturalness and expressiveness of speech synthesis through multi-dimensional feature extraction.

[0193] This embodiment optimizes the multi-task transformer deep learning model in step S204. By sharing the encoding layer and setting multiple parallel output layers, multiple prosody prediction tasks such as intonation, speaking rate, pauses, and stress can be processed simultaneously, improving the efficiency and accuracy of feature extraction.

[0194] The transformer's non-autoregressive speech synthesis model structure further optimizes the model architecture. An end-to-end lightweight speech synthesis model is constructed through the input layer, encoder, length adjuster, non-autoregressive decoder, and output layer. Each module performs its own function, achieving modularity and streamlining of the model structure, ensuring speech synthesis quality while reducing computational overhead.

[0195] The embodiments of the present invention achieve true real-time speech synthesis by processing input text in real time and generating continuous speech output. Users can hear the synthesized speech results while speaking, without having to wait for the entire sentence to be input. This real-time nature not only improves the naturalness and fluency of voice interaction, but also provides users with a more intuitive and convenient voice communication experience.

[0196] By simulating subtle features of real-life pronunciation, such as intonation, pauses, stress, and speaking speed, the embodiments of the present invention can generate speech that closely resembles a real human voice. This not only improves the fidelity of speech synthesis, but also enhances its acceptability and user satisfaction. In practical applications, this highly natural speech output can provide users with a more realistic and vivid voice communication experience, enhancing the immersiveness and interactivity of voice interaction.

[0197] The embodiments of this invention improve the efficiency of computing resource utilization by optimizing the algorithm and model structure. On the one hand, by streamlining the model structure and reducing unnecessary computation, energy consumption and costs during speech synthesis are reduced. On the other hand, by utilizing efficient computing resources and parallel processing technology, the processing speed and real-time performance of speech synthesis are improved. This efficient use of computing resources not only improves speech synthesis performance but also opens up the possibility of its application in more scenarios.

[0198] Reference below Figure 8 , which shows a schematic structural diagram of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0199] like Figure 8 As shown, computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of computer system 800 are also stored in RAM 803. CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0200] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. Removable media 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read from the removable media can be installed in the storage section 808 as needed.

[0201] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 809 and / or installed from removable media 811. When the computer program is executed by the central processing unit 801, the above-described functions defined in the system of the present invention are performed.

[0202] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0204] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor, and the names of these units do not, in some cases, limit the units themselves.

[0205] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A low-latency speech synthesis method, characterized in that: include: S10: normalize the input text to obtain a normalized text; S20: performing word segmentation processing, part-of-speech tagging processing, and prosodic information analysis on the standardized text to obtain a language-structured text; S30: extracting features from the language structured text using a deep learning algorithm to obtain a text feature vector; S40: using a transformer-based non-autoregressive speech synthesis model to convert the text feature vector into a speech signal; Step S40 specifically includes: S401: receiving the text feature vector through an input layer of a speech synthesis model, normalizing the text feature vector, and outputting the normalized text feature vector; S402: Inputting the normalized text feature vector into the encoder of the speech synthesis model, and encoding the normalized text feature vector using the multi-head self-attention module within the encoder to generate a high-dimensional semantic feature vector; S403: Inputting the high-dimensional semantic feature vector into the length adjuster of the speech synthesis model, predicting the time step information of the speech signal, and performing time alignment processing on the high-dimensional semantic feature vector based on the time step information of the speech signal to generate a time-aligned high-dimensional semantic feature vector; S404: Inputting the time-aligned high-dimensional semantic feature vector into the non-autoregressive decoder of the speech synthesis model, and generating a time series of speech signals using a parallel decoding method of the non-autoregressive decoder; S405: Post-processing the time series of the speech signal through the output layer of the speech synthesis model, wherein the post-processing includes sound quality enhancement and format conversion of the speech signal to generate a final playable speech signal.

2. The method according to claim 1, characterized in that Step S10 specifically includes: S101: Clean the input text to remove invalid characters, redundant information, and non-printing characters to obtain a clean text; S102: performing format unification and standardization processing on the clean text to obtain a normalized text.

3. The method according to claim 1, characterized in that Step S20 specifically includes: S201: performing word segmentation processing on the normalized text, and obtaining a word segmentation result using a word segmentation algorithm; S202: performing part-of-speech tagging on the word segmentation result, using a pre-trained part-of-speech tagging model to obtain a text sequence with part-of-speech tags; S203: Extracting grammatical structure features based on the text sequence with part-of-speech tags; S204: using a multi-task transformer deep learning model, taking the grammatical structure features as input, extracting high-dimensional feature representations containing grammatical information and contextual information through a shared encoding layer, passing the high-dimensional feature representations as input to multiple output layers for different prosody prediction tasks, generating intonation feature sequences, speech rate variation sequences, pause position sequences, and stress sequences, respectively, and integrating the intonation feature sequences, speech rate variation sequences, pause position sequences, and stress sequences into prosody information; S205: Combining the word segmentation result, the text sequence with part-of-speech tags, and the prosody information to generate a language-structured text.

4. The method according to claim 1, wherein Step S30 specifically includes: S301: Encode the language structured text using a representation model BERT based on a bidirectional Transformer encoder to generate a text context semantic representation; S302: Extracting multi-dimensional semantic features based on the text context semantic representation, wherein the multi-dimensional semantic features include: grammatical semantic association features and part-of-speech tagging information; S303: extracting prosodic features using a long short-term memory model based on the grammatical and semantic association features and the part-of-speech tagging information, wherein the prosodic features include intonation features, speech rate features, pause features, and stress features; S304: Integrate the multi-dimensional semantic features and the prosody-related features into a text feature vector.

5. The method according to claim 1, wherein Before step S10, the method further includes: training the speech synthesis model to obtain a trained speech synthesis model; The training process of the speech synthesis model includes the following steps: S501: Acquire a bimodal dataset comprising text data and corresponding speech data, and perform text data processing and speech data processing on the bimodal dataset; the text data processing is used to perform denoising, word segmentation, part-of-speech tagging, and prosodic information analysis on the text data to obtain structured text training data; the speech data processing is used to perform noise removal and format conversion on the speech data to obtain standardized speech training data; the structured text training data, the standardized speech training data, and time step information extracted from the standardized speech training data constitute a standardized training dataset; S502: Divide the standardized training data set into a training subset, a validation subset, and a test subset; S503: Perform feature extraction on the structured text training data in the training subset using a deep learning algorithm to obtain a text feature vector; S504: Construct a transformer-based non-autoregressive speech synthesis model, wherein the speech synthesis model includes the following components: An input layer, configured to receive the text feature vector, perform normalization processing on the text feature vector, and output the normalized text feature vector; An encoder, configured to encode the normalized text feature vector using a multi-head self-attention module within the encoder to generate a high-dimensional semantic feature vector; a length adjuster, configured to predict time step information of a speech signal based on the high-dimensional semantic feature vector and time step information in the training subset, wherein the training subset includes the structured text training data, the standardized speech training data, and the time step information extracted from the standardized speech training data; and perform time alignment processing on the high-dimensional semantic feature vector based on the predicted time step information to generate a time-aligned high-dimensional semantic feature vector; a non-autoregressive decoder for processing the time-aligned high-dimensional semantic feature vectors in parallel using a parallel decoding method to generate a time series of speech signals; The output layer is used to post-process the time series of the speech signal to generate a final playable speech signal; S505: Training the speech synthesis model using the training subset, detecting gradient values ​​generated when updating parameters during the training process, clipping gradient values ​​that exceed a predetermined range to within the predetermined range; and dynamically adjusting the learning rate based on the loss of the validation subset in each round of training; S506: Perform performance verification on the trained speech synthesis model, which specifically includes: evaluating the speech quality, synthesis accuracy and delay time of the speech synthesis model on the verification subset and the test subset to obtain performance verification results; and optimizing the parameters of the speech synthesis model according to the performance verification results.

6. The method according to claim 1, wherein The method further comprises step S60: performing denoising and volume adjustment on the voice signal to generate a processed voice signal, and playing or saving the processed voice signal through an audio output device.

7. A low-latency speech synthesis device, characterized in that: include: A text normalization processing module is used to normalize the input text to obtain normalized text; A language structured processing module is used to perform part-of-speech tagging, word segmentation and prosodic information analysis on the standardized text to obtain a language structured text; A feature extraction module, which uses a deep learning algorithm to extract features from the language structured text to obtain a text feature vector; A speech synthesis module, using a transformer-based non-autoregressive speech synthesis model to convert the text feature vector into a speech signal; The speech synthesis module specifically includes: An input normalization submodule receives the text feature vector through the input layer of the speech synthesis model, normalizes the text feature vector, and outputs the normalized text feature vector; An encoder submodule inputs the normalized text feature vector into the encoder of the speech synthesis model, and uses the multi-head self-attention module within the encoder to encode the normalized text feature vector to generate a high-dimensional semantic feature vector; a length adjustment submodule, which inputs the high-dimensional semantic feature vector into the length adjuster of the speech synthesis model, predicts the time step information of the speech signal, and performs time alignment processing on the high-dimensional semantic feature vector based on the time step information of the speech signal to generate a time-aligned high-dimensional semantic feature vector; a non-autoregressive decoding submodule, inputting the time-aligned high-dimensional semantic feature vector into the non-autoregressive decoder of the speech synthesis model, and generating a time series of speech signals using a parallel decoding method of the non-autoregressive decoder; The output post-processing submodule performs post-processing on the time series of the speech signal through the output layer of the speech synthesis model, wherein the post-processing includes sound quality enhancement and format conversion of the speech signal to generate a final playable speech signal.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement a low-delay speech synthesis method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a low-delay speech synthesis method as described in any one of claims 1 to 6 is implemented.

Images