Text-guided speech synthesis methods, devices, computer equipment, and storage media

Abstract

This application belongs to the field of artificial intelligence technology and relates to a text-guided speech synthesis method. The method includes: annotating a speech dataset with style tags and injecting scene noise to obtain a reference speech set; inputting the reference speech set and the text dataset into an acoustic model; encoding the style tags using a style encoder to obtain style encoding features; encoding the reference speech using a reference encoder to obtain reference speech encoding features; encoding the text using a text encoder to obtain text encoding features; inputting all encoded features into an acoustic structure to obtain speech acoustic features; and inputting the speech acoustic features into a vocoder to synthesize a waveform to obtain predicted synthesized speech for training, thereby obtaining a speech synthesis model. This application also provides a text-guided speech synthesis device, computer equipment, and storage medium. Furthermore, this application relates to blockchain technology, allowing the text to be converted to be stored in the blockchain. This application improves the efficiency and quality of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text-guided speech synthesis method, apparatus, computer device, and storage medium. Background Technology

[0002] With continuous breakthroughs in artificial intelligence technology, large-scale speech synthesis models have experienced unprecedented development in recent years. As an important human-computer interaction tool, speech synthesis technology has permeated multiple fields such as intelligent customer service, voice assistants, and education and training, greatly improving user experience and efficiency. Particularly in scenarios such as financial and medical services, speech synthesis technology is used in intelligent voice customer service for answering financial questions and promoting medical knowledge. Its natural and fluent voice and personalized expression capabilities bring customers more humanized financial and medical services. Especially in the financial sector, speech synthesis technology is also widely used in telephone banking, intelligent investment advisory, and other scenarios, enhancing customer trust and satisfaction through the synthesis of realistic voices.

[0003] In recent years, with increasing emphasis on privacy protection and growing demand for diverse and flexible speech synthesis audio, an innovative "text-guided speech synthesis method" has emerged. This method introduces two key input parameters: content prompt (the text to be synthesized) and style prompt (a text describing the desired speech style). This enables precise control and personalized customization of speech synthesis. The content prompt is the specific text content the user wishes to synthesize, while the style prompt describes the desired speech style, such as speech rate, pitch, and emotion.

[0004] However, despite significant progress in text-guided speech synthesis, existing models still suffer from several shortcomings. First, the limitation of single-language support is a major constraint. Current large-scale speech synthesis models typically only handle content and style cues in a single language, failing to integrate multilingual content and perform style transfer, thus limiting their widespread application in a globalized context. Second, the limited style description capabilities are another major weakness. When using style cues to describe speech style, models are primarily limited to aspects such as speech rate, pitch, emotion, signal-to-noise ratio, gender, and accent, lacking the ability to simulate speech styles under different noise levels. Therefore, it is difficult to synthesize realistic audio in complex and diverse real-world scenarios (such as making phone calls on the street or eating in a restaurant). Finally, the limitations of acoustic models also restrict the performance of large-scale speech synthesis models. Currently, the acoustic models in these models are mainly based on diffusion models, which have certain shortcomings in synthesis speed, audio quality, and adaptability to specific scenarios, making it difficult to meet users' demands for efficient and high-quality speech synthesis. Summary of the Invention

[0005] The purpose of this application is to provide a text-guided speech synthesis method, apparatus, computer device, and storage medium to solve the technical problems of existing speech synthesis methods, such as limited language options, limited style descriptions, and poor efficiency and quality.

[0006] To address the aforementioned technical problems, this application provides a text-guided speech synthesis method, employing the following technical solution:

[0007] Acquire speech datasets and text datasets in different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data;

[0008] Style labeling and scene noise injection are performed on the speech dataset to obtain a reference speech set;

[0009] The reference speech set and the text dataset are input into a pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder;

[0010] The style encoder encodes the style labels of the text dataset and the reference speech set to obtain style-encoded features;

[0011] The reference speech in the reference speech set is encoded by the reference encoder to obtain the reference speech coding features;

[0012] The text dataset is encoded using the text encoder to obtain text encoding features;

[0013] The style coding features, the reference speech coding features, and the text coding features are input into the acoustic structure to obtain speech acoustic features;

[0014] The vocoder is used to synthesize waveforms from the acoustic features of the speech to obtain predicted synthesized speech.

[0015] According to the preset loss function, the loss is calculated based on the reference speech and the predicted synthesized speech. The model parameters are adjusted based on the loss, and the training continues iteratively until the iteration stopping condition is met, so as to obtain the final speech synthesis model.

[0016] The text to be converted is obtained and input into the speech synthesis model to obtain the target synthesized speech.

[0017] Furthermore, the step of performing style labeling and scene noise injection on the speech dataset to obtain a reference speech set includes:

[0018] The style label of each voice data in the voice dataset is determined according to a preset style dimension, and the voice data is labeled based on the style label to obtain a labeled voice set;

[0019] A scene noise dataset is obtained, and a data fusion algorithm is used to fuse the scene noise dataset with the labeled speech set to obtain a reference speech set containing noise.

[0020] Furthermore, the step of determining the style label for each piece of speech data in the speech dataset according to a preset style dimension includes:

[0021] The speech data is analyzed using a speech signal processing algorithm to obtain the corresponding speech rate, pitch, and signal-to-noise ratio;

[0022] Determine the speech rate label and pitch label of the speech data based on the speech rate and the pitch.

[0023] The signal-to-noise ratio is compared with a preset signal-to-noise ratio threshold to obtain a signal-to-noise ratio label;

[0024] The speech dataset is input into the trained emotion recognition model for emotion classification to obtain the corresponding emotion labels;

[0025] Using a trained gender recognition model, the gender of the speaker in the speech dataset is determined, and the corresponding gender label is obtained.

[0026] The speech dataset is input into the trained accent recognition model to determine the accent type of the speech data, and an accent label is obtained based on the accent type.

[0027] The speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label are combined to obtain the style label.

[0028] Furthermore, the style encoder includes a BERT embedding layer, a spatial expansion layer, and a style encoding layer. The step of encoding style labels of the text dataset and the reference speech set using the style encoder to obtain style encoded features includes:

[0029] The style labels of the text dataset and the reference speech set are vectorized using the BERT embedding layer to obtain style embedding vectors.

[0030] The style embedding vector is input into the spatial extension layer, and the style extension vector is obtained by concatenating the style embedding vector and the introduced style cue vector.

[0031] Style features are extracted from the style extension vector using the self-attention mechanism of the style encoding layer to obtain style encoding features.

[0032] Furthermore, the text encoder includes a text embedding layer, a Transformer encoding layer, and a pooling layer. The step of encoding the text dataset using the text encoder to obtain text encoding features includes:

[0033] The text embedding layer performs vector embedding on the text data in the text dataset to obtain text embedding vectors;

[0034] The text embedding vector is input into the Transformer encoding layer, and semantic features are extracted through a multi-layer self-attention mechanism to obtain semantic encoding features;

[0035] The semantic encoded features are pooled through the pooling layer to obtain text encoded features.

[0036] Furthermore, the acoustic structure includes a reversible transform layer, a stream decoder, and a residual layer. The step of inputting the style coding features, the reference speech coding features, and the text coding features into the acoustic structure to obtain speech acoustic features includes:

[0037] The style coding features, the reference speech coding features, and the text coding features are transformed by the reversible transformation layer to obtain transformed coding features;

[0038] The transform-coded features are decoded by the stream decoder to obtain an acoustic feature sequence;

[0039] The acoustic feature sequence is input into the residual layer for residual connection to obtain speech acoustic features.

[0040] Furthermore, the step of calculating the loss based on the reference speech and the predicted synthesized speech according to a preset loss function includes:

[0041] Calculate the adversarial loss and L1 loss between the reference speech and the predicted synthesized speech;

[0042] Calculate the similarity between the style coding features and the reference speech coding features to obtain the feature correlation loss;

[0043] The adversarial loss, the L1 loss, and the feature-related loss are weighted and summed to obtain the final loss.

[0044] To address the aforementioned technical problems, this application also provides a text-guided speech synthesis device, employing the following technical solution:

[0045] The acquisition module is used to acquire speech datasets and text datasets in different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data;

[0046] The annotation injection module is used to annotate the speech dataset with style labels and inject scene noise to obtain a reference speech set;

[0047] An input module is used to input the reference speech set and the text dataset into a pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder;

[0048] The style encoding module is used to encode the style labels of the text dataset and the reference speech set through the style encoder to obtain style encoding features;

[0049] The speech encoding module is used to encode the reference speech of the reference speech set through the reference encoder to obtain the reference speech encoding features;

[0050] A text encoding module is used to encode the text dataset using the text encoder to obtain text encoding features;

[0051] An acoustic feature extraction module is used to input the style coding features, the reference speech coding features, and the text coding features into the acoustic structure to obtain speech acoustic features;

[0052] The speech prediction module is used to synthesize the acoustic features of the speech using the vocoder to obtain predicted synthesized speech.

[0053] The iterative module is used to calculate the loss based on the reference speech and the predicted synthesized speech according to a preset loss function, adjust the model parameters based on the loss, and continue iterative training until the iteration stopping condition is met to obtain the final speech synthesis model.

[0054] The speech synthesis module is used to acquire the text to be converted, input it into the speech synthesis model, and obtain the target synthesized speech.

[0055] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0056] The computer device includes a memory and a processor, the memory storing computer-readable instructions, and the processor executing the computer-readable instructions to implement the steps of the text-guided speech synthesis method as described above.

[0057] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0058] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the text-guided speech synthesis method as described above.

[0059] Compared with the prior art, this application has the following main advantages:

[0060] This application provides a text-guided speech synthesis method. By acquiring speech and text datasets in different languages ​​and performing style labeling and scene noise injection on the speech datasets, multilingual text-guided speech synthesis can be achieved. This allows the model to learn to accurately recognize and process speech signals in noisy environments, thereby improving the model's generalization ability and robustness. At the same time, it simulates real-world scenarios, improving the multi-scenario applicability of speech synthesis. By inputting reference speech and text datasets into a pre-built acoustic model, the style encoder, reference encoder, text encoder, acoustic structure, and vocoder of the acoustic model are trained to obtain the final speech synthesis model. This method can improve the naturalness and realism of speech synthesis, enhance the model's adaptability and generalization ability, and improve the efficiency and real-time performance of speech synthesis. Attached Figure Description

[0061] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1This is an exemplary system architecture diagram to which this application can be applied;

[0063] Figure 2 This is a flowchart of an embodiment of the text-guided speech synthesis method according to this application;

[0064] Figure 3 This is a schematic diagram of the structure of one embodiment of the text-guided speech synthesis device according to this application;

[0065] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0067] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0068] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0069] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0070] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0071] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers.

[0072] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0073] It should be noted that the text-guided speech synthesis method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the text-guided speech synthesis device is generally located in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0075] Continue to refer to Figure 2 The flowchart illustrates an embodiment of a text-guided speech synthesis method according to this application, including the following steps:

[0076] Step S201: Obtain speech datasets and text datasets in different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data.

[0077] The speech datasets for different languages ​​contain a large amount of speech data in different languages, including but not limited to Chinese, English, Korean, and Japanese. Among them, Chinese also includes speech data from various dialects.

[0078] Speech datasets can be obtained from publicly available open-source speech datasets, such as the Chinese speech open-source dataset Aishell3 and the English speech open-source dataset LibriTTS. The Aishell3 dataset contains a large number of rich and diverse Chinese speech samples, including Chinese speech data with multi-dimensional features such as different genders, ages, accents, and emotional expressions.

[0079] The obtained raw speech dataset is preprocessed, including noise removal, silence removal, amplitude normalization, and speech quality adjustment, to obtain a preprocessed speech dataset. The corresponding text dataset is obtained through a speech recognition algorithm, and the speech dataset is annotated with text. The text data is matched with the corresponding acoustic features to establish a mapping relationship between text and speech, providing supervised learning data for subsequent model training.

[0080] The selection of voice datasets depends on the specific scenario and needs. For example, in the financial field, voice datasets can include voice data containing financial terminology, financial market data, stock trading information, financial reports, etc., or they can be recordings of conversations between financial customer service representatives and customers. In the medical field, voice datasets can include voice data containing medical diagnoses, treatment plans, drug names, disease names, etc., or they can include voice data containing medical guidance information such as drug usage instructions, health check guidance, and disease prevention advice, or they can be recordings of conversations between doctors and patients of different ages and genders during diagnosis and treatment.

[0081] Step S202: Style labeling and scene noise injection are performed on the speech dataset to obtain the reference speech set.

[0082] Specifically, the style label of each speech data in the speech dataset is determined according to the preset style dimension, and the speech data is labeled based on the style label to obtain the labeled speech set; the scene noise dataset is obtained, and the scene noise dataset is fused with the labeled speech set using a data fusion algorithm to obtain a reference speech set containing noise.

[0083] The preset style dimensions include speech rate, pitch, emotion, signal-to-noise ratio, gender, and accent. The preset speech processing algorithm analyzes the style label of each speech data in the speech dataset, and then the annotation tool is used to annotate the speech data with the corresponding style labels. That is, the style labels include speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label.

[0084] Based on the characteristics of noise environments in real-world application scenarios, data with similar noise features are selected from open-source datasets such as AudioSet to construct a scene noise dataset. A data fusion algorithm is then used to merge the labeled speech set with the selected scene noise dataset to generate a reference speech set containing noise. The reference speech set consists of speech data with a realistic scene feel.

[0085] Among them, data fusion algorithms can employ weighted average algorithms, signal superposition methods, and Gaussian mixture models (GMM), etc.

[0086] In some alternative implementations, data augmentation techniques are used to enrich and expand the noisy reference speech set, including changing the speech rate, volume, pitch, etc., to obtain an enhanced reference speech set.

[0087] By adding style tags and injecting scene noise, the model expands the description of special noise scenes on the basis of style dimension, which greatly enriches the style diversity of speech synthesis. In addition, the model can learn the variation rules and characteristics of speech under different noise backgrounds, so that it can accurately simulate the speech style under various special noise scenes according to the needs when synthesizing speech. This provides a more realistic and practical solution for speech synthesis in fields such as virtual reality and intelligent voice assistants in complex environments, and significantly improves the applicability and expressiveness of speech synthesis models in complex real-world scenarios.

[0088] Step S203: Input the reference speech set and text dataset into the pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder.

[0089] In this embodiment, the pre-built acoustic model is a novel acoustic model based on the Rectified Flow architecture, including a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder. The style encoder focuses on encoding the Style Prompt (speech style description text). Leveraging its powerful semantic understanding and feature extraction capabilities, it can accurately encode various style information within the Style Prompt, laying a solid foundation for the accurate shaping of the synthesized speech style. The reference encoder encodes reference speech from a reference speech set, providing crucial reference information for model learning. The text encoder is primarily responsible for encoding the Context Prompt (the text to be synthesized), enabling efficient feature extraction and semantic understanding of the input text data, transforming the Context Prompt into a text embedding with rich semantic information, providing accurate and comprehensive textual guidance for subsequent speech synthesis. The acoustic structure (Acoustic Model) is built based on the Rectified Flow architecture. The vocoder uses a HifiGAN vocoder. The final waveform is synthesized based on HifiGAN technology.

[0090] In one specific implementation, the acoustic model also includes an input layer, which preprocesses the input reference speech set and text dataset to obtain input data in a format suitable for model processing.

[0091] Step S204: The style labels of the text dataset and the reference speech set are encoded by a style encoder to obtain style encoded features.

[0092] In this embodiment, the style encoder employs an improved BERT model, including a BERT embedding layer, a spatial expansion layer, and a style encoding layer. The style label corresponding to each reference speech in the text dataset and the reference speech set is input into the style encoder. The style description text in the text dataset and the style label is processed sequentially through the BERT embedding layer, spatial expansion layer, and style encoding layer to obtain style-encoded features for style cues.

[0093] Furthermore, the style labels of the text dataset and the reference speech set are vectorized through the BERT embedding layer to obtain style embedding vectors; the style embedding vectors are input into the space expansion layer, and the style expansion vectors are obtained by concatenating the style embedding vectors and the introduced style cue vectors; the style features in the style expansion vectors are extracted through the self-attention mechanism of the style encoding layer to obtain style encoding features.

[0094] The style tag text is input into the BERT embedding layer for vector transformation. The vector transformation of the embedding layer includes token embedding, segmentation embedding and position embedding of the style tag text.

[0095] Word embeddings transform each word into a fixed-dimensional vector representation to represent its main semantic information; segment embeddings represent sentence structure to distinguish different sentences or paragraphs; and positional embeddings provide information about the position of words within a sentence, adding temporal information to the attention mechanism. The style embedding vector is obtained by summing the vectors corresponding to word embeddings, segment embeddings, and positional embeddings.

[0096] The spatial expansion layer introduces a learnable style cue vector pool to spatially expand the style embedding vectors, enriching the feature representation. Specifically, the style embedding vectors are input into the spatial expansion layer, which obtains a pre-trained cue vector pool containing S style cue vectors, where S is a positive integer. Then, k pre-trained style cue vectors are obtained from the cue vector pool, where k is a positive integer less than or equal to S. The style embedding vectors are concatenated with the k style cue vectors to obtain the expanded style vector. By introducing style cue vectors, additional information or constraints are introduced into the original feature space, thereby expanding the feature space and enabling the model to capture more features related to style information.

[0097] The style encoding layer consists of multiple stacked Transformer encoders that capture semantic information and contextual relationships in the style extension vector through a multi-layer self-attention mechanism and a feedforward neural network, thereby obtaining style encoding features.

[0098] In some alternative implementations, the style encoder also includes a linear layer connected after the last Transformer encoder. The linear layer performs a linear transformation on the features output by the last Transformer encoder to obtain the final style-encoded features for output.

[0099] Style encoding through a style encoder can improve the semantic understanding and feature extraction capabilities of style text, and accurately extract style information from style tags.

[0100] Step S205: Encode the reference speech in the reference speech set using a reference encoder to obtain the reference speech coding features.

[0101] In this embodiment, the reference encoder adopts a Transformer encoder structure, which is composed of multiple encoder layers stacked together. The semantic and contextual features of the reference speech are extracted through the multi-head attention mechanism of each encoder layer to obtain the reference speech coding features.

[0102] During model training, reference speech is enabled by default, and the encoding results of the reference speech by the reference encoder provide crucial reference information for model learning. However, during the inference phase, the reference speech is discarded to optimize system resource utilization and improve efficiency.

[0103] In some alternative implementations, the reference encoder also includes a linear layer connected after the last encoder layer. The linear layer performs a linear transformation on the features output by the last encoder layer to obtain the final reference speech coding features for output.

[0104] Step S206: Encode the text dataset using a text encoder to obtain text encoding features.

[0105] In this embodiment, the text encoder includes a text embedding layer, a Transformer encoding layer, and a pooling layer. The text dataset is input into the text encoder and processed through the text embedding layer, Transformer encoding layer, and pooling layer.

[0106] Furthermore, the text data in the text dataset is vector-embedded through a text embedding layer to obtain text embedding vectors; the text embedding vectors are input into the Transformer encoding layer, and semantic features are extracted through a multi-layer self-attention mechanism to obtain semantic encoding features; the semantic encoding features are pooled through a pooling layer to obtain text encoding features.

[0107] The text encoder employs a multi-layer Transformer encoding structure, which consists of multiple stacked encoders. The encoder model parameters are initialized, and hyperparameters such as the number of encoder layers, attention heads, and hidden layer dimensions are determined to obtain the text encoder. Specifically, the text encoder's Transformer encoding layer is composed of multiple connected Transformer encoders, each including a multi-head attention layer and a feedforward neural network layer.

[0108] The text dataset undergoes vector transformation via a text embedding layer. The resulting text embedding vectors are then fed into a Transformer encoder. Through a multi-layer self-attention mechanism and a feedforward neural network, the semantic information and contextual relationships of the text within the embedding vectors are captured, yielding encoded intermediate feature representations. In the final layer of the Transformer encoder, the semantic encoded features of the final output at each position are obtained. These semantic encoded features are then pooled using pooling layers, such as average pooling or max pooling, to obtain the text encoded features.

[0109] In some alternative implementations, the text encoder also includes a linear layer connected after the pooling layer, which performs a linear transformation on the text encoding features of the pooling operation before outputting them.

[0110] By encoding the input text using a text encoder, efficient feature extraction and semantic understanding of the input text information can be achieved, resulting in text encoding features with rich semantic information, providing accurate and comprehensive text information guidance for subsequent speech synthesis work.

[0111] Step S207: Input the style coding features, reference speech coding features, and text coding features into the acoustic structure to obtain speech acoustic features.

[0112] In this embodiment, the acoustic structure is built based on the Rectified Flow architecture, including a reversible transform layer, a stream decoder, and a residual layer. The reversible transform layer transforms the style coding features, reference speech coding features, and text coding features to obtain transform coding features. The stream decoder decodes the transform coding features to obtain an acoustic feature sequence. The acoustic feature sequence is then input into the residual layer for residual connection to obtain speech acoustic features.

[0113] Among them, the speech acoustic features include Mel spectrogram, duration information, pitch (F0) and energy. Mel spectrogram represents the visual representation of the frequency content of the speech signal; duration information represents the duration of each phoneme or syllable; pitch (F0) represents the fundamental frequency of the speech signal, which is related to human pitch perception; energy represents the intensity or loudness of the speech signal.

[0114] Specifically, style coding features, reference speech coding features, and text coding features are concatenated to obtain concatenated coding features. These concatenated coding features are then input into a reversible transform layer, which includes an affine coupling layer, a permutation layer, and a scaling layer. The concatenated coding features are processed sequentially through these layers to obtain transform coding features. The streaming decoder employs a Transformer decoding structure, consisting of multiple stacked decoders. It captures long-distance dependencies between transform coding features through multi-head attention and cross-attention mechanisms within each encoder, resulting in a predicted acoustic feature sequence. Residual concatenation of the acoustic feature sequence through residual layers enhances the model's expressive power and information transmission capabilities, avoids the vanishing gradient problem, and improves the model's training stability.

[0115] The affine coupling layer performs a linear transformation on the features using learnable parameters; the permutation layer rearranges the feature dimensions; and the scaling layer scales the features. A gating mechanism is introduced in each invertible transformation layer, using a sigmoid activation function to generate a gating signal to dynamically adjust the transformation parameters, enabling the model to adaptively control the intensity of each transformation layer.

[0116] In some optional implementations of this embodiment, the acoustic structure can be configured with multiple reversible transformation layers. A gating mechanism is introduced in each reversible transformation layer. A gating signal is generated through a sigmoid activation function to dynamically adjust the transformation parameters, enabling the model to adaptively control the intensity of each transformation layer.

[0117] By predicting speech acoustic features through acoustic structure, the efficiency and accuracy of the Rectified Flow architecture in handling complex acoustic tasks are fully utilized. This effectively overcomes the bottleneck of traditional acoustic models in terms of synthesis speed, making the speech synthesis process smoother and more efficient, and significantly improving the quality and speed of synthesized speech.

[0118] Step S208: The acoustic features of the speech are synthesized into waveforms using a vocoder to obtain the predicted synthesized speech.

[0119] In this embodiment, the vocoder is a HifiGAN vocoder. Through the GAN generator network in the vocoder, the acoustic features of the input speech are converted into a high-fidelity speech waveform, thus obtaining the predicted synthesized speech.

[0120] The synthesized speech is post-processed, including noise reduction and volume normalization, to obtain the final high-fidelity speech output.

[0121] Step S209: Calculate the loss based on the reference speech and the predicted synthesized speech according to the preset loss function, adjust the model parameters based on the loss, and continue iterative training until the iteration stopping condition is met to obtain the final speech synthesis model.

[0122] Specifically, the adversarial loss and L1 loss between the reference speech and the predicted synthesized speech are calculated; the similarity between the style-coded features and the reference speech-coded features is calculated to obtain the feature-related loss; the adversarial loss, L1 loss and feature-related loss are weighted and summed to obtain the final loss.

[0123] In this embodiment, the vocoder also includes a discriminator. During the model training phase, the discriminator evaluates the difference between the waveform of the generated predicted synthesized speech and the waveform of the real reference speech to obtain the adversarial loss.

[0124] The formula for calculating L1 loss is as follows:

[0125]

[0126] In the formula, n represents the number of reference speech samples in the reference speech set; y i y' represents the i-th reference speech in the reference speech set; i This represents the predicted synthesized speech corresponding to the i-th reference speech.

[0127] In the inference phase, in order to optimize system resource utilization and improve efficiency, the reference speech will be discarded. To achieve this smooth transition, in this embodiment, a computational strategy is introduced during the training process, namely, calculating the cosine similarity between style coding features and reference speech coding features as feature correlation loss. Thus, even without the reference speech, the synthesized speech that conforms to the expected style can still be accurately generated in the inference phase.

[0128] By using various loss methods such as adversarial loss, L1 loss, and feature correlation loss to optimize the model, we can improve the model's prediction accuracy, enhance its robustness, promote its generalization ability, and improve the naturalness and intelligibility of synthesized speech.

[0129] In this embodiment, the Adam or SGD optimizer is used to adjust the model parameters of the acoustic model according to the loss. The adjusted model continues to be trained iteratively until the iteration stopping condition is met, that is, the number of iterations reaches the preset number or the loss does not change significantly, and the final speech synthesis model is output.

[0130] Step S210: Obtain the text to be converted, input it into the speech synthesis model, and obtain the target synthesized speech.

[0131] In this embodiment, the text to be converted is input into a speech synthesis model. The text to be converted is style-encoded by an internal style encoder to obtain style semantic features. The text to be converted is input into a text encoder to extract text semantic features to obtain text semantic features. The style semantic features and text semantic features are input into an acoustic structure to predict acoustic features to obtain speech acoustic features. The speech acoustic features are then synthesized into speech waveforms by a vocoder to obtain the target synthesized speech.

[0132] It should be emphasized that, to further ensure the privacy and security of the text to be converted, the text can also be stored in a node of a blockchain.

[0133] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0134] This application achieves multilingual text-guided speech synthesis by acquiring speech and text datasets in different languages ​​and performing style labeling and scene noise injection on the speech datasets. This enables the model to learn to accurately recognize and process speech signals in noisy environments, thereby improving the model's generalization ability and robustness. At the same time, it simulates real-world scenarios, improving the applicability of speech synthesis in multiple scenarios. By inputting reference speech and text datasets into a pre-built acoustic model, the style encoder, reference encoder, text encoder, acoustic structure, and vocoder of the acoustic model are trained to obtain the final speech synthesis model. This improves the naturalness and realism of speech synthesis, enhances the model's adaptability and generalization ability, and improves the efficiency and real-time performance of speech synthesis.

[0135] In some optional implementations of this embodiment, the step of determining the style label of each piece of speech data in the speech dataset according to a preset style dimension includes:

[0136] Speech data is analyzed using speech signal processing algorithms to obtain the corresponding speech rate, pitch, and signal-to-noise ratio;

[0137] Determine the speech rate label and pitch label of the speech data based on speech rate and pitch.

[0138] The signal-to-noise ratio is compared with a preset signal-to-noise ratio threshold to obtain a signal-to-noise ratio label;

[0139] Input the speech dataset into the trained emotion recognition model to classify emotions and obtain the corresponding emotion labels;

[0140] A trained gender recognition model is used to determine the gender of the speaker in the speech dataset and obtain the corresponding gender label.

[0141] Input the speech dataset into the trained accent recognition model, determine the accent type of the speech data, and obtain the accent label based on the accent type;

[0142] The speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label are combined to obtain the style label.

[0143] In this embodiment, a speech signal processing algorithm is used to analyze the three dimensions of speech rate, signal-to-noise ratio, and pitch. By accurately analyzing and calculating the characteristic parameters of the speech signal, such as frequency, amplitude, and duration, the speech rate, signal-to-noise ratio, and pitch variation of the speech data can be accurately determined.

[0144] The speech rate of the speech data is divided into different levels such as slow, normal, and fast; the tone type of the speech data includes level tone, rising tone, and falling tone; and the signal-to-noise ratio of the speech data is divided into high, medium, and low levels.

[0145] The speech rate, pitch, and signal-to-noise ratio (SNR) of each speech data point are compared with preset range standards to determine the corresponding speech rate label, pitch label, and SNR label. For example, if the calculated SNR equals the preset SNR threshold, it is considered a medium SNR; if the calculated SNR is less than the preset SNR threshold, it is considered a low SNR; and if the calculated SNR is higher than the preset SNR threshold, it is considered a high SNR.

[0146] Using a trained emotion recognition model, emotion prediction is performed on the speech data in the speech dataset to determine the emotion category of the speech data, such as positive, negative, neutral, surprised, or worried. A trained gender recognition model is used to determine the gender of the speaker corresponding to the speech data. The gender recognition model can be trained and optimized using a support vector machine algorithm. Finally, a trained accent recognition model is used to determine the accent type of the speech data through speech recognition and accent recognition algorithms.

[0147] Finally, the speech data is summarized and labeled with the speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label to obtain the labeled speech set.

[0148] This application, by determining style labels for speech data, can greatly enrich the stylistic diversity of speech synthesis and improve the naturalness and expressiveness of speech synthesis.

[0149] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0150] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0152] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0153] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a text-guided speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0154] like Figure 3As shown, the text-guided speech synthesis device 300 described in this embodiment includes: an acquisition module 301, an annotation injection module 302, an input module 303, a style encoding module 304, a speech encoding module 305, a text encoding module 306, an acoustic feature extraction module 307, a speech prediction module 308, an iteration module 309, and a speech synthesis module 310. Wherein:

[0155] The acquisition module 301 is used to acquire speech datasets and text datasets of different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data;

[0156] The annotation injection module 302 is used to perform style label annotation and scene noise injection on the speech dataset to obtain a reference speech set;

[0157] The input module 303 is used to input the reference speech set and the text dataset into a pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder;

[0158] The style encoding module 304 is used to encode the style labels of the text dataset and the reference speech set through the style encoder to obtain style encoding features;

[0159] The speech encoding module 305 is used to encode the reference speech of the reference speech set through the reference encoder to obtain the reference speech encoding features;

[0160] The text encoding module 306 is used to encode the text dataset using the text encoder to obtain text encoding features;

[0161] The acoustic feature extraction module 307 is used to input the style coding feature, the reference speech coding feature and the text coding feature into the acoustic structure to obtain speech acoustic features;

[0162] The speech prediction module 308 is used to perform waveform synthesis on the acoustic features of the speech through the vocoder to obtain predicted synthesized speech;

[0163] The iteration module 309 is used to calculate the loss according to the reference speech and the predicted synthesized speech according to the preset loss function, adjust the model parameters based on the loss, and continue iterative training until the iteration stopping condition is met to obtain the final speech synthesis model.

[0164] The speech synthesis module 310 is used to acquire the text to be converted, input it into the speech synthesis model, and obtain the target synthesized speech.

[0165] It should be emphasized that, to further ensure the privacy and security of the text to be converted, the text can also be stored in a node of a blockchain.

[0166] Based on the aforementioned text-guided speech synthesis device 300, by acquiring speech and text datasets in different languages ​​and performing style labeling and scene noise injection on the speech datasets, multilingual text-guided speech synthesis can be achieved. This enables the model to learn to accurately recognize and process speech signals in noisy environments, thereby improving the model's generalization ability and robustness. At the same time, it simulates real-world scenarios, enhancing the multi-scenario applicability of speech synthesis. By inputting reference speech sets and text datasets into a pre-built acoustic model, the style encoder, reference encoder, text encoder, acoustic structure, and vocoder of the acoustic model are trained to obtain the final speech synthesis model. This improves the naturalness and realism of speech synthesis, enhances the model's adaptability and generalization ability, and simultaneously improves the efficiency and real-time performance of speech synthesis.

[0167] In some optional implementations, the annotation injection module 302 includes:

[0168] The annotation submodule is used to determine the style label of each speech data in the speech dataset according to a preset style dimension, and to annotate the speech data based on the style label to obtain an annotated speech set;

[0169] The injection submodule is used to acquire a scene noise dataset and use a data fusion algorithm to fuse the scene noise dataset with the labeled speech set to obtain a reference speech set containing noise.

[0170] By adding style tags and injecting scene noise, the diversity of speech synthesis styles is greatly enriched. In addition, when synthesizing speech, it can accurately simulate the speech style under various special noise scenarios according to the needs, which significantly improves the applicability and expressiveness of the speech synthesis model in complex real-world scenarios.

[0171] In some optional implementations of this embodiment, the annotation submodule is further used for:

[0172] The speech data is analyzed using a speech signal processing algorithm to obtain the corresponding speech rate, pitch, and signal-to-noise ratio;

[0173] Determine the speech rate label and pitch label of the speech data based on the speech rate and the pitch.

[0174] The signal-to-noise ratio is compared with a preset signal-to-noise ratio threshold to obtain a signal-to-noise ratio label;

[0175] The speech dataset is input into the trained emotion recognition model for emotion classification to obtain the corresponding emotion labels;

[0176] Using a trained gender recognition model, the gender of the speaker in the speech dataset is determined, and the corresponding gender label is obtained.

[0177] The speech dataset is input into the trained accent recognition model to determine the accent type of the speech data, and an accent label is obtained based on the accent type.

[0178] The speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label are combined to obtain the style label.

[0179] By defining style labels for speech data, we can greatly enrich the stylistic diversity of speech synthesis and improve its naturalness and expressiveness.

[0180] In some optional implementations, the style encoder includes a BERT embedding layer, a spatial expansion layer, and a style encoding layer, and the style encoding module 304 includes:

[0181] The embedding submodule is used to perform vector transformation on the style labels of the text dataset and the reference speech set through the BERT embedding layer to obtain style embedding vectors;

[0182] An extension submodule is used to input the style embedding vector into the spatial extension layer, and obtain the style extension vector by concatenating the style embedding vector and the introduced style cue vector;

[0183] The attention calculation submodule is used to extract style features from the style extension vector through the self-attention mechanism of the style encoding layer to obtain style encoding features.

[0184] Style encoding through a style encoder can improve the semantic understanding and feature extraction capabilities of style text, and accurately extract style information from style tags.

[0185] In some optional implementations of this embodiment, the text encoder includes a text embedding layer, a Transformer encoding layer, and a pooling layer, and the text encoding module 306 includes:

[0186] The text embedding submodule is used to perform vector embedding on the text data in the text dataset through the text embedding layer to obtain text embedding vectors;

[0187] The encoding submodule is used to input the text embedding vector into the Transformer encoding layer and extract semantic features through a multi-layer self-attention mechanism to obtain semantic encoding features;

[0188] The pooling submodule is used to perform pooling operations on the semantic encoding features through the pooling layer to obtain text encoding features.

[0189] By encoding the input text using a text encoder, efficient feature extraction and semantic understanding of the input text information can be achieved, resulting in text encoding features with rich semantic information, providing accurate and comprehensive text information guidance for subsequent speech synthesis work.

[0190] In some optional implementations, the acoustic structure includes a reversible transform layer, a stream decoder, and a residual layer, and the acoustic feature extraction module 307 is further used for:

[0191] The style coding features, the reference speech coding features, and the text coding features are transformed by the reversible transformation layer to obtain transformed coding features;

[0192] The transform-coded features are decoded by the stream decoder to obtain an acoustic feature sequence;

[0193] The acoustic feature sequence is input into the residual layer for residual connection to obtain speech acoustic features.

[0194] By predicting the acoustic features of speech through acoustic structure, the bottleneck of traditional acoustic models in terms of synthesis speed is effectively overcome, making the speech synthesis process smoother and more efficient, and significantly improving the quality and speed of synthesized speech.

[0195] In some optional implementations of this embodiment, the iteration module 309 is further configured to:

[0196] Calculate the adversarial loss and L1 loss between the reference speech and the predicted synthesized speech;

[0197] Calculate the similarity between the style coding features and the reference speech coding features to obtain the feature correlation loss;

[0198] The adversarial loss, the L1 loss, and the feature-related loss are weighted and summed to obtain the final loss.

[0199] By using various loss methods such as adversarial loss, L1 loss, and feature correlation loss to optimize the model, we can improve the model's prediction accuracy, enhance its robustness, promote its generalization ability, and improve the naturalness and intelligibility of synthesized speech.

[0200] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0201] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0202] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0203] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for text-guided speech synthesis methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0204] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the text-guided speech synthesis method.

[0205] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0206] By acquiring speech and text datasets in different languages ​​and performing style labeling and scene noise injection on the speech datasets, multilingual text-guided speech synthesis can be achieved. This enables the model to learn to accurately recognize and process speech signals in noisy environments, thereby improving the model's generalization ability and robustness. At the same time, it simulates real-world scenarios, enhancing the multi-scenario applicability of speech synthesis. By inputting reference speech and text datasets into a pre-built acoustic model, the style encoder, reference encoder, text encoder, acoustic structure, and vocoder of the acoustic model are trained to obtain the final speech synthesis model. This improves the naturalness and realism of speech synthesis, enhances the model's adaptability and generalization ability, and improves the efficiency and real-time performance of speech synthesis.

[0207] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the text-guided speech synthesis method described above.

[0208] By acquiring speech and text datasets in different languages ​​and performing style labeling and scene noise injection on the speech datasets, multilingual text-guided speech synthesis can be achieved. This enables the model to learn to accurately recognize and process speech signals in noisy environments, thereby improving the model's generalization ability and robustness. At the same time, it simulates real-world scenarios, enhancing the multi-scenario applicability of speech synthesis. By inputting reference speech and text datasets into a pre-built acoustic model, the style encoder, reference encoder, text encoder, acoustic structure, and vocoder of the acoustic model are trained to obtain the final speech synthesis model. This improves the naturalness and realism of speech synthesis, enhances the model's adaptability and generalization ability, and improves the efficiency and real-time performance of speech synthesis.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0210] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A text-guided speech synthesis method, characterized in that, Includes the following steps: Acquire speech datasets and text datasets in different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data; Style labeling and scene noise injection are performed on the speech dataset to obtain a reference speech set; The reference speech set and the text dataset are input into a pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder; The style encoder encodes the style labels of the text dataset and the reference speech set to obtain style-encoded features; The reference speech in the reference speech set is encoded by the reference encoder to obtain the reference speech coding features; The text dataset is encoded using the text encoder to obtain text encoding features; The style coding features, the reference speech coding features, and the text coding features are input into the acoustic structure to obtain speech acoustic features; The vocoder is used to synthesize waveforms from the acoustic features of the speech to obtain predicted synthesized speech. According to the preset loss function, the loss is calculated based on the reference speech and the predicted synthesized speech. The model parameters are adjusted based on the loss, and the training continues iteratively until the iteration stopping condition is met, so as to obtain the final speech synthesis model. The text to be converted is obtained and input into the speech synthesis model to obtain the target synthesized speech.

2. The text-guided speech synthesis method according to claim 1, characterized in that, The steps of performing style labeling and scene noise injection on the speech dataset to obtain the reference speech set include: The style label of each voice data in the voice dataset is determined according to a preset style dimension, and the voice data is labeled based on the style label to obtain a labeled voice set; A scene noise dataset is obtained, and a data fusion algorithm is used to fuse the scene noise dataset with the labeled speech set to obtain a reference speech set containing noise.

3. The text-guided speech synthesis method according to claim 2, characterized in that, The step of determining the style label for each piece of speech data in the speech dataset according to a preset style dimension includes: The speech data is analyzed using a speech signal processing algorithm to obtain the corresponding speech rate, pitch, and signal-to-noise ratio; Determine the speech rate label and pitch label of the speech data based on the speech rate and the pitch. The signal-to-noise ratio is compared with a preset signal-to-noise ratio threshold to obtain a signal-to-noise ratio label; The speech dataset is input into the trained emotion recognition model for emotion classification to obtain the corresponding emotion labels; Using a trained gender recognition model, the gender of the speaker in the speech dataset is determined, and the corresponding gender label is obtained. The speech dataset is input into the trained accent recognition model to determine the accent type of the speech data, and an accent label is obtained based on the accent type. The speech rate label, pitch label, signal-to-noise ratio label, emotion label, gender label, and accent label are combined to obtain the style label.

4. The text-guided speech synthesis method according to claim 1, characterized in that, The style encoder includes a BERT embedding layer, a spatial expansion layer, and a style encoding layer. The step of encoding style labels of the text dataset and the reference speech set using the style encoder to obtain style encoded features includes: The style labels of the text dataset and the reference speech set are vectorized using the BERT embedding layer to obtain style embedding vectors. The style embedding vector is input into the spatial extension layer, and the style extension vector is obtained by concatenating the style embedding vector and the introduced style cue vector. Style features are extracted from the style extension vector using the self-attention mechanism of the style encoding layer to obtain style encoding features.

5. The text-guided speech synthesis method according to claim 1, characterized in that, The text encoder includes a text embedding layer, a Transformer encoding layer, and a pooling layer. The step of encoding the text dataset using the text encoder to obtain text encoding features includes: The text embedding layer performs vector embedding on the text data in the text dataset to obtain text embedding vectors; The text embedding vector is input into the Transformer encoding layer, and semantic features are extracted through a multi-layer self-attention mechanism to obtain semantic encoding features; The semantic encoded features are pooled through the pooling layer to obtain text encoded features.

6. The text-guided speech synthesis method according to claim 1, characterized in that, The acoustic structure includes a reversible transform layer, a stream decoder, and a residual layer. The step of inputting the style coding features, the reference speech coding features, and the text coding features into the acoustic structure to obtain speech acoustic features includes: The style coding features, the reference speech coding features, and the text coding features are transformed by the reversible transformation layer to obtain transformed coding features; The transform-coded features are decoded by the stream decoder to obtain an acoustic feature sequence; The acoustic feature sequence is input into the residual layer for residual connection to obtain speech acoustic features.

7. The text-guided speech synthesis method according to claim 1, characterized in that, The step of calculating the loss based on the reference speech and the predicted synthesized speech according to a preset loss function includes: Calculate the adversarial loss and L1 loss between the reference speech and the predicted synthesized speech; Calculate the similarity between the style coding features and the reference speech coding features to obtain the feature correlation loss; The adversarial loss, the L1 loss, and the feature-related loss are weighted and summed to obtain the final loss.

8. A text-guided speech synthesis device, characterized in that, include: The acquisition module is used to acquire speech datasets and text datasets in different languages, wherein the speech data in the speech dataset and the text data in the text dataset are paired data; The annotation injection module is used to annotate the speech dataset with style labels and inject scene noise to obtain a reference speech set; An input module is used to input the reference speech set and the text dataset into a pre-built acoustic model, wherein the acoustic model includes a style encoder, a reference encoder, a text encoder, an acoustic structure, and a vocoder; The style encoding module is used to encode the style labels of the text dataset and the reference speech set through the style encoder to obtain style encoding features; The speech encoding module is used to encode the reference speech of the reference speech set through the reference encoder to obtain the reference speech encoding features; A text encoding module is used to encode the text dataset using the text encoder to obtain text encoding features; An acoustic feature extraction module is used to input the style coding features, the reference speech coding features, and the text coding features into the acoustic structure to obtain speech acoustic features; The speech prediction module is used to synthesize the acoustic features of the speech using the vocoder to obtain predicted synthesized speech. The iterative module is used to calculate the loss based on the reference speech and the predicted synthesized speech according to a preset loss function, adjust the model parameters based on the loss, and continue iterative training until the iteration stopping condition is met to obtain the final speech synthesis model. The speech synthesis module is used to acquire the text to be converted, input it into the speech synthesis model, and obtain the target synthesized speech.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the text-guided speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the text-guided speech synthesis method as described in any one of claims 1 to 7.

Images