Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for the financial field and the medical field. The method comprises the steps of obtaining a target emotion text, and performing text coding on the target emotion text to obtain a target emotion vector; obtaining the voice of a speaker, and carrying out speaker coding on the voice of the speaker to obtain a target speaker attribute vector; performing emotion simulation on the target emotion vector and the target speaker attribute vector through a probability control adapter to obtain Gaussian distribution parameters, and generating emotion style simulation information according to the Gaussian distribution parameters and random noise; obtaining a target text, and performing semantic conversion on the emotion style simulation information and the target text to obtain a comprehensive semantic vector; and performing speech synthesis according to the speaker attribute vector and the comprehensive semantic vector to obtain a target synthesized speech. According to the method and the device, subtle rhythm changes can be captured, and the speech synthesis accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the financial and medical fields. In particular, it relates to a speech synthesis method and device, electronic device, and storage medium. Background Technology

[0002] In the field of artificial intelligence, speech expressing specific emotional styles can be synthesized based on emotion categories and text. For example, in the insurance sector of finance, intelligent insurance customer service can output speech with professional, trustworthy, reassuring, and patient emotional styles to customers. This speech can be used to explain product features, rate structures, claims processes, etc., improving customer understanding and trust. As another example, in patient communication and education scenarios in the medical field, intelligent medical assistants can output speech with gentle, encouraging, and patient emotional styles to patients. This speech can be used to interpret symptoms and provide medication guidance.

[0003] In the process of speech synthesis, given the same input text and emotion category (such as anger), the speech synthesis model always generates almost the same speech output. This fails to simulate the inherent uncertainty and diversity in human speech (e.g., the same sentence may have subtle prosodic variations in different contexts), resulting in poor quality synthesized speech, such as sounding mechanical, monotonous, and lifeless. Summary of the Invention

[0004] The main objective of this application is to propose a speech synthesis method, apparatus, electronic device, and storage medium that can solve the technical problem of poor quality of synthesized speech caused by the inability to simulate the inherent uncertainty and diversity in human speech, and can capture subtle prosodic changes, thereby improving the accuracy of synthesized speech.

[0005] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising: Obtain the target sentiment text, and perform text encoding on the target sentiment text to obtain the target sentiment vector; Acquire the speaker's voice, perform speaker encoding on the speaker's voice, and obtain the target speaker attribute vector; The target emotion vector and the target speaker attribute vector are simulated using a pre-trained probabilistic control adapter to obtain Gaussian distribution parameters, and emotion style simulation information is generated based on the Gaussian distribution parameters and random noise; wherein the probabilistic control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; Obtain the target text, perform semantic transformation on the emotional style simulation information and the target text, and obtain a comprehensive semantic vector; Speech synthesis is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

[0006] Optionally, before performing emotion simulation on the target emotion vector and the speaker attribute vector using a pre-trained probability control adapter to obtain emotion style simulation information, the method further includes: Obtain sample sentiment text, perform text encoding on the sample sentiment text to obtain sample sentiment vector; Acquire sample speaker speech, perform speech encoding on the sample speaker speech to obtain sample speaker attribute vector; Multiple rounds of emotion simulation are performed on the sample emotion vector and the sample speaker attribute vector through an initial neural network model to obtain multiple sample Gaussian distribution parameters, and sample emotion style simulation information is generated based on each sample Gaussian distribution parameter and sample random noise. Obtain sample text, perform speech conversion on each sample sentiment style simulation information and the sample text, and obtain the sample text sentiment semantic vector; Speech reconstruction is performed on the sentiment semantic vector of each sample text to obtain the sample synthesized speech; The initial neural network model is trained based on synthesized speech from at least two of the samples to obtain the probability control adapter.

[0007] Optionally, training the initial neural network model based on synthesized speech from at least two of the samples to obtain the probability control adapter includes: All the synthesized speech samples are filtered to obtain the best and worst synthesized speech samples. The initial neural network model is trained by invoking a reinforcement learning algorithm and synthesizing speech based on the best sample and the worst sample to obtain the probability control adapter.

[0008] Optionally, the step of filtering all the synthesized speech samples to obtain the best and worst synthesized speech samples includes: Word detection is performed on the synthesized speech of each sample to obtain the word error rate; Character detection is performed on the synthesized speech of each sample to obtain the character error rate; Each synthesized speech sample is scored for prosody, and a prosody score is obtained. The speech quality score of each sample synthesized speech is obtained by fusing the word error rate, the character error rate and the prosody score. All the synthesized speech samples are sorted according to the speech quality score to obtain the best synthesized speech sample and the worst synthesized speech sample.

[0009] Optionally, the step of text encoding the target sentiment text to obtain a target sentiment vector includes: text encoding the target sentiment text using a pre-trained text sentiment encoder to obtain the target sentiment vector; The method further includes: Obtain sample emotional speech; The large language model is invoked to perform sentiment analysis on the sample emotional speech, resulting in sample sentiment description text; The sample sentiment description text is text-encoded using an initial text encoder to obtain the sample text sentiment vector; The sample emotional speech is encoded to obtain the sample speech emotion vector; The initial text encoder is trained based on the sample text sentiment vector and the sample speech sentiment vector to obtain the text sentiment encoder.

[0010] Optionally, training the initial text encoder based on the text sentiment vector and the speech sentiment vector to obtain the text sentiment encoder includes: Similarity is calculated between the text sentiment vector and the speech sentiment vector to obtain positive sentiment loss data; Obtain random text, encode the random text using the initial text encoder, and obtain a random text sentiment vector; The similarity between the random text sentiment vector and the speech sentiment vector is calculated to obtain negative sentiment loss data; The initial text encoder is adjusted based on the positive sentiment loss data and the negative sentiment loss data to obtain the text sentiment encoder.

[0011] Optionally, the step of synthesizing speech based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech includes: Stream matching is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the speech Mel spectrum; The speech spectrum of the speech is reconstructed to obtain the target synthesized speech.

[0012] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising: The text encoding module is used to acquire the target sentiment text, perform text encoding on the target sentiment text, and obtain the target sentiment vector. The speaker encoding module is used to acquire speaker speech, perform speaker encoding on the speaker speech, and obtain a target speaker attribute vector; The emotion simulation module is used to simulate the emotion of the target emotion vector and the target speaker attribute vector through a pre-trained probability control adapter, obtain Gaussian distribution parameters, and generate emotion style simulation information based on the Gaussian distribution parameters and random noise; wherein, the probability control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; The semantic conversion module is used to acquire the target text, perform semantic conversion on the sentiment style simulation information and the target text, and obtain a comprehensive semantic vector. The speech synthesis module is used to perform speech synthesis based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method described in the first aspect.

[0015] The speech synthesis method, apparatus, electronic device, and storage medium proposed in this application extract a target emotion vector from the target emotional text and a target speaker attribute vector from the speaker's speech during the speech synthesis process. Instead of directly synthesizing speech based on these two vectors, a pre-trained probabilistic control adapter simulates the emotion of the target emotion vector and the target speaker attribute vector to obtain emotional style simulation information. Utilizing the probabilistic control adapter's ability to capture subtle prosodic changes, the emotional style simulation information reflects the potential correlation between the emotional style indicated by the target emotional text and the speaker indicated by the speaker's speech, effectively simulating the inherent uncertainty and diversity in human speech. Next, semantic transformation is performed on the emotional style simulation information and the target text to obtain a comprehensive semantic vector; this comprehensive semantic vector indicates the speech content and emotional style. Finally, speech synthesis is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech, which refers to speech spoken by the speaker corresponding to the target emotional text and possessing the emotional style corresponding to the target emotional text. In summary, this application can capture subtle prosodic changes during the synthesis of speech containing a specific emotional style, improving the accuracy of synthesized speech.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application; Figure 2 This is a flowchart of a speech synthesis method provided in another embodiment of this application; Figure 3 yes Figure 2 The flowchart for step 205 in the document; Figure 4 This is a flowchart of a speech synthesis method provided in another embodiment of this application; Figure 5 yes Figure 4 The flowchart for step 406 in the document; Figure 6 yes Figure 5 The flowchart for step 501 in the document; Figure 7 This is a block diagram of the module structure of the speech synthesis device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0023] Large Language Models (LLMs): LLMs are deep learning-based language models capable of understanding, generating, and transforming natural language text. Typical characteristics include a large number of parameters, massive amounts of training data, and powerful contextual understanding capabilities. Common LLM architectures include Transformers (such as GPT, BERT, T5, etc.). Among these, autoregressive models (such as the GPT series) excel in text generation, while encoder / decoder models (such as BERT, RoBERTa) are strong in text understanding tasks.

[0024] Existing text-to-speech (TTS) technologies, especially those based on large language models (such as IndexTTS2), have made significant progress in emotional expression and duration control. These models have the following characteristics: 1. Autoregressive framework: By introducing a duration embedding vector and sharing weights with the semantic location encoding table, the autoregressive model can accurately generate speech of the corresponding duration based on the specified number of tokens, solving the problem that traditional autoregressive TTS models cannot strictly control the duration.

[0025] 2. Decoupling of Emotion and Timbre: Gradient Reversal Layer (GRL) adversarial training technique is adopted to force the emotional features extracted from emotional cues to eliminate speaker identity information as much as possible, thereby achieving the separation of emotional expression and speaker timbre, and allowing the combination and transfer of emotions from different speakers.

[0026] 3. Text feature enhancement: The hidden state of the T2S module's Transformer (rich in text context information) is fused into the input of the S2M module, which effectively improves speech clarity and intelligibility under high emotional intensity and reduces word error rate.

[0027] 4. Multi-stage training strategy: By training in stages (basic ability training, emotional ability training, and global fine-tuning), the model's emotional expression ability and robustness are steadily improved in the absence of large-scale high-quality emotional data.

[0028] Despite the success of the above model, the existing technology still has the following obvious defects and shortcomings: 1. Determinism and Uniqueness of Control Signals: The model's emotion control mechanism (emotion vector, duration embedding) is deterministic. For the same input text and control conditions, the model always generates almost identical speech outputs. This fails to simulate the inherent uncertainty and diversity in human speech; for example, the same sentence can have subtle prosodic variations in different contexts, resulting in a mechanical, monotonous, and lifeless output.

[0029] 2. Emotion control relies on predefined and discrete categories: its Text-to-Emotion (T2E) module depends on a predefined set of basic emotions. This approach suffers from an "emotional granularity bottleneck": it cannot express complex or mixed emotions—for example, complex emotions such as "bitter smile," "nostalgia," and "suppressed anger" cannot be described. It also suffers from poor scalability—the model's expressiveness is limited by the predefined emotion categories; adding new emotions requires re-collecting data and retraining the model. Furthermore, control is not intuitive—users can only choose from a limited set of categories, making it impossible to use richer, more precise natural language for descriptive control.

[0030] Based on this, embodiments of this application propose a speech synthesis method, a speech synthesis device, an electronic device, and a computer-readable storage medium. Specifically, a lightweight probability control adapter is introduced, which can simulate the uncertainty and diversity of human speech, capture subtle prosodic changes, and improve the accuracy of synthesized speech.

[0031] The speech synthesis method provided in this application is specifically applied to the field of deep learning in the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to terminals and servers, or it can be software running on the server. The server can be configured as an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the speech synthesis method, but it is not limited to the above forms.

[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0033] This application provides a speech synthesis method, a speech synthesis device, an electronic device, and a computer-readable storage medium, which are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0034] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to the user's identity or characteristics, such as voice data and text data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0035] Reference Figure 1 , Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application, which may include, but is not limited to, steps 101 to 105.

[0036] Step 101: Obtain the target sentiment text, perform text encoding on the target sentiment text, and obtain the target sentiment vector; Step 102: Obtain the speaker's voice, perform speaker encoding on the speaker's voice, and obtain the target speaker attribute vector; Step 103: Perform emotion simulation on the target emotion vector and the target speaker attribute vector through a pre-trained probabilistic control adapter to obtain Gaussian distribution parameters, and generate emotion style simulation information based on the Gaussian distribution parameters and random noise; wherein, the probabilistic control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; Step 104: Obtain the target text, perform semantic transformation on the sentiment style simulation information and the target text, and obtain a comprehensive semantic vector; Step 105: Speech synthesis is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

[0037] In steps 101 to 105 of this embodiment, during speech synthesis, a target emotion vector is extracted from the target emotion text, and a target speaker attribute vector is extracted from the speaker's speech. Speech is not directly synthesized based on these two vectors. Instead, a pre-trained probabilistic control adapter is used to simulate the emotion of the target emotion vector and the target speaker attribute vector, obtaining emotion style simulation information. Utilizing the probabilistic control adapter's ability to capture subtle prosodic changes, the emotion style simulation information reflects the potential correlation between the emotion style indicated by the target emotion text and the speaker indicated by the speaker's speech, effectively simulating the inherent uncertainty and diversity in human speech. Next, semantic transformation is performed on the emotion style simulation information and the target text to obtain a comprehensive semantic vector. This comprehensive semantic vector indicates the speech content and the speech emotion style. Finally, speech synthesis is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech, which refers to speech spoken by the speaker corresponding to the speaker's speech and possessing the emotion style corresponding to the target emotion text. In summary, this application can solve the technical problem of poor quality of synthesized speech caused by the inability to simulate the inherent uncertainty and diversity in human speech, and can capture subtle prosodic changes, thereby improving the accuracy of synthesized speech.

[0038] For example, in the insurance sector of finance, intelligent insurance customer service can deliver voice messages with professional, trustworthy, comforting, and patient emotional styles to customers. This voice can be used to explain product features, rate structures, claims processes, etc., thereby enhancing customer understanding and trust.

[0039] For example, in patient communication and education scenarios in the medical field, intelligent medical assistants can output voice messages with gentle, encouraging, and patient emotional styles to patients, which are used for interpreting medical conditions and providing medication guidance.

[0040] In step 101 of some embodiments, the target emotional text is acquired, and text encoding is performed on the target emotional text to obtain a target emotional vector. The target emotional text is text describing the emotional style, mainly using natural language to describe how the emotions in the target text used for synthesized speech change. For example, the target emotional style text may be "bitter smile," "nostalgic," or "suppressed anger."

[0041] In one example, the process of obtaining the target emotional text may include: acquiring the speaker's speech; calling a large language model to perform an emotional description of the speaker's speech, thus obtaining the target emotional text. Alternatively, the target emotional text can be obtained through a text input box.

[0042] In one example, the target sentiment text can be encoded using a pre-trained text sentiment encoder to obtain the target sentiment vector.

[0043] In one embodiment, prior to step 101, refer to Figure 2 Speech synthesis methods may also include: Step 201: Obtain sample emotional speech; Step 202: Call the large language model to perform sentiment analysis on the sample emotional speech to obtain the sample sentiment description text; Step 203: The sample sentiment description text is text encoded using the initial text encoder to obtain the sample text sentiment vector; Step 204: Encode the sample emotional speech to obtain the sample speech emotion vector; Step 205: Train the initial text encoder based on the sample text sentiment vector and the sample speech sentiment vector to obtain the text sentiment encoder.

[0044] In step 201, the sample emotional speech is speech containing emotional information. For example, if a speaker delivers a speech with a "bitter smile" emotional style, then that speech is a sample emotional speech containing the emotional information of "bitter smile". Sample emotional speech can be acquired through speech acquisition devices such as microphones. Alternatively, it can be obtained from a speech database (such as a dialogue speech database).

[0045] In step 202, a powerful large language model is used to input the sample emotional speech into the large language model, which then outputs a description of the emotional changes in the sample emotional speech, resulting in a sample emotional description text. At this point, the sample emotional speech and the sample emotional description text are data models indicating the same emotional style; that is, the sample emotional speech and the emotional description text can be used as a data pair for training the text emotion encoder.

[0046] In one example, multiple emotional speech samples can be used to generate individual emotional description texts for each sample, resulting in a dataset containing multiple data pairs. Each data pair includes a sample emotional speech and a sample emotional description text. For example, the emotional description text generated from the sample emotional speech using LLM might be "This speech begins with joy but gradually transitions to a slightly sad tone," thus forming a (sample emotional speech, sample emotional description text) data pair. To ensure data quality, the descriptions generated by LLM can be manually verified or filtered, and multi-round enhancement strategies may be introduced, such as using different LLM cue words to generate diverse descriptions.

[0047] In step 203, the initial text encoder is a neural network model. The sample sentiment description text is input into the initial text encoder, which encodes the text and outputs a sample text sentiment vector. The initial text encoder can be a Transformer-based model, such as BERT or CLIP, capable of encoding natural language sentiment descriptions into fixed-dimensional text embedding vectors.

[0048] In step 204, the sample emotional speech can be encoded using a speech emotion encoder to obtain a sample speech emotion vector. A speech emotion encoder refers to a model or component used to extract, embed, or encode emotional information from a speech signal into a vector representation.

[0049] Speech emotion encoders can include: (1) encoders based on handcrafted features: (1.1) MFCC / spectral feature + statistical feature encoding: MFCC, pitch, loudness, spectral centroid and other features are concatenated and encoded through fully connected layers or temporal convolutional layers. (1.2) D-correlation feature encoder (Delta, Delta-Delta): The first-order / second-order difference features of the time series are aggregated to obtain the temporal representation of the emotion information. (2) General encoders based on deep learning: (2.1) Convolutional neural network (CNN) encoder: The spectrogram (such as log-MinMel, FBank) is convolved to extract local time-frequency features, and finally global pooling is used to form a fixed-dimensional vector. (2.2) Recurrent neural network (RNN) / bidirectional RNN encoder: The frame-by-frame features are modeled temporally. Commonly used ones are LSTM and GRU, which output one or a set of emotion embeddings.

[0050] In one example, the pre-trained emotion perceiver conditioner from IndexTTS2 is used as a speech emotion encoder to encode the emotional speech input into a fixed-dimensional speech embedding vector. This encoder, originally used to extract emotional features, is fine-tuned here to fit the joint embedding space.

[0051] In step 205, the initial text encoder can be trained based on the sample text sentiment vector and the sample speech sentiment vector to obtain the text sentiment encoder.

[0052] The advantage of the embodiments of steps 201 to 205 above is that a text emotion encoder is obtained by training with data pairs of sample emotional speech and sample emotional description text, thereby achieving natural language control over emotional style.

[0053] In one embodiment, reference is made to Figure 3 Step 205 may include: Step 301: Calculate the similarity between the text sentiment vector and the speech sentiment vector to obtain positive sentiment loss data; Step 302: Obtain random text, encode the random text using the initial text encoder, and obtain the random text sentiment vector; Step 303: Calculate the similarity between the random text sentiment vector and the speech sentiment vector to obtain negative sentiment loss data; Step 304: Adjust the parameters of the initial text encoder based on the positive and negative sentiment loss data to obtain the text sentiment encoder.

[0054] The goal of this embodiment is to bring matching data pairs (sample emotional speech, sample emotional description text) closer together in the embedding space, while pushing unmatched data pairs (sample emotional speech, random text) further apart. Specifically, the InfoNCE (Normalized Temperature-Scaled Cross Entropy) loss function is used for training. After training through contrastive learning, the output vectors of the text encoder and the speech emotion encoder are mapped to the same joint embedding space, where similar emotions and descriptions are close together in the embedding space.

[0055] The advantage of the embodiments of steps 301 to 304 above is that the text sentiment encoder is trained using a contrastive learning mechanism, which improves the accuracy of obtaining emotional style from natural language.

[0056] It should be noted that during the inference phase, the user inputs any natural language sentiment description (such as "like a weary detective"), which the text sentiment encoder encodes into a sentiment vector. This sentiment vector is directly used as the conditional input to the probabilistic control adapter, enabling fine-grained and intuitive sentiment simulation.

[0057] In step 102 of some embodiments, the speaker's voice is acquired, and the speaker's voice is encoded to obtain a target speaker attribute vector. The speaker's voice is used to provide speaker attributes. For example, if it is desired that the synthesized voice represents the voice of an intelligent customer service representative, then a segment of voice spoken by the intelligent customer service representative is used as the speaker's voice. For example, the voice content of the speaker's voice is "I am intelligent customer service representative A".

[0058] Speech encoding can be performed on the speaker's speech to obtain the target speaker attribute vector. A speaker encoder usually refers to a type of encoding component used to extract a representation vector related to the speaker's identity from speech. Speaker encoders can include: (1) Speaker encoders based on traditional features, such as Gaussian mixture model-hidden Markov model (GMM-UBM) + i-vector / x-vector. i-vector: Compresses the speaker features of the entire speech into a fixed-dimensional vector as the speaker representation. x-vector: Speaker embeddings extracted based on time-aligned neural networks, usually fed into the subsequent classification head for training through aligned time features. Typical features: MFCC, energy, pitch, spectral contrast, etc., combined with subsequent post-processing (such as long and short time mean, reduction). (2) Speaker encoders based on deep learning: (2.1) DNN / CNN basic encoder: Extracts local time-frequency features through convolution and then aggregates them into a fixed-dimensional speaker vector. (2.2) End-to-end embedding model: Speaker Embedding Network directly outputs speaker vectors and is often used for speaker recognition / verification tasks.

[0059] In step 103 of some embodiments, an emotion simulation is performed on the target emotion vector and the target speaker attribute vector using a pre-trained probabilistic control adapter to obtain Gaussian distribution parameters, and emotion style simulation information is generated based on the Gaussian distribution parameters and random noise. The probabilistic control adapter is trained based on at least two samples of synthesized speech, and the prosodic information of any two samples of synthesized speech is different.

[0060] The probabilistic control adapter is a model designed in this application, which can be designed as a small feedforward neural network. The probabilistic control adapter includes: 1. An input layer that concatenates the target speaker attribute vector and the target emotion vector, both with dimension d, resulting in a 2d dimension after concatenation. 2. Hidden layers employ multiple linear fully connected layers, interspersed with activation functions to introduce non-linearity. 3. An output layer is a fully connected layer, outputting Gaussian distribution parameters, including mean and variance. By sampling noise from the Gaussian distribution and combining it with the mean (μ) and variance (σ) generated by the adapter, different emotional style simulation information z = μ + σ·ε (ε being random noise) can be obtained, resulting in the final synthesized speech with style differences.

[0061] In one embodiment, reference is made to Figure 4 Before step 103, the speech synthesis method may further include: Step 401: Obtain sample sentiment text, perform text encoding on the sample sentiment text to obtain sample sentiment vector; Step 402: Obtain the sample speaker's speech, perform speech encoding on the sample speaker's speech, and obtain the sample speaker's attribute vector; Step 403: Perform multiple rounds of emotion simulation on the sample emotion vector and sample speaker attribute vector through the initial neural network model to obtain multiple sample Gaussian distribution parameters, and generate sample emotion style simulation information based on the Gaussian distribution parameters and sample random noise. Step 404: Obtain sample text, perform speech conversion on the emotional style simulation information and sample text of each sample, and obtain the emotional semantic vector of the sample text. Step 405: Reconstruct the speech from the sentiment semantic vector of each sample text to obtain the synthesized speech of the sample. Step 406: Train the initial neural network model based on synthesized speech from at least two samples to obtain a probability control adapter.

[0062] In step 401, the sample emotional text is a text describing emotional style, mainly aiming to use natural language to describe how the emotions in the sample text used for synthesized speech change. For example, the sample emotional style text may be "bitter smile", "nostalgic", "suppressed anger", etc.

[0063] In one example, the process of obtaining sample sentiment text may include: acquiring sample speaker speech; calling a large language model to perform sentiment description on the sample speaker speech, thus obtaining sample sentiment text. Sample sentiment text can also be obtained through a text input box.

[0064] In one example, a pre-trained text sentiment encoder can be used to encode the sample sentiment text to obtain the sample sentiment vector.

[0065] In step 402, sample speaker speech is acquired, and speech encoding is performed on the sample speaker speech to obtain a sample speaker attribute vector. The sample speaker speech is used to provide speaker attributes. For example, if it is desired that the synthesized speech represents the voice of an intelligent customer service representative, then a segment of speech spoken by the intelligent customer service representative is used as the sample speaker speech. For example, the speech content of the sample speaker speech is "I am intelligent customer service representative B".

[0066] The speaker's speech can be encoded using a speaker encoder to obtain the speaker's attribute vector.

[0067] In step 403, the initial neural network model is designed as a small feedforward neural network. The initial neural network model includes: 1. An input layer that concatenates the target speaker attribute vector and the target emotion vector, both with dimension d, resulting in a 2d dimension after concatenation. 2. Hidden layers employ multiple linear fully connected layers, interspersed with activation functions to introduce non-linearity. 3. An output layer is a fully connected layer, outputting the Gaussian distribution parameters of the samples, including the mean and variance. By sampling sample noise from the Gaussian distribution and combining it with the mean and variance generated by the initial neural network model, different sample emotion style simulation information can be obtained, resulting in the final synthesized speech with style differences.

[0068] In step 404, the sample text is used to provide speech content. The emotional style simulation information and sample text for each sample can be converted into speech using a T2S module (such as a Transformer-based module) to obtain the emotional semantic vector of the sample text.

[0069] In step 405, the speech reconstruction module can reconstruct the sentiment semantic vector of each sample text to obtain the synthesized speech. Finally, in step 406, the initial neural network model is trained based on at least two synthesized speech samples to obtain the probability control adapter.

[0070] The advantage of the embodiments of steps 401 to 406 described above is that by using at least two samples of synthesized speech with different monopronunciation information generated based on the same input data for model training, the human speech simulation capability of the probabilistic control adapter is improved, thereby improving the accuracy of speech synthesis.

[0071] In one embodiment, reference is made to Figure 5 Step 406 may include: Step 501: Filter all synthesized speech samples to obtain the best and worst synthesized speech samples. Step 502: Call the reinforcement learning algorithm and train the initial neural network model based on the best sample synthesized speech and the worst sample synthesized speech to obtain the probability control adapter.

[0072] Specifically, by generating diverse samples of the same input data through an initial neural network model, multiple samples of synthesized speech can be collected, thereby constructing a preference data pair (the best sample synthesized speech and the worst sample synthesized speech). Then, reinforcement learning algorithms (such as PPO, DPO, etc.) are used to enable the initial neural network model to learn better generation results, thereby improving the performance of the probabilistic control adapter.

[0073] In one embodiment, reference is made to Figure 6 Step 501 may include: Step 601: Perform word detection on each sample of synthesized speech to obtain the word error rate; Step 602: Perform character detection on each sample of synthesized speech to obtain the character error rate; Step 603: Perform prosodic scoring on each sample of synthesized speech to obtain a prosodic score; Step 604: The word error rate, character error rate and prosody score are fused to obtain the speech quality score of each synthesized speech sample; Step 605: Sort all synthesized speech samples according to their speech quality scores to obtain the best and worst synthesized speech samples.

[0074] Specifically, for each sample of synthesized speech, the word error rate (WER), character error rate (SIM), and prosodic score are measured. Then, these three metrics are combined to rank the synthesized speech samples, thereby obtaining the best and worst synthesized speech samples. This improves the accuracy of speech evaluation.

[0075] In step 104 of some embodiments, the target text is used to provide speech content. A semantic transformation can be performed on the sentiment style simulation information and the target text using a T2S module to obtain a comprehensive semantic vector.

[0076] In step 105 of some embodiments, the speaker attribute vector and the comprehensive semantic vector can be synthesized by a speech synthesizer to obtain the target synthesized speech.

[0077] In one embodiment, step 105 may include: performing flow matching based on the speaker attribute vector and the comprehensive semantic vector to obtain the speech Mel spectrum; and performing speech reconstruction on the speech Mel spectrum to obtain the target synthesized speech. Flow matching can be implemented using an S2M module. Speech reconstruction can be implemented using a Vocoder (such as BigVGANv2).

[0078] In summary, the present application achieves at least the following beneficial effects: 1. Significantly improved diversity and naturalness of generated speech: Thanks to uncertainty-aware control, the model can generate multiple speech variations within a reasonable range for the same input. Furthermore, by utilizing the DPO reinforcement learning algorithm, the uniformity and mechanical feel of the generated results can be effectively avoided, making the synthesized speech closer to the varied characteristics of natural human conversation, sounding more lively and realistic. 2. Achieved infinitely fine-grained and highly intuitive emotion control: By constructing data pairs of emotional speech and emotional descriptive text, users are freed from the constraints of limited emotion labels and can use arbitrarily rich natural language to describe their desired emotional style (e.g., "like a weary detective"). This greatly expands the model's emotional expressiveness, providing unprecedented control precision and intuitiveness, and lowering the user's barrier to entry. 3. Enhanced expressive flexibility and application scope of the model: The combination of these two improvements not only generates high-quality speech but also makes it a truly "usable" and "easy-to-use" creative tool. It is well-suited for scenarios requiring a large variety of voice generation (such as game NPC dialogue) and occasions requiring precise emotional direction (such as film and television dubbing and audiobook reading), opening up broader possibilities for the application of AI voice synthesis in the creative industries.

[0079] Please see Figure 7 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. Figure 7 This is a block diagram of the module structure of a speech synthesis device provided in an embodiment of this application. The device includes: The text encoding module 701 is used to acquire the target sentiment text, perform text encoding on the target sentiment text, and obtain the target sentiment vector. Speaker encoding module 702 is used to acquire speaker speech, perform speaker encoding on speaker speech, and obtain target speaker attribute vector; The emotion simulation module 703 is used to simulate the emotion of the target emotion vector and the target speaker attribute vector through a pre-trained probabilistic control adapter, obtain Gaussian distribution parameters, and generate emotion style simulation information based on the Gaussian distribution parameters and random noise; wherein, the probabilistic control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; The semantic conversion module 704 is used to acquire the target text, perform semantic conversion on the sentiment style simulation information and the target text, and obtain a comprehensive semantic vector. The speech synthesis module 705 is used to synthesize speech based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

[0080] In one embodiment, the speech synthesis device may further include a first training module, configured to: acquire sample emotional text, perform text encoding on the sample emotional text to obtain a sample emotional vector; acquire sample speaker speech, perform speech encoding on the sample speaker speech to obtain a sample speaker attribute vector; perform multiple rounds of emotional simulation on the sample emotional vector and the sample speaker attribute vector through an initial neural network model to obtain multiple sample Gaussian distribution parameters, and generate sample emotional style simulation information based on each sample Gaussian distribution parameter and sample random noise; acquire sample text, perform speech conversion on each sample emotional style simulation information and the sample text to obtain a sample text emotional semantic vector; perform speech reconstruction on each sample text emotional semantic vector to obtain sample synthesized speech; and train the initial neural network model based on at least two sample synthesized speech samples to obtain a probability control adapter.

[0081] In one embodiment, the speech synthesis device may further include a second training module, configured to: acquire sample emotional speech; call a large language model to perform emotional analysis on the sample emotional speech to obtain sample emotional description text; perform text encoding on the sample emotional description text through an initial text encoder to obtain sample text emotional vector; encode the sample emotional speech to obtain sample speech emotional vector; and train the initial text encoder based on the sample text emotional vector and the sample speech emotional vector to obtain a text emotional encoder.

[0082] It should be noted that the specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0083] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0084] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the speech synthesis method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0085] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described speech synthesis method.

[0086] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0088] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0092] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the target sentiment text, and perform text encoding on the target sentiment text to obtain the target sentiment vector; Acquire the speaker's voice, perform speaker encoding on the speaker's voice, and obtain the target speaker attribute vector; The target emotion vector and the target speaker attribute vector are simulated using a pre-trained probabilistic control adapter to obtain Gaussian distribution parameters, and emotion style simulation information is generated based on the Gaussian distribution parameters and random noise; wherein the probabilistic control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; Obtain the target text, perform semantic transformation on the emotional style simulation information and the target text, and obtain a comprehensive semantic vector; Speech synthesis is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

2. The method according to claim 1, characterized in that, Before performing emotion simulation on the target emotion vector and the speaker attribute vector using a pre-trained probabilistic control adapter to obtain emotion style simulation information, the method further includes: Obtain sample sentiment text, perform text encoding on the sample sentiment text to obtain sample sentiment vector; Acquire sample speaker speech, perform speech encoding on the sample speaker speech to obtain sample speaker attribute vector; Multiple rounds of emotion simulation are performed on the sample emotion vector and the sample speaker attribute vector through an initial neural network model to obtain multiple sample Gaussian distribution parameters, and sample emotion style simulation information is generated based on each sample Gaussian distribution parameter and sample random noise. Obtain sample text, perform speech conversion on each sample sentiment style simulation information and the sample text, and obtain the sample text sentiment semantic vector; Speech reconstruction is performed on the sentiment semantic vector of each sample text to obtain the sample synthesized speech; The initial neural network model is trained based on synthesized speech from at least two of the samples to obtain the probability control adapter.

3. The method according to claim 2, characterized in that, The process of training the initial neural network model based on synthesized speech from at least two of the samples to obtain the probability control adapter includes: All the synthesized speech samples are filtered to obtain the best and worst synthesized speech samples. The initial neural network model is trained by invoking a reinforcement learning algorithm and synthesizing speech based on the best sample and the worst sample to obtain the probability control adapter.

4. The method according to claim 3, characterized in that, The step of filtering all the synthesized speech samples to obtain the best and worst synthesized speech samples includes: Word detection is performed on the synthesized speech of each sample to obtain the word error rate; Character detection is performed on the synthesized speech of each sample to obtain the character error rate; Each synthesized speech sample is scored for prosody, and a prosody score is obtained. The speech quality score of each sample synthesized speech is obtained by fusing the word error rate, the character error rate and the prosody score. All the synthesized speech samples are sorted according to the speech quality score to obtain the best synthesized speech sample and the worst synthesized speech sample.

5. The method according to any one of claims 1 to 4, characterized in that, The step of text encoding the target sentiment text to obtain the target sentiment vector includes: text encoding the target sentiment text using a pre-trained text sentiment encoder to obtain the target sentiment vector; The method further includes: Obtain sample emotional speech; The large language model is invoked to perform sentiment analysis on the sample emotional speech, resulting in sample sentiment description text; The sample sentiment description text is text-encoded using an initial text encoder to obtain the sample text sentiment vector; The sample emotional speech is encoded to obtain the sample speech emotion vector; The initial text encoder is trained based on the sample text sentiment vector and the sample speech sentiment vector to obtain the text sentiment encoder.

6. The method according to claim 5, characterized in that, The step of training the initial text encoder based on the text sentiment vector and the speech sentiment vector to obtain the text sentiment encoder includes: Similarity is calculated between the text sentiment vector and the speech sentiment vector to obtain positive sentiment loss data; Obtain random text, encode the random text using the initial text encoder, and obtain a random text sentiment vector; The similarity between the random text sentiment vector and the speech sentiment vector is calculated to obtain negative sentiment loss data; The initial text encoder is adjusted based on the positive sentiment loss data and the negative sentiment loss data to obtain the text sentiment encoder.

7. The method according to any one of claims 1 to 4, characterized in that, The step of synthesizing speech based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech includes: Stream matching is performed based on the speaker attribute vector and the comprehensive semantic vector to obtain the speech Mel spectrum; The speech spectrum of the speech is reconstructed to obtain the target synthesized speech.

8. A speech synthesis device, characterized in that, The device includes: The text encoding module is used to acquire the target sentiment text, perform text encoding on the target sentiment text, and obtain the target sentiment vector. The speaker encoding module is used to acquire speaker speech, perform speaker encoding on the speaker speech, and obtain a target speaker attribute vector. The emotion simulation module is used to simulate the emotion of the target emotion vector and the target speaker attribute vector through a pre-trained probability control adapter, obtain Gaussian distribution parameters, and generate emotion style simulation information based on the Gaussian distribution parameters and random noise; wherein, the probability control adapter is trained based on at least two sample synthesized speech, and the prosodic information of any two sample synthesized speech is different; The semantic conversion module is used to acquire the target text, perform semantic conversion on the sentiment style simulation information and the target text, and obtain a comprehensive semantic vector. The speech synthesis module is used to perform speech synthesis based on the speaker attribute vector and the comprehensive semantic vector to obtain the target synthesized speech.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images