Speech synthesis methods and apparatus, computer equipment, storage media

Abstract

This invention relates to the field of artificial intelligence technology and discloses a speech synthesis method and apparatus, computer device, and storage medium, comprising: serializing reference text to obtain a text token sequence; generating an emphasis mask token sequence, and updating the token of each target specified character in the emphasis mask token sequence from 0 to 1 in response to detecting the presence of a target specified character in the reference text; performing emotion intensity recognition on reference audio to obtain an emotion intensity token; concatenating a pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence to obtain a target fusion sequence; performing attention processing on the target fusion sequence using a large language model to obtain a target speech token sequence; and generating speech from the target speech token sequence to obtain the target speech. This invention can be applied to speech synthesis scenarios in fintech and healthcare, solving the technical problem of poor emotional expression in synthesized speech.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to speech synthesis scenarios in the fields of fintech and healthcare. In particular, it relates to a speech synthesis method and apparatus, computer equipment, and storage medium. Background Technology

[0002] In the field of artificial intelligence, synthetic speech with specific emotions can be generated based on text and audio that provides emotional context. For example, in the fintech field, text information (such as account balances, transaction details, card status, etc.) can be converted into natural and fluent speech in real time, providing self-service queries. As another example, in the healthcare field, key points or summaries of medical history spoken by doctors can be converted into speech, making it easier for patients to listen back and remember.

[0003] Currently, speech synthesis systems based on traditional deep neural networks and those based on large language models have significant advantages in terms of naturalness, cross-speaker capability, and zero-shot generalization performance. However, they still face technical bottlenecks in terms of emotional controllability (e.g., emotions such as "slight happiness" and "very angry" cannot be effectively expressed in speech), which limits the subtlety of the generated speech and makes it sound stiff and unnatural. Summary of the Invention

[0004] This invention provides a speech synthesis method and apparatus, computer equipment, and storage medium to solve the technical problem of poor emotional expression in synthesized speech.

[0005] Firstly, a speech synthesis method is provided, the method comprising: Obtain the reference text, and serialize the reference text to obtain the text token sequence; Generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0; and in response to detecting the presence of a target specified character in the reference text, update the token of each target specified character in the emphasis mask token sequence from 0 to 1. Obtain reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token; The pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence are concatenated to obtain the target fusion sequence; Attention processing is applied to the target fusion sequence using a large language model to obtain the target speech token sequence; The target speech token sequence is used to generate speech, thus obtaining the target speech.

[0006] Secondly, a speech synthesis apparatus is provided, the apparatus comprising: The text processing module is used to acquire reference text, serialize the reference text, and obtain a text token sequence; The mask generation module is used to generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and in response to detecting the presence of a target specified character in the reference text, to update the token of each target specified character in the emphasis mask token sequence from 0 to 1; The audio processing module is used to acquire reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token. The sequence generation module is used to concatenate the pre-acquired emotion category token, the emotion intensity token, the emphasis mask token sequence, and the text token sequence to obtain the target fusion sequence; The sequence processing module is used to perform attention processing on the target fusion sequence through a large language model to obtain a target speech token sequence; The speech generation module is used to generate speech from the target speech token sequence to obtain the target speech.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described speech synthesis method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0009] In the aforementioned speech synthesis method, speech synthesis device, computer equipment, and storage medium, the reference text is serialized to obtain a text token sequence; an emphasis mask token sequence with the same length as the reference text and all tokens being 0 is generated; in response to the detection of a target specified character in the reference text, the token of each target specified character in the emphasis mask token sequence is updated from 0 to 1; furthermore, the reference audio is subjected to emotion intensity recognition to obtain an emotion intensity token; then, the pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence are concatenated to obtain a target fusion sequence; next, attention processing is performed on the target fusion sequence using a large language model to obtain a target speech token sequence; finally, speech generation is performed on the target speech token sequence to obtain the target speech. In this invention, the emphasis mask token sequence can be used to highlight target specified characters in the reference text, achieving local emphasis control of speech; based on the emotion intensity token obtained from the recognition of the reference audio, global emotion intensity control of speech can be achieved; furthermore, attention processing is performed using a large language model to obtain the target speech token sequence, strictly controlling the emotion and intensity of the speech; finally, speech generation is performed on the target speech token sequence to obtain the target speech. In summary, the target speech generated by this invention can not only switch emotion categories when expressing emotions, but also present a continuous change from weak to strong, which enhances the sense of layering and realism in the speech and achieves a better emotional expression effect. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a speech synthesis method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S30; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation of step S33; Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S50; Figure 6 This is a schematic diagram of a speech synthesis device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] First, let's clarify some of the terms used in this invention: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0014] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0015] Text-to-Speech (TTS): TTS refers to converting text information into audible speech signals. It typically consists of two parts: a front-end (text processing) and a back-end (acoustic modeling and vocoder). The front-end maps text features to acoustic features, and then the vocoder converts the acoustic representation into waveform audio. Common vocoders include WaveNet, HiFi-GAN, and Parallel WaveGAN.

[0016] Large Language Model (LLM): An LLM is a model built using deep learning methods and trained on large-scale text data, capable of understanding and generating natural language. It possesses the ability to perform reasoning, answering questions, writing text, translating, summarizing, and conversing on various language tasks, and typically exhibits strong versatility and transfer learning capabilities.

[0017] Current TTS technologies can be broadly categorized into two types: TTS systems based on traditional deep neural networks and TTS systems based on large language models. While existing TTS systems offer significant advantages in naturalness, cross-speaker capability, and zero-shot generalization performance, several technical bottlenecks remain in terms of "emotional controllability." For example, current technologies commonly employ "emotional category labels" for control, such as happiness, anger, and sadness. These labels are discrete categories and cannot accurately express the continuous details of real-life emotions. Furthermore, emotions in real-world scenarios vary in intensity, such as "slight happiness" and "extreme anger," and traditional emotional labels cannot cover these intensity levels, limiting the nuances of generated speech and making it sound stiff and unnatural. Another example is the "emphasis" of individual words, a crucial component in emotional expression, involving multi-dimensional information such as fundamental frequency boosting, energy enhancement, and rhythmic variations. However, existing emphasis control techniques rely on manual annotation or specific acoustic models, which are not only costly and difficult to scale, but also lack mature solutions in existing discrete token-based TTS systems, resulting in uncontrollable or unstable emphasis effects. For example, most existing technologies cannot uniformly handle "global emotion intensity control" and "local emphasis control." Even some TTS systems with certain forms of emotion tags lack controllable local adjustment capabilities; while emphasis control systems are usually separated from emotion systems, lacking a unified input structure and training method. Furthermore, existing TTS systems based on large language models use discrete speech tokens as acoustic features, which are inherently difficult to directly carry continuous emotional information, making it difficult to transfer traditional emotion control methods that rely on continuous acoustic parameters. Existing systems lack effective control mechanisms that can directly act on LLM inputs, resulting in insufficient controllability.

[0018] To address the problems of the existing technologies mentioned above, this invention proposes a speech synthesis method, a speech synthesis device, a computer device, and a storage medium. Specifically, this invention proposes a speech synthesis method capable of simultaneously performing "global emotion intensity control" and "local emphasis control." This method is particularly suitable for TTS architectures based on large language models. By introducing emotion intensity tokens, emphasis mask tokens, and a unified input structure, the large language model can accurately and controllably express fine-grained emotions and emphasis. The speech synthesis method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The client can initiate a speech synthesis request to the server. After receiving the speech synthesis request, the server obtains the reference text, serializes the reference text to obtain a text token sequence; the server generates an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and in response to the detection of a target specified character in the reference text, updates the token of each target specified character in the emphasis mask token sequence from 0 to 1; the server obtains the reference audio, performs emotion intensity recognition on the reference audio to obtain an emotion intensity token; the server concatenates the pre-obtained emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence to obtain a target fusion sequence; the server performs attention processing on the target fusion sequence using a large language model to obtain a target speech token sequence; the server generates speech from the target speech token sequence to obtain the target speech. In this invention, an emphasis mask token sequence is used to highlight specific target characters in the reference text, achieving local emphasis control of speech. Based on emotion intensity tokens obtained from the recognition of the reference audio, global emotion intensity control of the speech is achieved. Furthermore, attention processing using a large language model yields a target speech token sequence, strictly controlling the emotion and intensity of the speech. Finally, the target speech token sequence is used to generate the target speech. In summary, the target speech generated by the server not only switches emotion categories when expressing emotion but also presents a continuous change from weak to strong, enhancing the layering and realism of the speech, resulting in good emotional expression. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0019] The speech synthesis method provided in this embodiment of the invention can also be applied to another application environment, which includes a client or a server. For example, the client acquires reference text, serializes the reference text to obtain a text token sequence; the client generates an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and updates the token of each target specified character in the emphasis mask token sequence from 0 to 1 in response to detecting the presence of a target specified character in the reference text; the client acquires reference audio, performs emotion intensity recognition on the reference audio to obtain an emotion intensity token; the client concatenates the pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence to obtain a target fusion sequence; the client performs attention processing on the target fusion sequence through a large language model to obtain a target speech token sequence; the client generates speech from the target speech token sequence to obtain the target speech. The target speech generated by the client can not only switch emotion categories when expressing emotions, but also present a continuous change from weak to strong, enhancing the sense of layering and realism in the speech, and achieving a better emotional expression effect.

[0020] The present invention will now be described in detail through specific embodiments.

[0021] Please see Figure 2 As shown, Figure 2 A schematic flowchart of a speech synthesis method provided in an embodiment of the present invention includes steps S10-S60.

[0022] S10: Obtain the reference text, serialize the reference text, and obtain the text token sequence.

[0023] The speech synthesis method provided by this invention can be applied to speech synthesis engines in various application scenarios, and the speech synthesis engine can be implemented through a server. For example, in the speech synthesis scenario of the fintech field, the speech synthesis engine can convert text information (such as account balance, transaction details, card status, etc.) into natural and fluent speech in real time, providing self-service queries. As another example, in the speech synthesis scenario of the healthcare field, the speech synthesis engine can convert the key points or disease summaries spoken by doctors into speech, making it easier for patients to listen back and remember.

[0024] Reference text: The text that provides the speech content / speaking content in speech synthesis. For example, in the insurance industry in the fintech field, the reference text is "May I ask which policy's coverage you would like to know? I can provide you with the relevant terms and explanations". After this reference text is converted into speech, it is convenient for multi-round conversations with users and can be applied to intelligent Q&A scenarios. Another example is in the medical industry in the field of healthcare, where the reference text is "May I ask which medical information you would like to know? I can provide you with the consultation process, appointment methods, and popular science explanations". After this reference text is converted into speech, it is convenient for multi-round conversations with users and can be applied to intelligent Q&A scenarios.

[0025] Tokens are the basic elements that the model uses to break down the input and generate the output. A sequence of text tokens refers to the sequence of basic units into which the reference text is broken down. Tokens can be words, parts of words, or even characters, depending on the tokenization scheme used. The token sequence is the basis for the model to understand and generate language, and when processing text, operations are usually carried out at the token level.

[0026] Serialization here is also known as tokenization, which means that tokenization is a method of dividing text into smaller units called tokens. In one example, the reference text can be tokenized through a tokenization algorithm to obtain a sequence of text tokens. Tokenization algorithms include word tokenization, etc. Word tokenization divides a piece of text into individual words based on a certain delimiter; then, different word-level tokens are formed according to the delimiter. Pre-trained word embeddings such as Word2Vec and GloVe are under word tokenization.

[0027] S20: Generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and in response to detecting the existence of a target specified character in the reference text, update the token of each target specified character in the emphasis mask token sequence from 0 to 1.

[0028] Emphasis control is an adjustment mechanism designed in the present invention, and its design goal is to achieve a speech highlighting effect at the word granularity, such as increasing the fundamental frequency, increasing the energy, and lengthening the duration, etc. Given the reference text, the present invention constructs an emphasis mask sequence that matches the number of words in the text.

[0029] The length of the reference text refers to the total number of characters contained in the reference text. For example, if the reference text is "You arereally nice today", then the length of the reference text is 5, and by default, all tokens are not emphasized. At this time, the emphasis mask token sequence is "0 0 0 0 0". If the detected target specified character is "really", then the emphasis mask token sequence is updated to "00 1 0 0". If the detected target specified characters are "really" and "nice", then the emphasis mask token sequence is updated to "00 1 1 0".

[0030] In one embodiment, the server receives a target specified character from the client.

[0031] In some embodiments of the present invention, the reference text is the response text in the current round of a multi-turn dialogue; before updating the token of each target-specified character in the emphasis mask token sequence from 0 to 1 in response to detecting the presence of a target-specified character in the reference text, the speech synthesis method may further include: obtaining the dialogue text before the current round of the multi-turn dialogue to obtain the historical dialogue text; and performing emphasis character detection on the historical dialogue text and the reference text using a large model to obtain the target-specified character.

[0032] The large model here is also a large language model. The above embodiments are mainly applied to the following multi-turn dialogue scenario: the intelligent assistant converses with the customer via voice. After the customer inputs the question text, the intelligent assistant retrieves the corresponding response text from the database, then generates the target speech based on the response text, and finally provides the target speech to the customer.

[0033] As can be seen from the above scheme, when the speech synthesis method is applied to multi-turn dialogue scenarios, it can combine historical dialogue text to detect whether there are key characters in the reference text, thereby obtaining the target specified characters. In this way, the flexibility of emotional expression can be improved in multi-turn dialogue scenarios, thus providing customers with target speech with more nuanced emotional expression.

[0034] S30: Obtain reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token.

[0035] Reference audio: Audio that provides emotional information in speech synthesis, in formats such as WAV and MP3. The goal of step S30 is to extract emotional intensity information from a reference audio and map it to discrete emotional intensity tokens.

[0036] Emotion intensity tokens can be discrete intervals, such as {0: neutral, 1: slightly happy, 2: happy, 3: angry, 4: sad, 5: surprised}, or multi-level intensity representations with continuous segments.

[0037] In some embodiments of the present invention, step S30, namely, identifying the emotional intensity of the reference audio to obtain an emotional intensity token, may include: extracting features from the reference audio to obtain audio features (Mel spectral coefficients (MFCC), fundamental frequency (F0), pitch profile, energy, time-domain / frequency-domain statistical features, acoustic features such as contrast, aliasing, etc.); classifying the audio features using emotional classification models such as ECAPA-TDNN and AConvNet to obtain emotional intensity; and mapping the emotional intensity to a token to obtain an emotional intensity token.

[0038] In other embodiments of the present invention, reference is made to Figure 3 Step S30, which involves performing emotion intensity recognition on the reference audio to obtain an emotion intensity token, may include the following steps: S31: Perform sentiment prediction on the reference audio to obtain the reference sentiment vector; S32: Calculate the distance between the reference sentiment vector and the preset neutral sentiment center vector to obtain the sentiment distance; S33: Obtain at least two emotional threshold intervals corresponding to different preset emotional intensities; S34: Based on the comparison between the emotional distance and the emotional threshold range, the target emotional threshold range is obtained, and the preset emotional intensity corresponding to the target emotional threshold range is used as the target emotional intensity; S35: Map tokens based on the target emotional intensity to obtain an emotional intensity token.

[0039] In step S31, the reference sentiment vector is located in a three-dimensional sentiment space. An independent sentiment prediction model (such as a sentiment vector extractor based on Transformer or CNN) is used to map the continuous sentiment intensity carried by the reference audio to a discretized control token adapted to LLM based on the three-dimensional sentiment space, generating the coordinates of the sentiment information of the reference audio in the three-dimensional sentiment space, i.e., the reference sentiment vector.

[0040] The three-dimensional emotional space, also known as the VAD three-dimensional emotional space, involves three dimensions: emotional value, emotional arousal, and emotional dominance.

[0041] In step S32, the neutral emotion center vector refers to the central position in the vector space that represents neutrality and is not biased towards any particular emotion polarity.

[0042] In one embodiment, the neutral emotion center vector is obtained through the following steps: obtaining a sample speech set, which includes sample speech, each sample speech having emotional three-dimensional coordinates in a three-dimensional emotion space, the emotional three-dimensional coordinates including emotional value, emotional activation, and emotional dominance; calculating the average emotional value based on the emotional value of each sample speech to obtain the average emotional value; calculating the average emotional activation based on the emotional activation of each sample speech to obtain the average emotional activation; calculating the average emotional dominance based on the emotional dominance of each sample speech to obtain the average emotional dominance; and merging the average emotional value, average emotional activation, and average emotional dominance to obtain the neutral emotion center vector.

[0043] For example, each sample voice in the sample voice set has a VAD three-dimensional coordinate (v, a, d). Calculate the average value of this coordinate for all sample voices, that is, calculate the average value of the coordinate values for the three dimensions respectively.

[0044] As can be seen from the above solution, by calculating the neutral emotion center vector in the VAD three-dimensional emotion space, the accuracy and generalization of the neutral emotion center vector are improved.

[0045] In step S33, the emotion intensity represents the strength of emotion expression in the voice, and is often represented by discrete levels (such as 0-5) or continuous scores (0.0-1.0). In one case, at least two emotion threshold intervals include a first emotion threshold interval and a second emotion threshold interval with increasing emotion intensity. In another case, at least two emotion threshold intervals include a first emotion threshold interval, a second emotion threshold interval, and a third emotion threshold interval with increasing emotion intensity.

[0046] In some embodiments of the present invention, referring to Figure 4 , step S33, that is, obtaining at least two emotion threshold intervals corresponding to different preset emotion intensities, may include the following steps: S331: Calculate the distance between the emotion three-dimensional coordinate of each sample voice and the neutral emotion center vector to obtain a candidate distance; S332: Based on the candidate distances of each sample voice, perform interval partitioning to obtain a first emotion threshold interval, a second emotion threshold interval, and a third emotion threshold interval.

[0047] In step S331, the distance between the emotion three-dimensional coordinate of each sample voice and the neutral emotion center vector can be calculated by the Euclidean distance formula to obtain a candidate distance.

[0048] In step S332, the upper limit value of the first emotion threshold interval is less than the lower limit value of the second emotion threshold interval, and the upper limit value of the second emotion threshold interval is less than the lower limit value of the third emotion threshold interval. For example, the first emotion threshold interval: 0 < d ≤ T1 → weak intensity; the second emotion threshold interval: T1 < d ≤ T2 → medium intensity; the third emotion threshold interval: d > T2 → strong intensity.

[0049] As can be seen from the above solution, based on the interval partitioning of the distance between the sample voice and the neutral emotion center vector, the accuracy and generalization of the interval partitioning are improved.

[0050] In one embodiment, step S332, which is to divide the interval based on the candidate distance of each sample speech to obtain a first emotion threshold interval, a second emotion threshold interval, and a third emotion threshold interval, may include the following steps: sorting the candidate distances of each sample speech to obtain a minimum distance and a maximum distance; dividing the distance interval formed based on the minimum distance and the maximum distance into three equal parts to obtain the first emotion threshold interval, the second emotion threshold interval, and the third emotion threshold interval.

[0051] For example, by traversing the entire sample speech set, let the minimum distance be a and the maximum distance be b, and divide it into three equal parts, we can get T1=2×a / 3+b / 3, T2=a / 3+2×b / 3, where less than T1 is weak intensity, greater than T2 is strong intensity, and the middle part is medium intensity.

[0052] As can be seen from the above scheme, the minimum and maximum distances among the candidate distances are used to divide the three emotion threshold intervals, which reduces the algorithm complexity and thus improves the efficiency of speech synthesis.

[0053] S40: Concatenate the pre-acquired sequence of sentiment category tokens, sentiment intensity tokens, emphasis mask tokens, and text tokens to obtain the target fusion sequence. Specifically, the concatenation is performed strictly in the order of sentiment category tokens, sentiment intensity tokens, emphasis mask tokens, and text tokens to obtain the target fusion sequence, which is {[sentiment category tokens][sentiment intensity tokens][emphasis mask tokens][text tokens]}.

[0054] In one embodiment, the emotion category can be specified directly or extracted from the reference audio along with an emotion intensity token.

[0055] S50: The target fusion sequence is processed by attention through a large language model to obtain the target speech token sequence. Specifically, the attention mechanism in the large language model obtains high weights from the preceding sequence, so that it participates in the influence each time a speech token is generated, strictly controlling the emotion and intensity of the generated speech.

[0056] In some embodiments of the present invention, reference is made to... Figure 5 Step S50, which involves performing attention processing on the target fusion sequence using a large language model to obtain the target speech token sequence, may include the following steps: S51: The target fusion sequence is processed by the large language model to obtain the first speech token, and the first speech token is concatenated to the end of the target fusion sequence to obtain the first fusion sequence; S52: The first fusion sequence is processed by a large language model to obtain a second speech token, which is then concatenated to the end of the first fusion sequence to obtain the second fusion sequence. S53: The second fusion sequence is processed by a large language model to obtain a third speech token; S54: Concatenate the first voice token, the second voice token, and the third voice token to obtain the target voice token sequence.

[0057] Specifically, the entire large language model is composed of a multi-layered transformer neural network. The target fusion sequence, containing [emotion category token][emotion intensity token][emphasis mask token sequence][text token sequence], is input into the large language model and sequentially processed by each transformer layer. After passing through a complete LLM module, a new voice token is generated. This voice token is then appended to the end of the target fusion sequence, resulting in {[emotion category token][emotion intensity token][emphasis mask token sequence][text token sequence][voice token]}. The LLM module then performs further computation to generate a second new voice token, which is also appended to the end of the sequence, resulting in {[emotion category token][emotion intensity token][emphasis mask token sequence][text token sequence][voice token][voice token]}. This process is repeated until a stopping condition is met (usually a stopping token is used; the stopping condition is met when the generated new voice token is the stopping token).

[0058] As can be seen from the above scheme, multiple attention calculations can be used to generate target speech tokens corresponding to four dimensions: emotion category, emotion intensity, emphasis mask, and reference text, based on the target fusion sequence. This improves the speech tokens' ability to express emotions and makes the emotional expression of the generated target speech more subtle.

[0059] S60: Generate speech from the target speech token sequence to obtain the target speech. Specifically, map the target speech token sequence to acoustic control vectors (e.g., emotion vector, timbre vector, speech rate, volume, etc.); decode the acoustic control vectors using a vocoder to obtain the target speech.

[0060] When this invention is applied to production operations, it can have the following beneficial effects: (1) By constructing a unified global emotion intensity and local emphasis control mechanism, the speech synthesis system can present fine-grained emotion changes in a way that is closer to natural language expression. The introduction of emotion intensity tokens enables the speech synthesis system to not only switch emotion categories when expressing emotions, but also to present continuous changes from weak to strong, which enhances the sense of hierarchy and realism in speech.

[0061] (2) The local emphasis control mechanism further enhances speech expression capabilities, enabling the speech synthesis system to accurately highlight key information in sentences, thereby improving semantic clarity and emotional expressiveness. The design of the emphasis mask token is fully compatible with TTS systems based on large language models, requiring no additional annotation or complex acoustic modeling, significantly reducing development costs.

[0062] (3) It can significantly improve the controllability, flexibility and expressiveness of speech synthesis, and provide a new solution for high-quality and controllable emotional speech generation. It has broad engineering and promotional value for various application scenarios such as intelligent speech interaction, education and training, and entertainment content generation.

[0063] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one with the speech synthesis methods described in the above embodiments. For example... Figure 6 As shown, the speech synthesis device includes a text processing module 101, a mask generation module 102, an audio processing module 103, a sequence generation module 104, a sequence processing module 105, and a speech generation module 106. Detailed descriptions of each functional module are as follows: Text processing module 101 is used to acquire reference text, serialize the reference text, and obtain a text token sequence; The mask generation module 102 is used to generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and in response to detecting the presence of a target specified character in the reference text, to update the token of each target specified character in the emphasis mask token sequence from 0 to 1; The audio processing module 103 is used to acquire reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token. The sequence generation module 104 is used to concatenate the pre-acquired emotion category token, the emotion intensity token, the emphasis mask token sequence and the text token sequence to obtain the target fusion sequence; Sequence processing module 105 is used to perform attention processing on the target fusion sequence through a large language model to obtain a target speech token sequence; The speech generation module 106 is used to generate speech from the target speech token sequence to obtain the target speech.

[0065] In one embodiment, the sequence processing module 105 is specifically used for: The target fusion sequence is subjected to a first attention process by the large language model to obtain a first voice token, and the first voice token is concatenated to the end of the target fusion sequence to obtain a first fusion sequence. The first fusion sequence is subjected to a second attention process through the large language model to obtain a second voice token, and the second voice token is concatenated to the end of the first fusion sequence to obtain a second fusion sequence. The second fusion sequence is subjected to a third attention process using the large language model to obtain a third speech token; The target voice token sequence is obtained by concatenating the first voice token, the second voice token, and the third voice token.

[0066] In one embodiment, the audio processing module 103 is specifically used for: Perform sentiment prediction on the reference audio to obtain a reference sentiment vector; The distance between the reference emotion vector and the preset neutral emotion center vector is calculated to obtain the emotion distance; Obtain at least two emotional threshold ranges corresponding to different preset emotional intensities; The target emotional threshold range is obtained by comparing the emotional distance with the emotional threshold range, and the preset emotional intensity corresponding to the target emotional threshold range is used as the target emotional intensity. The emotional intensity token is obtained by mapping the tokens based on the target emotional intensity.

[0067] In one embodiment, the reference emotion vector is located in a three-dimensional emotion space; before calculating the distance between the reference emotion vector and a preset neutral emotion center vector to obtain the emotion distance, the speech synthesis device further includes a neutral emotion determination module, specifically used for: Obtain a sample speech set, which includes sample speech, each of which has emotional three-dimensional coordinates in a three-dimensional emotional space, the emotional three-dimensional coordinates including emotional value, emotional activation degree and emotional dominance; The average emotional value is calculated based on the emotional value of each of the sample voices. The average emotional activation level is calculated based on the emotional activation level of each of the sample voices to obtain the average emotional activation level. The average emotional dominance is calculated based on the average emotional dominance of each of the sample voices; The neutral emotional center vector is obtained by merging the average emotional value, the average emotional activation, and the average emotional dominance.

[0068] In one embodiment, the at least two emotion threshold intervals include a first emotion threshold interval, a second emotion threshold interval, and a third emotion threshold interval with increasing emotion intensity; the audio processing module 103 is further specifically used for: The distance between the three-dimensional emotional coordinates of each sample speech and the neutral emotional center vector is calculated to obtain the candidate distance; Based on the candidate distance of each of the sample speech, the intervals are divided to obtain the first emotion threshold interval, the second emotion threshold interval, and the third emotion threshold interval; wherein, the upper limit of the first emotion threshold interval is less than the lower limit of the second emotion threshold interval, and the upper limit of the second emotion threshold interval is less than the lower limit of the third emotion threshold interval.

[0069] In one embodiment, the audio processing module 103 is further specifically used for: The candidate distances for each of the sample speech samples are sorted to obtain the minimum and maximum distances; The distance interval formed based on the minimum distance and the maximum distance is divided into three equal parts to obtain the first emotional threshold interval, the second emotional threshold interval, and the third emotional threshold interval.

[0070] In one embodiment, the reference text is the response text in the current round of a multi-turn dialogue; before updating the token of each of the target specified characters in the emphasis mask token sequence from 0 to 1 in response to detecting the presence of a target specified character in the reference text, the speech synthesis device further includes a character detection module, specifically used for: Obtain the dialogue text preceding the current round of a multi-round dialogue to get the historical dialogue text; The target character is obtained by performing key character detection on the historical dialogue text and the reference text using a large model.

[0071] This invention provides a speech synthesis device that utilizes an emphasis mask token sequence to highlight specific target characters in a reference text, achieving local emphasis control of the speech. Based on emotion intensity tokens obtained from the recognition of reference audio, it enables global emotion intensity control of the speech. Furthermore, it employs a large language model for attention processing to obtain a target speech token sequence, strictly controlling the emotion and intensity of the speech. Finally, it generates the target speech from the target speech token sequence. In summary, the target speech generated by this invention not only switches emotion categories when expressing emotions but also presents a continuous change from weak to strong, enhancing the layering and realism of the speech, resulting in better emotional expression.

[0072] For specific limitations regarding the speech synthesis device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0073] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech synthesis method on the server side.

[0074] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech synthesis method on the client side.

[0075] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the reference text, and serialize the reference text to obtain the text token sequence; Generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0; and in response to detecting the presence of a target specified character in the reference text, update the token of each target specified character in the emphasis mask token sequence from 0 to 1. Obtain reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token; The pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence are concatenated to obtain the target fusion sequence; Attention processing is applied to the target fusion sequence using a large language model to obtain the target speech token sequence; The target speech token sequence is used to generate speech, thus obtaining the target speech.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the reference text, and serialize the reference text to obtain the text token sequence; Generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0; and in response to detecting the presence of a target specified character in the reference text, update the token of each target specified character in the emphasis mask token sequence from 0 to 1. Obtain reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token; The pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence are concatenated to obtain the target fusion sequence; Attention processing is applied to the target fusion sequence using a large language model to obtain the target speech token sequence; The target speech token sequence is used to generate speech, thus obtaining the target speech.

[0077] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0080] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0081] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0082] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0083] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the reference text, and serialize the reference text to obtain the text token sequence; Generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0; and in response to detecting the presence of a target specified character in the reference text, update the token of each target specified character in the emphasis mask token sequence from 0 to 1. Obtain reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token; The pre-acquired emotion category token, emotion intensity token, emphasis mask token sequence, and text token sequence are concatenated to obtain the target fusion sequence; Attention processing is applied to the target fusion sequence using a large language model to obtain the target speech token sequence; The target speech token sequence is used to generate speech, thus obtaining the target speech.

2. The method according to claim 1, characterized in that, The step of performing attention processing on the target fusion sequence using a large language model to obtain a target speech token sequence includes: The target fusion sequence is subjected to a first attention process by the large language model to obtain a first voice token, and the first voice token is concatenated to the end of the target fusion sequence to obtain a first fusion sequence. The first fusion sequence is subjected to a second attention process through the large language model to obtain a second voice token, and the second voice token is concatenated to the end of the first fusion sequence to obtain a second fusion sequence. The second fusion sequence is subjected to a third attention process using the large language model to obtain a third speech token; The target voice token sequence is obtained by concatenating the first voice token, the second voice token, and the third voice token.

3. The method according to claim 1, characterized in that, The step of performing emotion intensity recognition on the reference audio to obtain an emotion intensity token includes: Perform sentiment prediction on the reference audio to obtain a reference sentiment vector; The distance between the reference emotion vector and the preset neutral emotion center vector is calculated to obtain the emotion distance; Obtain at least two emotional threshold ranges corresponding to different preset emotional intensities; The target emotional threshold range is obtained by comparing the emotional distance with the emotional threshold range, and the preset emotional intensity corresponding to the target emotional threshold range is used as the target emotional intensity. The emotional intensity token is obtained by mapping the tokens based on the target emotional intensity.

4. The method according to claim 3, characterized in that, The reference sentiment vector is located in a three-dimensional sentiment space; Before calculating the distance between the reference sentiment vector and the preset neutral sentiment center vector to obtain the sentiment distance, the method further includes: Obtain a sample speech set, which includes sample speech, each of which has emotional three-dimensional coordinates in a three-dimensional emotional space, the emotional three-dimensional coordinates including emotional value, emotional activation degree and emotional dominance; The average emotional value is calculated based on the emotional value of each of the sample voices. The average emotional activation level is calculated based on the emotional activation level of each of the sample voices to obtain the average emotional activation level. The average emotional dominance is calculated based on the average emotional dominance of each of the sample voices; The neutral emotional center vector is obtained by merging the average emotional value, the average emotional activation, and the average emotional dominance.

5. The method according to claim 4, characterized in that, At least two of the said emotional threshold intervals include a first emotional threshold interval, a second emotional threshold interval, and a third emotional threshold interval with increasing emotional intensity; The process of obtaining at least two emotional threshold intervals corresponding to different preset emotional intensities includes: The distance between the three-dimensional emotional coordinates of each sample speech and the neutral emotional center vector is calculated to obtain the candidate distance; Based on the candidate distance of each of the sample speech, the intervals are divided to obtain the first emotion threshold interval, the second emotion threshold interval, and the third emotion threshold interval; wherein, the upper limit of the first emotion threshold interval is less than the lower limit of the second emotion threshold interval, and the upper limit of the second emotion threshold interval is less than the lower limit of the third emotion threshold interval.

6. The method according to claim 5, characterized in that, The process of dividing the speech into intervals based on candidate distances of each of the sample voices to obtain the first emotion threshold interval, the second emotion threshold interval, and the third emotion threshold interval includes: The candidate distances for each of the sample speech samples are sorted to obtain the minimum and maximum distances; The distance interval formed based on the minimum distance and the maximum distance is divided into three equal parts to obtain the first emotional threshold interval, the second emotional threshold interval, and the third emotional threshold interval.

7. The method according to any one of claims 1 to 5, characterized in that, The reference text is the response text in the current round of a multi-round dialogue; Before updating the token of each target-specified character in the emphasis mask token sequence from 0 to 1 in response to detecting the presence of the target-specified character in the reference text, the method further includes: Obtain the dialogue text preceding the current round of a multi-round dialogue to get the historical dialogue text; The target character is obtained by performing key character detection on the historical dialogue text and the reference text using a large model.

8. A speech synthesis device, characterized in that, The device includes: The text processing module is used to acquire reference text, serialize the reference text, and obtain a text token sequence; The mask generation module is used to generate an emphasis mask token sequence with the same length as the reference text and all tokens being 0, and in response to detecting the presence of a target specified character in the reference text, to update the token of each target specified character in the emphasis mask token sequence from 0 to 1; The audio processing module is used to acquire reference audio, perform emotion intensity recognition on the reference audio, and obtain an emotion intensity token. The sequence generation module is used to concatenate the pre-acquired emotion category token, the emotion intensity token, the emphasis mask token sequence, and the text token sequence to obtain the target fusion sequence; The sequence processing module is used to perform attention processing on the target fusion sequence through a large language model to obtain a target speech token sequence; The speech generation module is used to generate speech from the target speech token sequence to obtain the target speech.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images