Speech synthesis method and device based on emotion control, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, the technical scheme can be applied to the field of financial science and technology and the field of medical health, and discloses a speech synthesis method and device based on emotion control, equipment and a medium, and the method comprises the steps: receiving text data inputted by a user and a selected target emotion tag; mapping the emotion label into a fixed-dimension emotion vector through a pre-trained emotion mapping model; extracting a semantic vector of the text by using a text encoder; performing feature fusion on the semantic vector and the emotion vector to generate a fusion vector containing emotion information; and based on the fusion vector, an audio feature sequence is generated through an acoustic model, and finally, a voice waveform bearing the target emotion is synthesized by a vocoder. According to the invention, by introducing an independent emotion coding and fusion mechanism, the user can directly and flexibly control the emotion type of the synthesized voice, the accuracy of emotion expression and the naturalness of the voice are ensured, and the emotion expressive force and affinity of human-computer interaction are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, fintech, and healthcare, and particularly to a speech synthesis method, apparatus, device, and medium based on emotion control. Background Technology

[0002] Speech synthesis technology aims to convert any text information into clear, fluent, and natural artificial speech in real time. As one of the core interfaces for human-computer interaction, it has been widely used in many fields such as intelligent voice assistants, audiobooks, in-vehicle navigation, intelligent customer service, and virtual digital humans. In recent years, its application scenarios have penetrated into fields with extremely high requirements for interaction quality, such as fintech (e.g., robo-advisors, telephone banking) and healthcare (e.g., medication reminders, mental health support). In recent years, with breakthroughs in deep learning technology, especially the emergence of TTS systems based on end-to-end modeling (e.g., Tacotron, WaveNet models), the naturalness and fluency of synthesized speech have been significantly improved, even reaching levels close to real human recordings in certain scenarios.

[0003] However, the allure of human voice communication lies not only in its semantic content but also in its rich emotional nuances and suprasegmental features (such as intonation, stress, rhythm, and timbre variation). These features are crucial for expressing the speaker's emotions, attitudes, and intentions. While current mainstream commercial TTS systems can generate high-quality speech, they suffer from significant limitations in emotional expression. These systems are typically optimized to output neutral, flat speech, lacking the necessary emotional expressiveness. This results in synthesized speech that sounds monotonous and mechanical, failing to provide a satisfactory user experience in applications requiring emotional resonance (such as storytelling, emotional customer service, and interactive entertainment). Particularly in the fintech sector, neutral speech struggles to convey a sense of urgency during risk warnings or sincerity and confidence during product recommendations. In healthcare, a lack of warmth in the voice fails to provide sufficient empathy and comfort to patients during psychological counseling or health follow-ups, limiting the technology's deeper application in these sensitive areas.

[0004] To overcome this limitation, emotional speech synthesis has become an important research direction in this field. Existing techniques attempt to inject emotion into synthesized speech by introducing emotion labels, style vectors, or reference audio. For example, some methods incorporate discrete emotion category labels (such as happy or sad) during model training, or introduce a continuous style vector in the latent space to control the style of the output speech. However, these methods still have many shortcomings: First, their control over emotion is often indirect and coarse, and users cannot precisely and flexibly specify the specific type and intensity of the desired emotion; second, due to the high complexity and subjectivity of emotional features, existing methods are prone to inaccurate emotional expression, mismatch with text semantics, and even decreased naturalness and distortion of speech when generating emotional speech; finally, different speakers have different emotional expression patterns, and existing models often have poor generalization ability, making it difficult to stably generate a variety of high-quality emotional speech while maintaining consistency in speaker timbre.

[0005] Therefore, there is an urgent need in this field for a new speech synthesis solution that enables users to intuitively and accurately control the emotional expression of synthesized speech, while ensuring high naturalness and sound quality stability of the generated speech, in order to meet the growing demand for highly expressive human-computer interaction, especially in professional fields such as finance and healthcare where there are stringent requirements for communication quality and emotional transmission. Summary of the Invention

[0006] This invention provides a speech synthesis method, apparatus, device, and medium based on emotion control, aiming to solve the problems of limited emotion expression and imprecise control in existing technologies.

[0007] Firstly, an emotion-controlled speech synthesis method is provided, comprising the following steps: Provide users with a set of emotion tags containing multiple basic emotion categories, and receive the target emotion tag selected by the user from the set of emotion tags; The pre-trained emotion mapping model is invoked to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. The text data input by the user is encoded using a text encoder to extract its deep semantic features and output as a text semantic vector. The text semantic vector and the emotion vector are fused at the feature level to construct a fused vector that contains both text semantic information and emotion style information. The fusion vector is input into the acoustic model, which decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism. The audio feature sequence is input into a vocoder, which reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0008] Secondly, an emotion-controlled speech synthesis device is provided, comprising: The emotion selection module provides users with a set of emotion tags containing multiple basic emotion categories, and receives the target emotion tag selected by the user from the set of emotion tags; The emotion encoding module calls a pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. The text encoding module uses a text encoder to encode the text data input by the user, extract its deep semantic features, and output a text semantic vector. The fusion module performs a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. An acoustic feature generation module inputs the fusion vector into an acoustic model, which then decodes and generates an audio feature sequence corresponding to the fusion vector; wherein the acoustic model is a sequence-to-sequence model based on an attention mechanism. The speech synthesis module inputs the audio feature sequence into the vocoder, which then reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described emotion-controlled speech synthesis method.

[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described emotion-controlled speech synthesis method.

[0011] The aforementioned speech synthesis method, device, equipment, and medium based on emotion control utilize an independent emotion encoding step to map the user-selected target emotion tag into a precise, machine-understandable emotion vector, providing a clear emotion control signal for the entire synthesis process. During emotion and semantic fusion, the emotion vector and the semantic vector of the text are deeply fused at the feature level, ensuring that emotional information is not simply superimposed but serves as an inherent control condition in the generation process, fundamentally guaranteeing a high degree of consistency between the semantic content and emotional tone of the synthesized speech. Through an acoustic model and vocoder, a speech waveform is generated that maintains the natural and fluent characteristics of the original speech while accurately carrying the target emotion. This technical solution completely changes the traditional speech synthesis system's singular and neutral emotional expression, enabling machine-generated speech to possess rich, accurate, and natural emotional expressiveness, much like human speech. This greatly enhances the intimacy and expressiveness of human-computer interaction, meeting the urgent needs of high-end application scenarios such as intelligent customer service, virtual humans, and audiobooks for emotional interaction. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a speech synthesis method based on emotion control in one embodiment of the present invention; Figure 2 This is a schematic diagram of an emotion selection process in one embodiment of the present invention; Figure 3 This is a schematic diagram of a text encoding process in one embodiment of the present invention; Figure 4 This is a schematic diagram of a process for fusing text and emotion vectors in one embodiment of the present invention; Figure 5 This is a flowchart illustrating an embodiment of the emotion consistency verification of the present invention; Figure 6 This is a schematic diagram of a strength encoding process according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a speech synthesis device based on emotion control in one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 As shown, Figure 1 A flowchart illustrating the emotion-controlled speech synthesis method provided in this embodiment of the invention includes the following steps: S10. Provide the user with a set of emotion tags containing multiple basic emotion categories, and receive the target emotion tag selected by the user from the set of emotion tags; Furthermore, such as Figure 2 As shown, in a specific embodiment, step S10 specifically includes: S11. Provide a graphical user interface, on which a variety of selectable emotion tags are clearly listed; S12. Receive the user's selection of a specific emotion tag by clicking or touching; S13. The emotion tag selected by the user is determined as the target emotion tag for this speech synthesis task.

[0016] Furthermore, the set of emotion tags includes multiple emotions such as happiness, sadness, anger, surprise, indifference, and fear.

[0017] In this embodiment, the specific implementation of the emotion tag set can be a drop-down menu containing icons and text descriptions, a set of radio buttons, or an emotion wheel interface. For example, on the agent's workbench of an intelligent customer service system, the interface provides emotion tags strongly related to the business scenario, such as "standard response," "enthusiastic recommendation," "patient explanation," and "emergency alarm," rather than just basic emotion categories. After exchanging text with the user, the agent can select the "enthusiastic recommendation" tag with one click based on the content of the conversation and the attitude they wish to convey. This design makes emotion control intuitive and efficient, directly converting the user's interaction intent into machine-recognizable control signals, ensuring the accuracy of emotion input from the source.

[0018] S20. Call the pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. Furthermore, such as Figure 3 As shown, in a specific embodiment, step S20 specifically includes: S21. Load the emotion mapping model that has been pre-trained on a large-scale emotional speech dataset; S22. The target emotion label is provided as input data to the emotion mapping model; S23. Perform forward computation through the emotion mapping model and output a corresponding, fixed-dimensional real-valued vector as the emotion vector.

[0019] In this embodiment, the emotion mapping model can be a simple embedding layer or a small fully connected neural network. The emotion mapping model is trained on a large speech dataset containing accurate emotion annotations. The learning objective is to maximize the cosine or Euclidean distance between the output vectors corresponding to different emotion categories (such as "happy" and "sad") in the vector space. For example, the "happy" label might be mapped to a 128-dimensional vector, whose numerical distribution characteristics differ significantly from the 128-dimensional vector mapped to the "sad" label. This step transforms abstract, discrete emotion labels into precise, continuous mathematical representations, providing stable and quantifiable control conditions for the subsequent fusion process, fundamentally avoiding the instability of the synthesis effect caused by the ambiguity of emotion definitions.

[0020] S30. Use a text encoder to encode the text data input by the user, extract its deep semantic features, and output it as a text semantic vector. In this embodiment, the text encoder can employ a pre-trained language model, such as the encoder portion of BERT or RoBERTa, or a custom text encoder based on Transformer. Its function is to convert the input text sequence (e.g., "This product is amazing!") into one or more fixed-dimensional vector representations, which capture sentence-level semantic information and grammatical structure. Compared to simply extracting word vectors, using a deep pre-trained model can better understand context and semantic weight; for example, it can recognize "amazing" as a strong positive sentiment keyword. This step ensures the accuracy of the semantic foundation of the synthesized speech, laying the groundwork for precise alignment of emotion and semantics.

[0021] S40. Perform a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. Furthermore, such as Figure 4 As shown, in one specific embodiment, step S40 specifically includes: S41. Using an attention mechanism, calculate the correlation weights between the emotion vector and different parts of the text semantic vector; S42. Based on the relevance weights, the text semantic vector is weighted and modulated to enhance the parts of the text related to emotional expression. S43. Combine the modulated text semantic vector with the emotion vector to form a fusion vector that contains both text semantic information and emotional style information.

[0022] In this embodiment, the specific implementation of the attention mechanism is as follows: The emotion vector is used as the Query, and all character or word-level vector sequences output by the text encoder are used as Keys and Values. A set of attention weights is obtained by calculating the similarity between the Query and each Key. This set of weights reveals which parts of the text (such as sentiment words, interjections, and interrogative words) are most relevant to the expression of the target emotion. Then, these weights are used to perform a weighted summation of the Values ​​(i.e., the text vector sequence) to obtain an emotion-modulated text representation. Finally, this representation is concatenated with the original emotion vector. This scheme achieves selective emphasis of emotion on the text, rather than simply and crudely covering the entire text. For example, when synthesizing speech expressing sadness, the model automatically assigns higher weights to negative words in the text, thus giving a stronger expression in terms of rhythm (such as tone and pauses), making the generated emotional speech not only emotionally accurate but also highly consistent with the text content, significantly improving its naturalness.

[0023] S50. The fusion vector is input into the acoustic model, and the acoustic model decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism; In this embodiment, the acoustic model can, for example, adopt an architecture similar to Tacotron 2. The acoustic model takes the fusion vector output in step S40 as conditional input and, through its encoder-decoder structure, autoregressively generates a series of Mel spectrogram frames. Since the fusion vector contains explicit emotional information, the acoustic model is guided by this emotion when generating audio features for each frame. For example, when the emotion vector indicates "anger," the acoustic model tends to generate Mel spectrograms with a higher fundamental frequency, stronger energy, and faster speech rate; while when the emotion is "sadness," it tends to generate Mel spectrograms with a lower fundamental frequency, weaker energy, and slower speech rate. This scheme transforms the fusion information from the previous step into specific acoustic parameters that can be understood by the vocoder, which is key to achieving emotional speech synthesis.

[0024] S60. The audio feature sequence is input into a vocoder, and the vocoder reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0025] In this embodiment, the vocoder can be a neural network vocoder such as WaveNet, WaveGlow, or HiFi-GAN. The vocoder receives the Mel spectrum sequence generated by the acoustic model and converts it into raw waveform sample points in the time domain. A high-quality vocoder can well preserve the emotion-related acoustic features contained in the Mel spectrum, ultimately outputting a high-fidelity, natural, and fluent speech waveform. This solution completes the conversion from parameters to final speech, which is the final step in the final presentation of emotional expression, and its quality directly determines the user experience.

[0026] Furthermore, such as Figure 5 As shown, in one specific embodiment, the speech synthesis method based on emotion control further includes an emotion consistency verification step, specifically including: S71. Input the synthesized speech waveform signal into a pre-trained emotion recognition model; S72. Extract the actual emotional features contained in the speech waveform signal through the emotion recognition model; S73. Compare the identified emotional features with the target emotional label initially selected by the user; S74. If the inconsistency of the comparison results is higher than the preset threshold, then trigger re-synthesis or send a prompt message to the user.

[0027] In this embodiment, the emotion consistency verification step serves as an optional quality assurance closed loop. The pre-trained emotion recognition model can be an audio classification model based on CNN or Transformer. For example, after synthesizing a speech with the target emotion of "happy," it is immediately detected using the emotion recognition model. If the model identifies "neutral" or "sad" with the highest confidence level, and the confidence difference exceeds a threshold (e.g., 0.5), the system can automatically resynthesize using a slightly adjusted emotion vector strength, or notify the administrator for manual review. This mechanism effectively improves the reliability and consistency of the system output, especially in demanding commercial scenarios, preventing unacceptable speech from occasional model malfunctions and ensuring service quality.

[0028] Furthermore, such as Figure 6 As shown, in one specific embodiment, the speech synthesis method based on emotion control further includes: S111, Receive the user's specification of the intensity of emotional expression; S211, Intensity Coding Steps: Map the user-specified emotional intensity level to an intensity control coefficient; S311, Intensity fusion step: The intensity control coefficient is applied to the emotion vector to adjust the influence of the emotion vector on the final synthesized speech, so that the generated speech expresses the target emotion and conforms to the specified intensity level.

[0029] In this embodiment, the user can specify the emotional intensity using a slider or a level selector (e.g., levels 1-5). The intensity encoding step can linearly or non-linearly map level i (e.g., 1-5) to a coefficient s (e.g., 0.5-2.0). In the fusion step S40, the emotional vector E is multiplied by the coefficient s to obtain the modulated emotional vector E' = s * E, which is then fused with the text semantic vector. When s > 1, the emotional expression is enhanced; when s < 1, the emotional expression is weakened. This provides a finer control granularity, allowing the same text to synthesize speech with different levels of emotion, ranging from "slightly joyful" to "ecstatic," greatly expanding the applicable scenarios and expressiveness of this solution.

[0030] In one specific embodiment, the present invention can be deeply applied to intelligent risk control and customer service scenarios. When a credit card anti-fraud system detects a high-risk transaction from another location, it needs to automatically call the cardholder for verification. Traditional TTS (Text-to-Speech) alerts are delivered in a neutral tone, which may not attract enough attention from users.

[0031] When applying this solution, the system's business logic will automatically determine that the scenario requires a mixed emotional label of "high vigilance" and "seriousness" and set a high intensity level (e.g., level 4, up to level 5).

[0032] The text information to be verified (such as "Dear customer, we have detected an abnormal transaction on your card ending in XXXX. Did you make this transaction?") will be fused with the aforementioned emotional vector. The synthesized speech will no longer be a flat reading, but will exhibit acoustic characteristics such as a slightly faster speaking speed, higher pitch, and emphasis on keywords (such as "abnormal transaction"). The overall sound will be serious and urgent, effectively arousing the listener's attention and vigilance.

[0033] These urgency-filled voice prompts significantly improve the effectiveness of the identity verification process. They are more likely than neutral voice prompts to prompt cardholders to take the process seriously and respond immediately, thereby preventing potential fraud losses at the first moment and improving the effectiveness of financial security services.

[0034] Furthermore, in robo-advisor product recommendation scenarios, for conservative financial products, a "calm and trustworthy" mood can be specified; for high-growth products, a "optimistic and positive" mood can be specified. This makes the machine-provided financial advice sound more professional and persuasive, conveying understanding and confidence in product characteristics through voice, thereby enhancing user trust and improving marketing conversion rates.

[0035] In one specific embodiment, the present invention can be applied to chronic disease management follow-up and mental health intervention scenarios. For diabetic patients who need long-term medication, the system needs to regularly remind them of medication use and inquire about their lifestyle.

[0036] When using this solution, the system can automatically select the appropriate emotion based on the patient's recent blood glucose data. If the data is stable, an encouraging and affirming emotion will be used (e.g., "Your blood glucose control has been very good recently, please keep it up!"). If the data is abnormal, a caring and reminding emotion will be used instead of a blaming emotion (e.g., "We noticed that your blood glucose levels have been a bit high in the last two days. Perhaps you should pay attention to your diet?").

[0037] By fusing emotion vectors with text, the synthesized speech will exhibit a warm tone and gentle rhythm when expressing "encouragement," and a moderate speaking speed, soft tone, and patience when expressing "concern."

[0038] Through emotionally engaging voice interaction, compliance and user experience during medical follow-ups are greatly improved. Patients no longer experience cold, impersonal machine commands, but rather empathetic and caring reminders, which helps establish a positive doctor-patient interaction model, especially effective for elderly patients or those requiring psychological support.

[0039] In mental health support scenarios, such as guiding users through relaxation exercises, the system can use "calm" and "soothing" emotion tags to synthesize gentle and calm guiding language, like a gentle breeze. In cognitive behavioral therapy, when correcting negative cognitions, it can use "firm" and "supportive" emotions. This precise emotion control capability enables digital health interventions to deliver more accurate psychological support signals, improving the quality and humanization of telemedicine services.

[0040] The technical solution proposed in this invention achieves several significant technical effects by constructing an end-to-end controllable generation link from emotional intention input to emotional speech output. Its core effect lies in fundamentally improving the emotional expression capability and controllability of the speech synthesis system, breaking the limitation of traditional systems that can only generate neutral speech.

[0041] Specifically, the solution introduces an independent emotion encoding and fusion mechanism to transform user-specified discrete emotion tags into precise, quantifiable emotion vectors, and then utilizes deep learning methods such as attention mechanisms to deeply fuse these vectors with the text semantics. This solution ensures that emotional information is not simply post-processed or superimposed, but rather serves as an intrinsic, global control condition in the speech generation process, thereby enabling precise regulation at the source of acoustic feature generation. This results in synthesized speech that not only maintains high naturalness and clarity but also achieves a high degree of unity and natural fit between semantic content and emotional tone.

[0042] Furthermore, the solution supports fine-grained and separate control over emotion type and intensity, allowing users to freely combine emotions and intensities like adjusting a color palette to generate emotional voices at different levels, from "slight annoyance" to "rage," greatly expanding the system's performance range and applicable scenarios. Optional emotion consistency verification steps further constitute a quality assurance closed loop, improving the reliability and stability of the system's output.

[0043] In summary, this technical solution enables machine-generated speech to possess rich, accurate, and natural emotional expressiveness close to human level, greatly enhancing the intimacy, persuasiveness, and communication efficiency of human-computer interaction. It provides strong technical support for application scenarios requiring highly expressive speech, such as intelligent customer service, virtual humans, and audiobooks, and has significant industrial application value.

[0044] like Figure 7 As shown, this embodiment of the invention also provides a speech synthesis device based on emotion control, comprising: The emotion selection module 10 provides the user with a set of emotion tags containing multiple basic emotion categories, and receives the target emotion tag selected by the user from the set of emotion tags; Furthermore, in one specific embodiment, the emotion selection module 10 is specifically used for: It provides a graphical user interface that clearly lists a variety of emotion tags to choose from; It can receive user selections of specific emotion tags via clicks or touches; The emotion tag selected by the user is determined as the target emotion tag for this speech synthesis task.

[0045] Furthermore, the set of emotion tags includes multiple emotions such as happiness, sadness, anger, surprise, indifference, and fear.

[0046] The emotion encoding module 20 calls a pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. Furthermore, in one specific embodiment, the emotion encoding module 20 is specifically used for: Load the emotion mapping model that has been pre-trained on a large-scale emotional speech dataset; The target emotion label is provided as input data to the emotion mapping model; The emotion mapping model performs forward computation and outputs a corresponding, fixed-dimensional real-valued vector as the emotion vector.

[0047] The text encoding module 30 uses a text encoder to encode the text data input by the user, extract its deep semantic features, and output a text semantic vector. The fusion module 40 performs a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. Furthermore, in one specific embodiment, the fusion module 40 is specifically used for: An attention mechanism is used to calculate the correlation weights between different parts of the emotion vector and the text semantic vector; Based on the relevance weights, the text semantic vector is weighted and modulated to enhance the parts of the text related to emotional expression. The modulated text semantic vector and the emotion vector are combined to form a fused vector that contains both text semantic information and emotional style information.

[0048] The acoustic feature generation module 50 inputs the fusion vector into the acoustic model, and the acoustic model decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism; The speech synthesis module 60 inputs the audio feature sequence into the vocoder, which reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0049] Furthermore, in one specific embodiment, the emotion-controlled speech synthesis device further includes an emotion consistency verification module, specifically used for: The synthesized speech waveform signal is input into a pre-trained emotion recognition model; The emotion recognition model is used to extract the actual emotion features contained in the speech waveform signal; The identified emotional features are compared with the target emotional label initially selected by the user; If the inconsistency of the comparison results exceeds a preset threshold, a re-synthesis will be triggered or a prompt message will be sent to the user.

[0050] Furthermore, in one specific embodiment, the emotion-controlled speech synthesis device further includes: an emotion intensity configuration module, specifically used for: Receive user specifications regarding the intensity of emotional expression; Map the user-specified level of emotional intensity to an intensity control coefficient; The intensity control coefficient is applied to the emotion vector to adjust the influence of the emotion vector on the final synthesized speech, so that the generated speech expresses the target emotion and conforms to the specified intensity level.

[0051] For specific limitations regarding emotion-controlled speech synthesis devices, please refer to the limitations of emotion-controlled speech synthesis methods mentioned above, which will not be repeated here. Each module in the aforementioned emotion-controlled speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0052] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side emotion-controlled speech synthesis method.

[0053] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client-side method for emotion-based speech synthesis.

[0054] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: S10. Provide the user with a set of emotion tags containing multiple basic emotion categories, and receive the target emotion tag selected by the user from the set of emotion tags; S20. Call the pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. S30. Use a text encoder to encode the text data input by the user, extract its deep semantic features, and output it as a text semantic vector. S40. Perform a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. S50. The fusion vector is input into the acoustic model, and the acoustic model decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism; S60. The audio feature sequence is input into a vocoder, and the vocoder reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S10. Provide the user with a set of emotion tags containing multiple basic emotion categories, and receive the target emotion tag selected by the user from the set of emotion tags; S20. Call the pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. S30. Use a text encoder to encode the text data input by the user, extract its deep semantic features, and output it as a text semantic vector. S40. Perform a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. S50. The fusion vector is input into the acoustic model, and the acoustic model decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism; S60. The audio feature sequence is input into a vocoder, and the vocoder reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

[0056] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0057] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0058] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0059] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on emotion control, characterized in that, Includes the following steps: Provide users with a set of emotion tags containing multiple basic emotion categories, and receive the target emotion tag selected by the user from the set of emotion tags; The pre-trained emotion mapping model is invoked to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. The text data input by the user is encoded using a text encoder to extract its deep semantic features and output as a text semantic vector. The text semantic vector and the emotion vector are fused at the feature level to construct a fused vector that contains both text semantic information and emotion style information. The fusion vector is input into the acoustic model, which decodes and generates an audio feature sequence corresponding to the fusion vector; wherein, the acoustic model is a sequence-to-sequence model based on an attention mechanism. The audio feature sequence is input into a vocoder, which reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

2. The speech synthesis method based on emotion control according to claim 1, characterized in that, The step of fusing the text semantic vector and the emotion vector at the feature level to construct a fused vector that contains both text semantic information and emotional style information specifically includes: An attention mechanism is used to calculate the correlation weights between different parts of the emotion vector and the text semantic vector; Based on the relevance weights, the text semantic vector is weighted and modulated to enhance the parts of the text related to emotional expression. The modulated text semantic vector and the emotion vector are combined to form a fused vector that contains both text semantic information and emotional style information.

3. The speech synthesis method based on emotion control according to claim 1, characterized in that, It also includes an emotion consistency verification step, specifically including: The synthesized speech waveform signal is input into a pre-trained emotion recognition model; The emotion recognition model is used to extract the actual emotion features contained in the speech waveform signal; The identified emotional features are compared with the target emotional label initially selected by the user; If the inconsistency of the comparison results exceeds a preset threshold, a re-synthesis will be triggered or a prompt message will be sent to the user.

4. The speech synthesis method based on emotion control according to claim 1, characterized in that, Also includes: Receive user specifications regarding the intensity of emotional expression; Map the user-specified level of emotional intensity to an intensity control coefficient; The intensity control coefficient is applied to the emotion vector to adjust the influence of the emotion vector on the final synthesized speech, so that the generated speech expresses the target emotion and conforms to the specified intensity level.

5. The speech synthesis method based on emotion control according to claim 1, characterized in that, The step of calling the pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space specifically includes: Load the emotion mapping model that has been pre-trained on a large-scale emotional speech dataset; The target emotion label is provided as input data to the emotion mapping model; The emotion mapping model performs forward computation and outputs a corresponding, fixed-dimensional real-valued vector as the emotion vector.

6. The speech synthesis method based on emotion control according to claim 1, characterized in that, The provision of a set of emotion tags containing multiple basic emotion categories to the user, and the receipt of a target emotion tag selected by the user from the set of emotion tags, specifically includes: It provides a graphical user interface that clearly lists a variety of emotion tags to choose from; It can receive user selections of specific emotion tags via clicks or touches; The emotion tag selected by the user is determined as the target emotion tag for this speech synthesis task.

7. The speech synthesis method based on emotion control according to any one of claims 1-6, characterized in that, The set of emotion tags includes multiple emotions such as happiness, sadness, anger, surprise, indifference, and fear.

8. A speech synthesis device based on emotion control, characterized in that, include: The emotion selection module provides users with a set of emotion tags containing multiple basic emotion categories, and receives the target emotion tag selected by the user from the set of emotion tags; The emotion encoding module calls a pre-trained emotion mapping model to map the target emotion label into a fixed-dimensional emotion vector with high discriminative power in the vector space. The text encoding module uses a text encoder to encode the text data input by the user, extract its deep semantic features, and output a text semantic vector. The fusion module performs a feature-level fusion operation on the text semantic vector and the emotion vector to construct a fusion vector that contains both text semantic information and emotion style information. An acoustic feature generation module inputs the fusion vector into an acoustic model, which then decodes and generates an audio feature sequence corresponding to the fusion vector; wherein the acoustic model is a sequence-to-sequence model based on an attention mechanism. The speech synthesis module inputs the audio feature sequence into the vocoder, which then reconstructs the corresponding speech waveform signal carrying the emotion specified by the target emotion tag.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the emotion-controlled speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the emotion-controlled speech synthesis method as described in any one of claims 1 to 7.

Images