Speech synthesis method and device, computer device and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device, computer equipment and a storage medium, which comprise the following steps: obtaining reference speech data and speech synthesis text; performing emotion extraction on the reference speech data to obtain reference emotion embedding; fusing the reference speech data, the speech synthesis text and preset noise features to obtain original fusion features; determining a dynamic emotion injection coefficient based on a preset denoising time step parameter; performing denoising processing on the original fusion features based on the dynamic emotion injection coefficient and the reference emotion embedding to obtain target synthesis spectral features; and performing audio conversion based on the target synthesis spectral features to obtain target speech data. The application can be applied to the speech synthesis scene of financial technology and medical health, and improves the emotion expression accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, and particularly to a speech synthesis method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Speech synthesis technology can be used to convert input text into speech with the emotional style of a reference audio. Speech synthesis technology can be applied in various scenarios, such as in the financial sector, for synthesizing speech in intelligent customer service, virtual voice assistants, operational process guidance, and the introduction of financial products and knowledge; and in the healthcare sector, for synthesizing speech in telemedicine, medical advice reminders, disease introductions, and rehabilitation guidance.

[0003] Currently, diffusion transformers can be used as speech synthesis models. In practical applications, audio noise, text conditions, and reference audio conditions are typically concatenated, and the concatenated features are then input into the speech synthesis model. However, this simple and direct concatenation method dilutes the emotional information in the reference audio, making it difficult to achieve precise emotional control during speech synthesis, thus affecting the accuracy of emotional expression in speech synthesis. Summary of the Invention

[0004] This application provides a speech synthesis method, apparatus, computer device, and storage medium to solve the technical problem that the emotional information is diluted due to the direct splicing of audio noise, text conditions, and reference audio conditions, making it difficult to achieve accurate emotional control during speech synthesis and affecting the accuracy of emotional expression in speech synthesis.

[0005] Firstly, a speech synthesis method is provided, including: Obtain reference speech data and speech-synthesized text; Emotion extraction is performed on the reference speech data to obtain the reference emotion embedding; The reference speech data, the synthesized speech text, and preset noise features are fused to obtain the original fused features; The dynamic emotion injection coefficient is determined based on the preset denoising time step parameters; Based on the dynamic emotion injection coefficient and the reference emotion embedding, the original fused features are denoised to obtain the target synthesized spectral features; Audio conversion is performed based on the target synthesized spectral features to obtain target speech data.

[0006] Secondly, a speech synthesis device is provided, comprising: The data acquisition module is used to acquire reference speech data and speech-synthesized text; The emotion extraction module is used to extract emotions from the reference speech data to obtain a reference emotion embedding; The feature fusion module is used to fuse the reference speech data, the speech-synthesized text, and preset noise features to obtain the original fused features; The injection coefficient determination module is used to determine the dynamic emotion injection coefficient based on preset denoising time step parameters; The denoising module is used to denoise the original fused features based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain the target synthesized spectral features; The audio conversion module is used to perform audio conversion based on the target synthesized spectral features to obtain target speech data.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech synthesis method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described speech synthesis method.

[0009] The aforementioned speech synthesis method, apparatus, computer equipment, and storage medium, targeting speech synthesis scenarios in fintech and healthcare, firstly acquires reference speech data and speech synthesis text; then, it extracts emotion from the reference speech data to accurately capture emotional information, obtaining a reference emotion embedding. Next, it fuses the reference speech data, speech synthesis text, and preset noise features to obtain original fused features. Then, based on preset denoising time step parameters, it determines dynamic emotion injection coefficients, allowing for flexible adjustment of the emotion injection level according to the speech synthesis process. Further, it denoises the original fused features based on the dynamic emotion injection coefficients and reference emotion embedding to obtain target synthesis spectral features, effectively removing noise interference while appropriately injecting emotion, resulting in clearer and more accurate target synthesis spectral features. Finally, it performs audio conversion based on the target synthesis spectral features to generate high-quality and emotionally rich target speech data. This addresses the technical problem of diluted emotional information caused by directly splicing audio noise, text conditions, and reference audio conditions, which hinders precise emotion control during speech synthesis and affects the accuracy of emotional expression in speech synthesis. This method improves the accuracy and naturalness of emotional expression in speech synthesis. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a speech synthesis method in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S50; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation of step S51; Figure 5 yes Figure 3 A schematic diagram of a specific implementation method for step S53; Figure 6 This is a schematic diagram of a speech synthesis device in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device according to one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0014] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0015] Speech synthesis technology, originally used to convert input text into speech with the emotional style of a reference audio, combines text information with the features of a reference audio. First, it performs linguistic analysis on the text, extracting prosody, semantics, and other content. Then, it extracts personalized features from the reference audio, such as timbre, pronunciation style, and emotional expression, including pitch, duration, and intonation patterns. Finally, through deep learning and other algorithmic models, it presents the text content in the unique style of the reference audio, ultimately generating natural, fluent, expressive speech that closely resembles the style of the reference audio. This technology has wide applications in fields such as virtual broadcasters and audiobook production.

[0016] The Diffusion Transformer (DiT) is a generative model that combines a diffusion model with the Transformer architecture, and can be used for speech synthesis. The core mechanism of the Diffusion Transformer is achieved through a progressive denoising process: starting from random noise, the model learns the temporal dependencies and acoustic feature distribution of the speech signal step by step under the guidance of the Transformer's self-attention mechanism, ultimately generating high-quality Mel-spectrum or audio waveforms. Compared to traditional U-Net-based diffusion models, DiT leverages the Transformer's long-range modeling capabilities to more efficiently capture details such as prosody and intonation in speech. Simultaneously, it accelerates the generation process through parallel inference, demonstrating advantages in naturalness and robustness in scenarios such as zero-shot speech cloning and multilingual synthesis.

[0017] Speech synthesis technology can be applied to various scenarios. In the financial sector, it can be used to synthesize speech for intelligent customer service, virtual voice assistants, operational process guidance, and the introduction of financial products and knowledge. In the healthcare sector, it can be used to synthesize speech for telemedicine, medical advice reminders, disease introductions, and rehabilitation guidance.

[0018] Currently, diffusion transformers can be used as speech synthesis models. In practical applications, audio noise, text conditions, and reference audio conditions are typically concatenated, and the concatenated features are then input into the speech synthesis model. However, this simple and direct concatenation method dilutes the emotional information in the reference audio, making it difficult to achieve precise emotional control during speech synthesis, thus affecting the accuracy of emotional expression in speech synthesis.

[0019] Furthermore, the generation process of the diffusion transformer is divided into a preliminary "contour building" phase and a later "detail filling" phase. If the same intensity of emotional conditions is applied throughout the entire process, emotional features often interfere with the restoration of acoustic details in the later stages of generation, leading to increased background noise, decreased sound quality, or timbre distortion in the synthesized speech.

[0020] Based on this, embodiments of this application provide a speech synthesis method, apparatus, computer device, and storage medium to solve the technical problem that the emotional information is diluted due to the direct splicing of audio noise, text conditions, and reference audio conditions, making it difficult to achieve accurate emotional control during speech synthesis and affecting the accuracy of emotional expression in speech synthesis.

[0021] The speech synthesis method, apparatus, computer equipment, and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0022] The speech synthesis method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can obtain reference speech data and synthesized speech text from the client; extract emotion from the reference speech data to obtain a reference emotion embedding; fuse the reference speech data, synthesized speech text, and preset noise features to obtain the original fused features; determine the dynamic emotion injection coefficients based on preset denoising time step parameters; denoise the original fused features based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain the target synthesized spectral features; perform audio conversion based on the target synthesized spectral features to obtain the target speech data, and feed the target speech data back to the client.

[0023] In this application, targeting speech synthesis scenarios in fintech and healthcare, the following steps are taken: First, reference speech data and speech synthesis text are acquired. Emotion extraction is performed on the reference speech data to accurately capture emotional information, resulting in a reference emotion embedding. The reference speech data, speech synthesis text, and preset noise features are then fused to obtain the original fused features. Next, dynamic emotion injection coefficients are determined based on preset denoising time step parameters, allowing for flexible adjustment of the emotion injection level according to the speech synthesis process. Furthermore, the original fused features are denoised based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain target synthesis spectral features. This effectively removes noise interference while appropriately injecting emotion, resulting in clearer and more accurate target synthesis spectral features. Finally, audio conversion is performed based on the target synthesis spectral features to generate high-quality and emotionally rich target speech data. This addresses the technical problem of diluted emotional information caused by directly splicing audio noise, text conditions, and reference audio conditions, which hinders precise emotion control during speech synthesis and affects the accuracy of emotional expression. This application improves the accuracy and naturalness of emotional expression in speech synthesis.

[0024] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description uses specific embodiments to illustrate this application.

[0025] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech synthesis method provided in this application embodiment includes the following steps: S10. Obtain reference speech data and speech-synthesized text; Reference speech data refers to existing speech segments that serve as a reference for speech synthesis features such as emotion, timbre, and intonation. In fintech scenarios, reference speech data could be the voice of a professional financial advisor explaining investment strategies; in healthcare scenarios, it could be the voice of a doctor patiently answering a patient's questions.

[0026] Speech-synthesized text is text content that needs to be converted into speech using speech synthesis technology. In fintech scenarios, speech-synthesized text can be investment reports, market analyses, product introductions, and other texts; in healthcare scenarios, it can be medical records, doctor's orders, health education articles, and other texts.

[0027] In some embodiments, the reference speech data and the speech-synthesized text may be input by the user in the client as input data for speech synthesis.

[0028] S20. Perform emotion extraction on the reference speech data to obtain the reference emotion embedding; It's important to understand that sentiment extraction here refers to analyzing reference speech data, extracting its emotional features, and converting them into embedding vectors. Specifically, step S20, which involves extracting sentiment from the reference speech data to obtain the reference sentiment embedding, includes the following steps: Emotional features are extracted from the reference speech data to obtain the original emotional embedding; The original sentiment embedding is transformed to obtain a reference sentiment embedding.

[0029] First, emotional features are extracted from the reference speech data to obtain the original emotional embedding, which can capture key emotional information in the speech. Then, the original emotional embedding is transformed to obtain the reference emotional embedding, which can adjust the data dimensions to better adapt to subsequent model processing and improve the accuracy and efficiency of sentiment analysis.

[0030] Specifically, pre-trained speech emotion recognition models (such as recurrent neural networks and convolutional neural network models) can be used to mine feature information that reflects emotional state from reference speech data, such as changes in tone, speech rate, volume, and timbre. The feature information can be converted into a computable vector form, and the original emotion embedding can be accurately extracted. This can preserve the rich emotional information in the reference speech data and provide a real and reliable emotional basis for subsequent emotion analysis and speech synthesis.

[0031] Next, a convolutional layer is introduced to adjust the dimension of the original sentiment embedding, mapping the original sentiment embedding to the hidden layer dimension of the diffusion transformer to obtain the reference sentiment embedding, so as to better adapt to different application scenarios and model requirements.

[0032] For example, suppose the original sentiment embedding has a dimension of B×N×2 (where B is the batch size, which represents the number of speech segments in a batch during sentiment feature extraction; N is the number of frames; and 2 represents the dimension of the sentiment embedding), and the convolutional layer is a 1×1 convolutional layer. The original sentiment embedding is mapped to the hidden layer dimension D of the diffusion transform through the convolutional layer to obtain the reference sentiment embedding with a dimension of B×N×D.

[0033] S30. The reference speech data, the synthesized speech text, and the preset noise features are fused to obtain the original fused features; It's important to understand that the fusion here refers to merging the feature vectors of the reference speech data, the feature vectors of the synthesized speech text, and preset noise features to obtain the original fused feature vector. This allows the synthesized speech to retain the style of the reference speech while accurately representing the text content, while the introduction of noise features increases the realism of the speech, making it closer to speech in real-world scenarios. Specifically, step S30, which involves fusing the reference speech data, the synthesized speech text, and the preset noise features to obtain the original fused feature vector, includes the following steps: Spectral transformation is performed on the reference speech data to obtain the reference spectral features; Text encoding is performed on the synthesized speech text to obtain text encoding features; The reference spectral features, text encoding features, and noise features are concatenated to obtain the original fused features.

[0034] Specifically, reference speech data can be processed using specific algorithms (such as Fourier transform) to convert the speech signal from the time domain to the frequency domain. This can accurately capture the frequency characteristics of the reference speech, preserve key information such as timbre and intonation, and obtain reference spectral features. These reference spectral features contain speech frequency information and reflect the energy distribution and other characteristics of the reference speech data at different frequencies.

[0035] Next, natural language processing techniques (such as word embedding) can be used to map each word or character in the synthesized text into a vector of a specific dimension, thereby converting the entire text into a vector sequence, i.e., text encoding features. Text encoding features are numerical vectors obtained after text encoding, containing semantic, syntactic, and other information about the text.

[0036] It should be noted that noise features are pre-defined to simulate noise in real-world environments, such as background noise and equipment noise, in order to increase the diversity and realism of speech synthesis. In essence, generative models (such as diffusion transformers) aim to create something from nothing; simply put, it's the process of generating target speech from pure noise. Therefore, the model's input needs to be noise, and the target is the corresponding real speech.

[0037] Finally, the reference spectral features, text encoding features, and noise features are input into the embedding layer. The embedding layer processes each of these features separately. For each feature vector, the embedding layer performs a linear or nonlinear transformation based on its internal parameters, mapping it to a suitable vector space. These transformed feature vectors are then concatenated sequentially to form a new, original fused feature vector containing multiple information elements. This original fused feature vector fully utilizes information from speech, text, and noise. In speech synthesis, the spectral features of speech contain important information such as pitch, timbre, and rhythm; text encoding features provide content information; and noise features can simulate various interferences in the real environment, making the synthesized speech more natural and realistic.

[0038] It should be noted that the dimensions of the original fusion features are the same as those of the reference sentiment embedding, both being B×N×D.

[0039] S40. Determine the dynamic emotion injection coefficient based on the preset denoising time step parameters; It should be noted that the preset denoising time step parameters are parameters used by the diffusion transformer to control the time step size during speech synthesis. These parameters are typically determined based on the specific needs and scenarios of speech synthesis, such as setting them to 100 steps, where each step constitutes one denoising time step. The preset denoising time step parameters affect the progress and effectiveness of denoising.

[0040] Next, the following formula (1) is introduced to determine the dynamic emotion injection coefficient based on the denoising time step parameter: (1); in, Inject coefficients into dynamic emotions; The denoising time step parameter represents the current denoising step number in the diffusion model; For emotion enhancement threshold, a preset time step threshold is set when... Enhance emotional attention when Reduce emotional attention; For high gain coefficient, when time step When using Calculations can be performed to enhance emotions; For low gain coefficient, time step At that time, the dynamic emotion injection coefficient was This can achieve emotional weakening.

[0041] It should be noted that the dynamic emotion injection coefficient It is a time step that varies with the denoising time. The varying scalar coefficients are used to adjust the strength of the reference emotion embedding as a condition in the speech synthesis process.

[0042] In the early stages of speech synthesis ( This mainly involves constructing the macroscopic structure of speech, including pitch contours, rhythmic patterns, and emotional coloring. At this point... By forcibly amplifying the attention score of the emotion embedding, the model "prioritizes" emotion features in the early stages of generation, thereby shaping a full and accurate emotional rhythm. In the later stages of speech synthesis (… This focuses on the restoration of microscopic acoustic details, such as harmonic structure, fine spectral texture, and noise floor elimination. Actively suppressing the attention of emotional embedding avoids excessive interference of emotional features with detail generation, preventing increased background noise, decreased sound quality, or timbre distortion. The final synthesized speech is both emotionally rich and naturally faithful.

[0043] In some embodiments, denoising time step parameters are assumed. Setting it to 100 steps sets the emotional enhancement threshold. It can be set to 50 for a high gain coefficient. It can be set to 3, low gain coefficient It can be set to 0.5.

[0044] S50. Based on the dynamic emotion injection coefficient and the reference emotion embedding, the original fused features are denoised to obtain the target synthesized spectral features; It is important to understand that the denoising process here refers to the process of using a diffusion transformer to denoise the original fused features based on the denoising time step parameter control. During the denoising process, the contribution strength of the reference emotion embedding is adjusted by the dynamic emotion injection coefficient, conditioned on the reference emotion embedding, so that the synthesized speech is clear and emotional.

[0045] Among them, such as Figure 3 As shown, step S50, which involves denoising the original fused features based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain the target synthesized spectral features, includes the following steps: S51. Based on the original fusion features and the reference sentiment embedding, cross-attention calculation is performed to obtain the original fusion feature attention score and the original sentiment attention score. It is important to understand that the cross-attention calculation here refers to performing cross-attention calculation on the original fused features and the reference sentiment embedding. This can clarify the relationship between the original fused features and the reference sentiment embedding, allowing subsequent processing to more effectively adjust the original fused features in conjunction with the reference sentiment embedding.

[0046] Among them, such as Figure 4 As shown, step S51, which involves cross-attention calculation based on the original fusion features and the reference sentiment embedding to obtain the original fusion feature attention score and the original sentiment attention score, includes the following steps: S511. Based on the preset query projection matrix and the original fusion features, perform aggregation calculations to obtain the query matrix; S512. Based on the preset key projection matrix and the original fusion features, perform aggregation calculation to obtain the fusion key matrix; S513. Based on the key projection matrix and the reference sentiment embedding, perform aggregation calculation to obtain the sentiment key matrix; S514. Perform cross-attention calculation based on the query matrix and fusion key matrix to obtain the attention score of the original fusion feature; S515. Perform cross-attention calculation based on the query matrix and sentiment key matrix to obtain the original sentiment attention score.

[0047] Specifically, the original fused features are multiplied with a preset query projection matrix to obtain a query matrix. This query matrix contains key query information extracted from the original fused features and is used for subsequent attention calculations with other matrices to determine the degree of correlation between different elements.

[0048] The original fused features and the preset key projection matrix are multiplied to obtain the fused key features. The fused key features contain key-related information from the original fused features and are used to interact with the query matrix to calculate the attention score within the original fused features.

[0049] A matrix multiplication operation is performed between the reference sentiment embedding and a predefined key projection matrix to obtain a sentiment key matrix. This sentiment key matrix contains key-related information from the reference sentiment embedding and is used to interact with the query matrix to calculate the attention score between the original fused features and the reference sentiment embedding. The sentiment key matrix has a dimension of N×D.

[0050] After obtaining the query matrix, fusion key matrix, and sentiment key matrix, the dot product between the query matrix and the fusion key matrix is ​​calculated to obtain the fusion similarity matrix, which is the attention score of the original fusion feature. By calculating the attention score of the original fusion feature, the correlation between different elements within the original fusion feature can be clarified, and the most important part can be identified, so that speech synthesis can pay more attention to the key information in the original fusion feature. See the following formula (2) for details: (2); in, The attention score represents the original fused feature. For querying the matrix, This is the transposed fusion bond matrix.

[0051] The dot product between the query matrix and the sentiment key matrix is ​​calculated to obtain the sentiment similarity matrix, which is the original sentiment attention score. By calculating the original sentiment attention score, it is possible to determine which elements in the original fusion features are more critical for expressing the reference sentiment. In the subsequent speech synthesis process, the connection between these elements and sentiment can be strengthened in a targeted manner, so that the synthesized speech can more accurately express the required sentiment. See the following formula (3) for details: (3); in, The original emotional attention score, For querying the matrix, This is the transposed sentiment key matrix.

[0052] S52. Calculate the value matrix based on the preset value projection matrix, the original fusion features, and the reference sentiment embedding to obtain the value matrix; In some embodiments, step S52, namely, calculating the value matrix based on the preset value projection matrix, the original fusion features and the reference sentiment embedding, to obtain the value matrix, includes the following features.

[0053] The preset value projection matrix and the original fusion features are aggregated and calculated to obtain the fusion value matrix; The preset value projection matrix and the reference sentiment embedding are aggregated and calculated to obtain the sentiment value matrix; The fused value matrix and the sentiment value matrix are concatenated using features to obtain the value matrix.

[0054] Specifically, The original fused features and the preset value projection matrix are multiplied together to obtain a fused value matrix, which contains the representation information of the original fused features in the new space. The dimension of the fused value matrix is ​​N×D.

[0055] The reference sentiment embedding and the preset value projection matrix are multiplied to obtain the sentiment value matrix, which contains the representation of the reference sentiment embedding in the new feature space. The dimension of this sentiment value matrix is ​​N×D.

[0056] Finally, the fused value matrix and the sentiment value matrix are concatenated along the feature dimension to obtain the final value matrix. This concatenation operation integrates the original fused features and the information from the reference sentiment embedding, enabling subsequent processing to simultaneously consider both content information and emotional information of the speech. The dimension of this value matrix is ​​2N×D.

[0057] It should be noted that the preset query projection matrix, key projection matrix, and value projection matrix are all parameters obtained through pre-training. The initial query projection matrix and key projection matrix can be set to 0, 1, or other values, and then the values ​​in these projection matrices are gradually updated during the training process. Specific training details are not limited in this application.

[0058] S53. Based on the dynamic emotion injection coefficient, emotion enhancement is performed on the original fusion feature attention score, the original emotion attention score and the value matrix to obtain the target fusion feature; It is important to understand that the emotional enhancement here refers to amplifying or adjusting the emotional-related features in the original fusion feature attention score, original emotional attention score, and value matrix based on the dynamic emotional injection coefficient, so that the synthesized speech can better express the required emotions.

[0059] Among them, such as Figure 5 As shown, step S53, which involves enhancing the original fusion feature attention score, original emotion attention score, and value matrix based on the dynamic emotion injection coefficient to obtain the target fusion feature, includes the following steps: S531. Based on the dynamic emotion injection coefficient, the original emotion attention score is adjusted to obtain the target emotion attention score; It should be noted that in the diffusion converter, the denoising process is performed based on the denoising time step parameters. Therefore, for each denoising time step, the corresponding dynamic emotion injection coefficient is obtained based on that denoising time step. Then, the dynamic emotion injection coefficient and the original emotion attention score are multiplied to obtain the target emotion attention score, as shown in the following formula (4): (4); in, For the target emotional attention score, The original emotional attention score, This is the dynamic emotion injection coefficient corresponding to the denoising time step.

[0060] Understandably, by adjusting the original emotional attention score through dynamic emotional injection coefficients, the process of emotional enhancement can be made more flexible and precise, enabling "prioritization" of emotional features in the early stages of speech synthesis to create a full and accurate emotional rhythm; and focusing on the repair of acoustic details in the later stages of speech synthesis.

[0061] S532. Based on the original fusion feature attention score, target emotion attention score and value matrix, perform aggregation calculation to obtain the target fusion feature.

[0062] It is important to understand that the aggregation calculation here refers to the calculation and processing of the original fusion feature attention score, target emotional attention score and value matrix according to the processing principle of cross-attention calculation mechanism to obtain the target fusion feature. The target fusion feature not only retains the accurate information of the original feature, but also incorporates appropriate emotional expression, so that the synthesized speech can convey information more naturally and accurately, while meeting the needs of emotional expression, improving the quality of speech synthesis and user satisfaction.

[0063] In step S532, which involves aggregating and calculating the target fusion features based on the original fusion feature attention score, the target emotion attention score, and the value matrix, the following steps are included: The original fused feature attention score and the target emotion attention score are concatenated to obtain the initial fused feature attention score. The cross-attention weights are obtained by normalizing the data dimensions based on the initial fusion feature attention scores and value matrix. The target fusion features are obtained by weighted summation based on the cross-attention weights and value matrices.

[0064] Specifically, the target emotional attention score obtained after emotional enhancement is concatenated with the original fusion feature attention score along the feature dimension to obtain the initial fusion attention score. This approach simultaneously considers the importance of both the original features and the emotional features, enabling subsequent processing to balance accurate information delivery and emotional expression. See formula (5) below for details: (5); in, This represents the initial fusion attention score.

[0065] Furthermore, the data dimensions of the value matrix are obtained, and the data dimensions of the initial fused feature attention scores and the value matrix are normalized to obtain the cross-attention weights, as shown in the following formula (6): (6); in, For cross-attention weights, To take the square root of the data dimension of the value matrix, The initial fusion attention score, This is the activation function used for normalization calculations.

[0066] After obtaining the cross-attention weights, the corresponding elements of the cross-attention weights and the value matrix are multiplied together, and then all products are summed to obtain the target fusion feature. See the following formula (7) for details: (7); in, To achieve the goal of feature fusion, For cross-attention weights, It is a value matrix.

[0067] S54. Perform residual linking on the original fusion features and the target fusion features to obtain the target synthesized spectral features.

[0068] Specifically, the original fused features and the target fused features are residually connected through the residual connection layer in the diffusion converter to obtain the target synthesized spectral features.

[0069] It should be noted that the denoising process of the diffusion converter is executed based on the denoising time step parameters. Therefore, for each denoising time step, steps S51 to S54 above will be executed once to obtain the spectral features under that denoising time step, until the last time step. The obtained spectral features are the target synthesized spectral features.

[0070] S60. Based on the target synthesized spectral features, perform audio conversion to obtain the target speech data.

[0071] The target synthesized spectral features are converted into audio using a pre-trained vocoder, transforming them into a time-domain speech signal to obtain the target speech data. This target speech data is a speech file that can be played on various devices.

[0072] Following step S60 in some embodiments, the synthesized target speech data is fed back to the client for audio playback. For example, in a fintech scenario, the generated target speech data is used in an intelligent investment advisory system to deliver investment advice to users. In a healthcare scenario, the target speech data is used in a medical consultation robot to provide patients with answers to their medical conditions and health guidance.

[0073] As can be seen, in the above solution, for speech synthesis scenarios in fintech and healthcare, the following steps are taken: First, reference speech data and speech synthesis text are acquired. Emotion extraction is performed on the reference speech data to accurately capture emotional information, resulting in a reference emotion embedding. The reference speech data, speech synthesis text, and preset noise features are then fused to obtain the original fused features. Next, dynamic emotion injection coefficients are determined based on preset denoising time step parameters, allowing for flexible adjustment of the emotion injection level according to the speech synthesis process. Furthermore, the original fused features are denoised based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain target synthesis spectral features. This effectively removes noise interference while appropriately injecting emotion, resulting in clearer and more accurate target synthesis spectral features. Finally, audio conversion is performed based on the target synthesis spectral features to generate high-quality and emotionally rich target speech data. This addresses the technical problem of diluted emotional information caused by directly splicing audio noise, text conditions, and reference audio conditions, which makes precise emotion control difficult during speech synthesis and affects the accuracy of emotional expression in speech synthesis. This solution improves the accuracy and naturalness of emotional expression in speech synthesis.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0075] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one with the speech synthesis methods described in the above embodiments. For example... Figure 6 As shown, the speech synthesis device includes a data acquisition module 101, an emotion extraction module 102, a feature fusion module 103, an injection coefficient determination module 104, a noise reduction module 105, and an audio conversion module 106. Detailed descriptions of each functional module are as follows: Data acquisition module 101 is used to acquire reference speech data and speech-synthesized text; The emotion extraction module 102 is used to extract emotions from the reference speech data to obtain the reference emotion embedding; The feature fusion module 103 is used to fuse reference speech data, speech-synthesized text, and preset noise features to obtain the original fused features; The injection coefficient determination module 104 is used to determine the dynamic emotion injection coefficient based on the preset denoising time step parameters; Denoising module 105 is used to denoise the original fused features based on dynamic emotion injection coefficients and reference emotion embedding to obtain target synthesized spectral features; The audio conversion module 106 is used to perform audio conversion based on the target synthesized spectral features to obtain target speech data.

[0076] In one embodiment, the emotion extraction module 102 is specifically used for: Emotional features are extracted from the reference speech data to obtain the original emotional embedding; The original sentiment embedding is transformed to obtain a reference sentiment embedding.

[0077] In one embodiment, the feature fusion module 103 is specifically used for: Spectral transformation is performed on the reference speech data to obtain the reference spectral features; Text encoding is performed on the synthesized speech text to obtain text encoding features; The reference spectral features, text encoding features, and noise features are concatenated to obtain the original fused features.

[0078] In one embodiment, the noise reduction module 105 is specifically used for: Cross-attention calculation is performed based on the original fusion features and the reference sentiment embedding to obtain the original fusion feature attention score and the original sentiment attention score; The value matrix is ​​calculated based on the preset value projection matrix, the original fusion features, and the reference sentiment embedding. Based on the dynamic emotion injection coefficient, emotion enhancement is performed on the original fusion feature attention score, the original emotion attention score, and the value matrix to obtain the target fusion feature; The original fusion features and the target fusion features are residually linked to obtain the target synthetic spectral features.

[0079] In one embodiment, the noise reduction module 105 is specifically used for: The original emotional attention score is adjusted based on the dynamic emotional injection coefficient to obtain the target emotional attention score; The target fusion features are obtained by aggregating and calculating the attention scores of the original fusion features, the target emotion attention scores, and the value matrix.

[0080] In one embodiment, the noise reduction module 105 is specifically used for: The original fused feature attention score and the target emotion attention score are concatenated to obtain the initial fused feature attention score. The cross-attention weights are obtained by normalizing the data dimensions based on the initial fusion feature attention scores and value matrix. The target fusion features are obtained by weighted summation based on the cross-attention weights and value matrices.

[0081] In one embodiment, the noise reduction module 105 is specifically used for: The query matrix is ​​obtained by performing aggregation calculations based on the preset query projection matrix and the original fusion features; The fusion key matrix is ​​obtained by performing aggregation calculations based on the preset key projection matrix and the original fusion features; The sentiment key matrix is ​​obtained by aggregation calculation based on the key projection matrix and the reference sentiment embedding; Cross-attention calculation is performed based on the query matrix and the fusion key matrix to obtain the attention score of the original fusion feature; Cross-attention calculation is performed based on the query matrix and the sentiment key matrix to obtain the original sentiment attention score.

[0082] This application provides a speech synthesis device for speech synthesis scenarios in fintech and healthcare. First, it acquires reference speech data and speech synthesis text. Emotion extraction is performed on the reference speech data to accurately capture emotional information, resulting in a reference emotion embedding. The reference speech data, speech synthesis text, and preset noise features are then fused to obtain original fused features. Next, dynamic emotion injection coefficients are determined based on preset denoising time step parameters, allowing for flexible adjustment of the emotion injection level according to the speech synthesis process. Further, the original fused features are denoised based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain target synthesis spectral features, effectively removing noise interference while appropriately injecting emotion, resulting in clearer and more accurate target synthesis spectral features. Finally, audio conversion is performed based on the target synthesis spectral features to generate high-quality and emotionally rich target speech data. This addresses the technical problem of diluted emotional information caused by directly splicing audio noise, text conditions, and reference audio conditions, which hinders precise emotion control during speech synthesis and affects the accuracy of emotional expression. This application improves the accuracy and naturalness of emotional expression in speech synthesis.

[0083] For specific limitations regarding the speech synthesis device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0084] Please see Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the speech synthesis method of the embodiments of this application, including: Obtain reference speech data and speech-synthesized text; Emotion extraction is performed on the reference speech data to obtain the reference emotion embedding; The reference speech data, the synthesized speech text, and the preset noise features are fused to obtain the original fused features; The dynamic emotion injection coefficient is determined based on the preset denoising time step parameters; The original fused features are denoised based on the dynamic emotion injection coefficient and the reference emotion embedding to obtain the target synthesized spectral features; Audio conversion is performed based on the target synthesized spectral features to obtain the target speech data.

[0085] The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0086] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain reference speech data and speech-synthesized text; Emotion extraction is performed on the reference speech data to obtain the reference emotion embedding; The reference speech data, the synthesized speech text, and the preset noise features are fused to obtain the original fused features; The dynamic emotion injection coefficient is determined based on the preset denoising time step parameters; The original fused features are denoised based on the dynamic emotion injection coefficient and the reference emotion embedding to obtain the target synthesized spectral features; Audio conversion is performed based on the target synthesized spectral features to obtain the target speech data.

[0087] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0091] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain reference speech data and speech-synthesized text; Emotion extraction is performed on the reference speech data to obtain the reference emotion embedding; The reference speech data, the synthesized speech text, and preset noise features are fused to obtain the original fused features; The dynamic emotion injection coefficient is determined based on the preset denoising time step parameters; Based on the dynamic emotion injection coefficient and the reference emotion embedding, the original fused features are denoised to obtain the target synthesized spectral features; Audio conversion is performed based on the target synthesized spectral features to obtain target speech data.

2. The speech synthesis method according to claim 1, characterized in that, The step of denoising the original fused features based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain the target synthesized spectral features includes: Cross-attention calculation is performed based on the original fusion features and the reference sentiment embedding to obtain the original fusion feature attention score and the original sentiment attention score. The value matrix is ​​calculated based on the preset value projection matrix, the original fusion features, and the reference sentiment embedding to obtain the value matrix. Based on the dynamic emotion injection coefficient, the original fusion feature attention score, the original emotion attention score, and the value matrix are emotionally enhanced to obtain the target fusion feature; The original fusion feature and the target fusion feature are residually linked to obtain the target synthesized spectral feature.

3. The speech synthesis method according to claim 2, characterized in that, The process of enhancing the original fusion feature attention score, the original emotion attention score, and the value matrix based on the dynamic emotion injection coefficient to obtain the target fusion feature includes: Based on the dynamic emotion injection coefficient, the original emotion attention score is adjusted to obtain the target emotion attention score; The target fusion feature is obtained by performing aggregation calculations based on the original fusion feature attention score, the target emotion attention score, and the value matrix.

4. The speech synthesis method according to claim 3, characterized in that, The process of aggregating and calculating the target fusion features based on the original fusion feature attention score, the target emotion attention score, and the value matrix to obtain the target fusion features includes: The original fused feature attention score and the target emotion attention score are concatenated to obtain the initial fused feature attention score; Based on the initial fusion feature attention scores and the data dimensions of the value matrix, normalization processing is performed to obtain the cross-attention weights; The target fusion feature is obtained by weighted summation based on the cross-attention weights and the value matrix.

5. The speech synthesis method according to claim 2, characterized in that, The process of calculating cross-attention based on the original fusion features and the reference sentiment embedding to obtain the original fusion feature attention score and the original sentiment attention score includes: An aggregation calculation is performed based on the preset query projection matrix and the original fusion features to obtain the query matrix; Based on the preset key projection matrix and the original fusion features, an aggregation calculation is performed to obtain the fusion key matrix; Based on the key projection matrix and the reference sentiment embedding, an aggregation calculation is performed to obtain the sentiment key matrix; Cross-attention calculation is performed based on the query matrix and the fusion key matrix to obtain the attention score of the original fusion feature; The original emotional attention score is obtained by performing cross-attention calculation based on the query matrix and the sentiment key matrix.

6. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The process of fusing the reference speech data, the synthesized speech text, and preset noise features to obtain the original fused features includes: The reference speech data is subjected to spectral transformation to obtain reference spectral features; The synthesized speech text is text-encoded to obtain text-encoded features; The original fused feature is obtained by concatenating the reference spectral features, the text encoding features, and the noise features.

7. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The step of extracting emotion from the reference speech data to obtain a reference emotion embedding includes: Emotional features are extracted from the reference speech data to obtain the original emotion embedding; The original emotion embedding is subjected to dimensionality transformation to obtain the reference emotion embedding.

8. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire reference speech data and speech-synthesized text; The emotion extraction module is used to extract emotions from the reference speech data to obtain a reference emotion embedding; The feature fusion module is used to fuse the reference speech data, the speech-synthesized text, and preset noise features to obtain the original fused features; The injection coefficient determination module is used to determine the dynamic emotion injection coefficient based on preset denoising time step parameters; The denoising module is used to denoise the original fused features based on the dynamic emotion injection coefficients and the reference emotion embedding to obtain the target synthesized spectral features; The audio conversion module is used to perform audio conversion based on the target synthesized spectral features to obtain target speech data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech synthesis method as described in any one of claims 1 to 7.

Images