Speech synthesis method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, provides a speech synthesis method and system, electronic equipment and a medium, and can be applied to intelligent interaction scenes in financial and medical scenes. The method comprises the steps of obtaining a to-be-synthesized target text, a target speaker identifier and a reference prompt voice containing tone and style information; encoding the target text to generate a text embedding vector sequence; generating a speaker embedding vector based on the target speaker identifier; performing feature extraction and dimension mapping on the reference prompt voice, and performing time sequence alignment based on the text sequence length to obtain a prompt feature vector sequence; performing element-by-element feature fusion on the three vectors to construct a fusion condition sequence; and finally, inputting an acoustic model to predict a target Mel spectrum and generating voice. By fusing the multi-dimensional condition features, the problem that it is difficult to consider identity stability and style expressive force in speech synthesis in a dialogue scene is solved.

Description

Technical Field

[0001] This application relates to the fields of medical technology, financial technology, and artificial intelligence technology, and in particular to a speech synthesis method, system, electronic device, and storage medium. Background Technology

[0002] In the field of speech synthesis technology, how to ensure that synthesized speech maintains stable speaker identity features while possessing rich intonation and style that conforms to the context is a pressing technical challenge. Existing technologies typically have two limitations: the first is relying solely on speaker identifiers to generate speech. While this method can guarantee stable timbre, the synthesized speech often lacks a flat tone and emotional variation; the second is relying on reference audio for style transfer. If the reference audio contains background noise or is too short, the extracted acoustic features become extremely unstable, leading to degraded sound quality or timbre drift in the synthesized speech.

[0003] Therefore, existing technologies struggle to balance identity stability and style richness, resulting in a poor human-computer interaction experience. Summary of the Invention

[0004] This application provides a speech synthesis method, system, electronic device, and computer-readable storage medium, aiming to improve the problem of low naturalness of synthesized speech caused by the difficulty in balancing identity stability and style diversity in existing speech synthesis.

[0005] In a first aspect, embodiments of this application provide a speech synthesis method, including: Acquire the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; The target text is encoded to generate a text embedding vector sequence, wherein the text embedding vector sequence consists of multiple text embedding vectors arranged in chronological order; Based on the target speaker identifier, generate the corresponding speaker embedding vector; Feature extraction and dimension mapping are performed on the reference prompt speech to obtain a mapped feature sequence, and the mapped feature sequence is temporally aligned based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence; A fusion condition sequence is constructed based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, wherein the fusion condition sequence is obtained by performing element-wise feature fusion on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; The fusion condition sequence is input into a preset acoustic model for prediction to obtain the target Mel spectrum, and the target speech is generated based on the target Mel spectrum.

[0006] Secondly, embodiments of this application provide a speech synthesis system based on multi-condition fusion, comprising: The data acquisition module is used to acquire the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; The text encoding module is used to encode the target text and generate a text embedding vector sequence; The speaker processing module is used to generate a corresponding speaker embedding vector based on the target speaker identifier; The prompt processing module is used to extract features and map dimensions of the reference prompt speech to obtain a mapped feature sequence, and to perform temporal alignment of the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence. The conditional fusion module is used to construct a fusion conditional sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; The speech generation module is used to input the fusion condition sequence into a preset acoustic model for prediction, obtain the target Mel spectrum, and generate target speech based on the target Mel spectrum.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.

[0009] The technical solution provided in this application acquires three-dimensional information of text, speaker identifiers, and prompt speech. It then uses temporal alignment technology to adjust the length of the prompt speech features to match the text sequence and constructs a unified fusion condition sequence using an element-wise fusion approach. By introducing a temporal alignment mechanism, the non-correspondence problem of heterogeneous modal data (text and audio) in the temporal dimension is solved. By constructing a ternary fusion condition sequence containing semantics, identity, and style, the acoustic model can simultaneously learn the nonlinear relationship between these three attributes within a unified feature space. The element-wise fusion strategy has a complementary enhancement effect: a stable speaker embedding vector serves as a global constraint, effectively suppressing noise interference that may be introduced due to poor quality reference prompt speech, ensuring the lower bound stability of the synthesized speech; while the dynamic prompt feature vector provides fine-grained prosodic adjustment, endowing the synthesized speech with rich emotional expressiveness. Ultimately, this method significantly improves the naturalness, robustness, and user experience of synthesized speech in dialogue scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a processing architecture diagram of multi-condition fusion speech synthesis provided in an embodiment of this application; Figure 2 This is a flowchart of a speech synthesis method provided in an embodiment of this application; Figure 3 This is a structural diagram of a speech synthesis system based on multi-condition fusion provided in an embodiment of this application; Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] like Figure 1 As shown, Figure 1 This is a schematic diagram of the image processing framework of the embodiment of this application. In this method, the target text to be synthesized, the target speaker identifier, and a reference prompt speech containing tone and style information are obtained; the target text is encoded to generate a text embedding vector sequence, wherein the text embedding vector sequence consists of multiple text embedding vectors arranged in temporal order; based on the target speaker identifier, a corresponding speaker embedding vector is generated; features are extracted and dimension-mapped from the reference prompt speech to obtain a mapped feature sequence, and the mapped feature sequence is temporally aligned based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence. A fusion condition sequence is constructed based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, wherein the fusion condition sequence is obtained by performing element-wise feature fusion on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; the fusion condition sequence is input into a preset acoustic model for prediction to obtain the target Mel spectrum, and the target speech is generated based on the target Mel spectrum.

[0014] It should be noted that this method can be applied to intelligent interaction scenarios in finance and healthcare, and can meet various application needs. For example, in the fintech field, it can be used for speech synthesis in customer service assistants for financial businesses such as insurance; in the medical technology field, it can also be used in the processing of intelligent terminals in medical services, or in other speech processing applications. Specific embodiments in this application are not limited to these applications.

[0015] It should be noted that the above method can be implemented through a server, which can be a standalone server or a server cluster consisting of multiple servers, without any specific limitation.

[0016] It should be noted that with the development of human-computer interaction technology, users' requirements for speech synthesis have shifted from simple clarity to a pursuit of human-like realism. Especially in continuous dialogue scenarios, machines not only need to maintain a fixed timbre (such as a specific assistant persona), but also need to express tones of happiness, regret, or urgency based on the dialogue content. This application introduces a multi-condition fusion mechanism to mathematically superimpose text content, speaker identity, and transient style in vector space, thereby achieving highly realistic speech synthesis.

[0017] In this application, the Mel spectrum refers to a time-frequency representation that simulates human auditory perception, mapping the frequency axis onto the Mel scale, and is often used as an intermediate acoustic feature in speech synthesis systems. Its physical significance lies in preserving the formant structure (which determines timbre) and energy distribution (which determines loudness) of speech, while removing high-frequency details that are insensitive to the human ear.

[0018] In this application, an embedding vector refers to a vector that maps discrete symbols (such as characters or identifiers) to real-valued vectors in a continuous vector space. For example, an identifier can be mapped to a 256-dimensional vector, where the numerical distribution implies the characteristic attributes of the object.

[0019] This application provides a speech synthesis method, please refer to the embodiments therein. Figure 1 This includes the following steps: S10: Obtain the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; In this step, the speech synthesis system receives input data in parallel from three dimensions, which together determine the final generated speech. The target text, i.e., the content the user wants the machine to read aloud, determines the semantic content and duration of the speech. The target speaker identifier is a numerical code used by the system to uniquely distinguish different speakers, determining the base timbre of the speech. The prompt speech is a pre-recorded audio clip whose content does not need to match the target text; its purpose is to provide a style template. This prompt speech contains intonation and style information; for example, if the prompt speech is fast-paced and high-pitched, the system will mimic this rhythm to read the target text.

[0020] S20: Encode the target text to generate a text embedding vector sequence, wherein the text embedding vector sequence consists of multiple text embedding vectors arranged in time sequence; In this step, text encoding involves complex linguistic feature extraction. First, a front-end module performs character-to-phoneme conversion, transforming Chinese characters into phoneme sequences and performing prosody prediction. Then, a text encoder maps discrete phoneme identifiers into dense vectors. Let the length of the phoneme sequence be N, and the vector dimension corresponding to each phoneme be d (e.g., d=100), then the generated text embedding vector sequence can be represented as T = [T_1, T_2,..., T_N]. Each vector T_i contains the pronunciation features of the current character and its contextual semantic relationships.

[0021] S30: Generate the corresponding speaker embedding vector based on the target speaker identifier; This step aims to obtain a global, static identity feature vector. Unlike feature extraction from audio, the method of generating vectors based on identifiers is unaffected by environmental noise, ensuring that the synthesized speech maintains consistent identity features. The corresponding speaker embedding vector can be obtained based on the target speaker identifier, combined with a speaker encoder.

[0022] S40: Perform feature extraction and dimension mapping on the reference prompt speech to obtain a mapped feature sequence, and perform temporal alignment on the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence; This step is crucial for achieving style transfer. Since the duration of the reference audio prompts and the target text readings are usually inconsistent, and their feature dimensions also do not match, direct fusion is not possible. This step transforms the disordered audio signals into ordered, text-corresponding style control sequences through feature space transformation and temporal scaling.

[0023] S50: Construct a fusion condition sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, wherein the fusion condition sequence is obtained by performing element-wise feature fusion on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence.

[0024] S60: Input the fusion condition sequence into a preset acoustic model for prediction to obtain the target Mel spectrum, and generate the target speech based on the target Mel spectrum.

[0025] The acoustic model is responsible for decoding the highly abstract fusion features into a visualized acoustic spectrum (i.e., the Mel spectrum), and finally the vocoder restores the spectrum into a playable waveform file.

[0026] As can be seen, the speech synthesis method provided in this application, through steps S10 to S60, achieves deep decoupling and reorganization of multi-dimensional features. Specifically, this method breaks through the limitations of single-condition control in traditional speech synthesis. By introducing a temporal alignment mechanism, it solves the problem of non-correspondence between heterogeneous modal data (text and audio) in the temporal dimension. By constructing a ternary fusion conditional sequence containing semantics, identity, and style, the acoustic model can simultaneously learn the nonlinear relationship between these three attributes in a unified feature space. The element-wise fusion strategy has a complementary enhancement effect: the stable speaker embedding vector serves as a global constraint, effectively suppressing noise interference that may be introduced due to poor quality of the reference prompt speech, ensuring the lower limit stability of the synthesized speech; while the dynamic prompt feature vector provides fine-grained prosodic adjustment, endowing the synthesized speech with rich emotional expressiveness. Ultimately, this method significantly improves the naturalness, robustness, and user experience of synthesized speech in dialogue scenarios.

[0027] In one embodiment, step S30, namely generating the corresponding speaker embedding vector based on the target speaker identifier, includes the following steps: S31: Obtain a pre-constructed speaker embedding matrix, wherein the speaker embedding matrix contains a one-to-one mapping relationship between multiple speaker identifiers and embedding vectors; S32: Retrieve the corresponding feature vector from the speaker embedding matrix based on the target speaker identifier, and use it as the speaker embedding vector.

[0028] Specifically, the speaker embedding matrix, serving as the core voiceprint parameter library of the system, is essentially a lookup table optimized through backpropagation during the model training phase. The matrix has a dimension of K×d, where K represents the total number of speakers supported by the system, and d represents the dimension of the feature vectors. During training, this matrix gradually learns and converges to the most essential acoustic feature distribution of each speaker, such as the fundamental frequency range and formant structure of the vocal cords. In practical inference applications, after receiving the target speaker identifier, the system does not need to perform complex acoustic analysis but directly indexes the matrix in O(1) time complexity to accurately extract the corresponding row vectors.

[0029] Specifically, this step mainly involves transforming discrete identity identifiers into continuous features that can be computed by computers, and mainly includes the following processes: Training Phase: The system maintains a learnable embedding matrix, assigning a randomly initialized vector S∈R to each speaker in the dialogue. d As the model is trained, the vector is continuously updated, eventually capturing the speaker's unique voiceprint features (such as fundamental frequency and formant distribution).

[0030] Inference Phase: When the system receives the target speaker identifier specified by the user, it directly retrieves the unique static vector S corresponding to that speaker through a table lookup operation. At this time, the dimension of vector S is d (e.g., 256 dimensions), which is a global feature and does not change over time.

[0031] It is worth noting that although a vector S representing the identity is obtained, the text to be synthesized is a sequence T = [T_1, T_2, ..., T_N] of length N. Mathematically, a static vector S (dimension 1×d) cannot be directly added element-wise to a dynamic sequence T (N×d). Therefore, to achieve subsequent fusion, this application introduces sequence expansion or broadcasting operations during the inference stage. Specifically, the system copies the obtained static vector S N times along the time axis to construct a new speaker embedding sequence, denoted as S'. S' = [S_1, S_2, ..., S_N], where each element S_1, S_2, ..., S_N in the sequence is numerically identical to the original vector S.

[0032] It's important to understand that this formula assigns a global identity attribute to each local text lexical unit. Whether processing the first or last character of a sentence, the associated speaker identity feature S_n remains consistent. This ensures a high degree of timbre uniformity throughout the entire speech synthesis process, preventing voice distortion halfway through reading.

[0033] In this embodiment, the lookup-based implementation effectively avoids the drawbacks of existing technologies that rely on real-time extraction of speaker features from reference audio. Since reference audio often contains environmental noise, room reverberation, or differences in recording equipment, the features extracted in real-time are prone to fluctuation, leading to timbre drift in the synthesized speech. In this embodiment, the embedding vector obtained through the lookup table is the globally optimal solution trained on large-scale clean data, exhibiting extremely high stability and purity. This means that regardless of changes in the quality of the user-input reference prompt speech, the base timbre of the synthesized speech remains highly consistent, significantly enhancing the system's robustness.

[0034] In one embodiment, the feature extraction and dimension mapping process in step S40 includes the following steps: S41: Extract the original Mel-spectral features of the reference prompt speech; S42: Construct a projection layer and use the projection layer to perform a linear mapping on the original Mel spectrum features, converting the feature dimensions of the original Mel spectrum features to be consistent with the feature dimensions of the text embedding vector sequence, thereby obtaining the mapped feature sequence.

[0035] In this embodiment, a spatial mapping strategy is adopted to address the issue of mismatched physical dimensions between different modal data. First, the time-domain reference prompt speech is converted into a Mel spectrum in the frequency domain. Its feature dimension is typically determined by the number of frequency bands (e.g., 80 dimensions), which is much smaller than the dimension of the text embedding vector (e.g., 512 dimensions) that carries rich semantics. Therefore, the system constructs a projection layer, which is typically composed of a fully connected layer or a convolutional neural network. During processing, the original low-dimensional Mel spectrum features are input to the projection layer, and through matrix multiplication and bias addition operations, they are forcibly mapped to a high-dimensional space consistent with the text vector dimension.

[0036] As can be seen, in this embodiment, this process not only solves the compatibility problem in the physical dimension, but also achieves semantic alignment from the acoustic feature space to the hidden semantic space. The projection layer automatically adjusts its weights during training, thereby intelligently selecting the feature components in the Mel spectrum most relevant to tone and emotion, while suppressing irrelevant information such as background noise. This transformation of the feature space lays a solid foundation for the subsequent accurate fusion of style features and textual semantic features.

[0037] In one embodiment, the timing alignment process in step S40 includes the following steps: S43: Determine the length difference relationship between the length of the mapped feature sequence in the time dimension and the sequence length of the text embedding vector sequence; S44: Based on the length difference relationship, the alignment module is used to adjust the mapping feature sequence to the sequence length to generate the prompt feature vector sequence.

[0038] In this embodiment, since the duration of the user-provided prompt voice (e.g., 2 seconds) and the duration of the text to be synthesized (e.g., 5 seconds) typically do not have a one-to-one correspondence, direct fusion would lead to dimensional misalignment. Therefore, this embodiment introduces an alignment module to address this temporal mismatch problem. The system first calculates the difference between the number of frames in the mapped feature sequence and the length of the text embedding vector sequence. Then, it uses an alignment algorithm to construct a new sequence, ensuring that the length of the new sequence is strictly equal to the length of the text sequence, and that the features at each position in the new sequence originate from a reasonable distribution of the original audio features.

[0039] Specifically, the process mainly involves converting audio signals into fixed-length style control variables that correspond one-to-one with the text.

[0040] First, feature extraction is performed on the reference prompt speech. The system extracts Mel-spectral features and maps them to dimension d through a projection layer. The length of the resulting feature sequence depends on the duration of the audio and is independent of the text length. Subsequently, the system performs a crucial temporal alignment operation. Since the length of the text embedding vector sequence T generated in step S20 is N (i.e., containing N tokens), the features of the prompt speech must be recombined to the same length in order to perform element-wise fusion later. The system uses an attention mechanism or interpolation algorithm to aggregate the original audio features into a new sequence of length N.

[0041] After the alignment process described above, the system finally obtains a sequence of cue feature vectors with shape M∈R. d A vector group, mathematically expressed as: M = [M_1, M_2, ..., M_N] Here, N represents the length of the target text sequence (i.e., the length of T in step S20), not the speaker number. This means that regardless of the length of the original prompt speech (e.g., 3 seconds or 10 seconds), the final generated vector set M always contains N vectors to ensure a precise one-word-one-style correspondence with the text lexical units. M_i indicates that each element M_i in the sequence is a feature vector of dimension d. It represents the intensity of tone, pitch, or speed that should be applied when reading the i-th word.

[0042] The above formula can be understood as follows: the entire vector set M can be viewed as a style control curve parallel to the text content. For example, if the energy values ​​of M_1 to M_3 are high, it means that the first three words of the sentence need to be read with enthusiasm; if the value of M_N is flat, it means that the end of the sentence needs to be read smoothly. This formula quantifies the distribution of style information on the time axis and is the mathematical foundation for achieving fine-grained emotion control.

[0043] In this embodiment, through this mandatory temporal alignment operation, the system ensures that each text token is accurately assigned to a corresponding style feature vector. This fundamentally avoids style gaps in synthesized speech caused by inconsistent lengths (such as the first half being emotional while the second half is emotionless) or prosodic misalignment, ensuring the continuity and integrity of style rendering throughout the entire sentence.

[0044] In one embodiment, step S44 above, namely the alignment step, specifically includes the following steps: S441: Calculate the alignment weight between the mapped feature sequence and the text embedding vector sequence using an attention pooling mechanism or a segmented averaging mechanism; S442: Aggregate the mapping feature sequence based on the alignment weight to generate the prompt feature vector sequence with the same length as the sequence length.

[0045] To more clearly illustrate the processing logic of this step, the alignment weights and aggregation process are explained in detail here. Alignment weights refer to the strength of association or probability of correlation between the current text word and a specific time frame in the reference speech within the semantic space. For example, if the text word is cheerful, and the third frame of the reference speech has high-energy laughter features, then the vector similarity between the two in the hidden space will be very high, and the calculated weight may be close to 0.9; conversely, if the third frame is silent, the weight may be close to 0.0. Aggregation is the process of weighted summation of the original features based on the aforementioned weights.

[0046] Specifically, for the i$-th word in the text sequence, the system calculates the weight distribution a_{i,1}, a_{i,2}, ..., a_{i,J}$ between it and all J audio frames in the mapped feature sequence. Then, it uses this weight distribution to perform a weighted summation of all audio frame vectors. Through this calculation, the system generates a new cue feature vector sequence of length N, where each vector is a style feature extracted and reconstructed from the original audio that is most suitable for the current text.

[0047] In addition, if a segmented averaging mechanism is used, the audio frame sequence is evenly divided into N segments according to the text length N, and then the arithmetic mean of all vectors in each segment is calculated and used as the style vector of the corresponding text word.

[0048] This embodiment provides two alignment algorithms adapted to different scenarios. The attention pooling mechanism endows the model with semantic awareness. Specifically, the system uses the current text embedding vector as the query vector and the audio feature sequence as the key vector. By calculating the dot product similarity between the two, it determines which segment of the audio should be emphasized in the current text. For example, when processing the word "sadness," the model automatically assigns higher weights to segments with lower energy and slower speech in the prompt speech and performs weighted aggregation. The segmented averaging mechanism, on the other hand, is an efficient hard alignment method that evenly segments the audio sequence according to the text length and takes the average, suitable for scenarios with limited computing resources.

[0049] As can be seen, the aggregation mechanism that introduces attention weights endows style transfer with a high degree of semantic awareness. The model no longer rigidly stretches audio, but intelligently determines the tone required for the current text and accurately extracts the corresponding acoustic segments from the reference audio. This dynamic feature recombination greatly enhances the naturalness and logic of the synthesized speech prosody, avoiding the problem of style-content mismatch.

[0050] In one embodiment, step S50, namely constructing a fusion condition sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, includes the following steps: S51: Traverse each time step position in the text embedding vector sequence; S52: Obtain the text embedding vector corresponding to the current time step position and the corresponding prompt feature vector in the prompt feature vector sequence; S53: Add the text embedding vector, the prompt feature vector and the speaker embedding vector corresponding to the current time step position element by element to obtain the fusion condition vector corresponding to the current time step position; S54: Based on the fusion condition vectors corresponding to all time step positions, the fusion condition sequence is formed.

[0051] In the specific fusion operation, this embodiment employs an element-wise addition mathematical strategy. For each time step in the sequence, the system acquires the text vector (representing content), the broadcast speaker vector (representing identity), and the aligned cue vector (representing style), and accumulates the values ​​of these three vectors in their corresponding dimensions. From a signal processing perspective, this is similar to the superposition process of the fundamental wave (speaker), carrier wave (text), and modulated wave (style).

[0052] First, the system initiates a traversal loop, starting from the first time step (i.e., the first word) and continuing to the Nth time step for the length N of the text sequence. In each specific time step i, the system first reads the i-th vector T_i (assuming dimension d) from the text embedding vector sequence, and simultaneously reads the i-th vector M_i (also with dimension d) from the aligned cue feature vector sequence.

[0053] Next, for the globally unique speaker embedding vector S (of dimension d), the system performs a broadcast operation during computation, logically treating it as a constant vector that exists at every time step i. Then, the core element-wise addition operation is performed. This means the system extracts the values ​​of T_i, S, and M_i in the first dimension and adds them together to obtain the first dimension value of the fusion vector C_i; similarly, the same addition operation is performed on the second to d dimensions. Finally, the fusion condition vector C_i for the current time step is generated. After the loop ends, all the generated C_i are arranged in order to form the final fusion condition sequence.

[0054] Specifically, the system performs a fusion calculation for each position (i.e., the token position) in the text sequence. Let the current position being processed be i, and the target speaker's ID be n, then the fusion formula is: C_i =T_i +S_n+M_n In this formula, the physical meaning of each variable is as follows: i represents the position index (Time Step) of the currently processed text embedding vector in the sequence.

[0055] n represents the unique identifier of the target speaker (Speaker ID).

[0056] T_i represents the text embedding vector at sequence position i. It carries the semantic content information at the current moment.

[0057] S_n represents the speaker embedding vector obtained based on speaker ID n. This is a global vector representing the speaker's inherent vocal attributes (such as timbre). During computation, this vector S_n is broadcast and applied to every position i in the sequence, meaning that the identity features of the same speaker are present throughout the entire speech.

[0058] M_n represents the cue feature vector originating from the reference cue speech of speaker n and aligned to the current position i. Although the formula uniformly uses the subscript n to mark its source (i.e., the style feature belongs to the input sample of speaker n), in the actual operation logic, M_n here specifically refers to the style feature component that precisely corresponds to the current text position i after the temporal alignment in step S40.

[0059] In this step, the formula C_i = T_i + S_n + M_n is used for fusion, mathematically achieving decoupled control and linear superposition of semantics, identity, and style. T_i provides the content skeleton, S_n defines the baseline timbre plane, and M_n introduces dynamic emotional fluctuations onto the baseline plane. This feature combination method based on a specific speaker ID n ensures that the synthesis process strictly adheres to the identity constraints of the designated target person. Simultaneously, the introduction of M_n allows the model to perform fine-grained prosodic rendering based on the style of the reference sample n, thus solving the problems of style loss or identity instability caused by single conditions in existing technologies.

[0060] As can be seen, in this embodiment, compared to the traditional splicing fusion strategy (sponging leads to a doubling of feature dimensions, thus significantly increasing the number of parameters and computational burden of subsequent models), element-wise addition achieves deep fusion of multi-source information while maintaining the same feature dimensions. This fusion method forces the model to learn the nonlinear coupling relationship between the three sources within the same hidden layer feature subspace, and produces a highly fault-tolerant complementary effect: when the prompt speech quality is poor or the features are not obvious, the style vector value approaches zero, and the system automatically degenerates into a standard TTS model, relying on the strong speaker vector to ensure the lower limit stability of the output; while when the prompt speech style is strong, the fluctuation of the style vector value will significantly modulate the final fused features, thereby achieving strong control over speech emotion.

[0061] In one embodiment, step S60, which involves inputting the fusion condition sequence into a preset acoustic model for prediction to obtain the target Mel spectrum, includes the following steps: S61: Input the fusion condition sequence into a decoder based on a Transformer structure to predict the target Mel spectrum corresponding to the target text; S62: Input the target Mel spectrum into the vocoder for waveform reconstruction and output the synthesized target speech.

[0062] During the speech generation stage, the system leverages the powerful sequence modeling capabilities of the Transformer decoder to process the fused feature sequences. The Transformer's core multi-head self-attention mechanism can capture long-range dependencies within the sequence in parallel; for example, the tone setting at the beginning of a sentence can be passed on through the attention mechanism and influence the tone generation at the end of the sentence. Subsequently, the generated Mel spectrum is fed into a vocoder (such as HiFi-GAN). Since the Mel spectrum only contains amplitude information and loses phase information, the vocoder infers reasonable phase information through a complex generative adversarial network, restoring the frequency domain spectrum to a high-sampling-rate time-domain waveform.

[0063] In this embodiment, the process fully combines the advantages of the Transformer in capturing long temporal dependencies with the advantages of the vocoder in waveform reconstruction. Combined with the high-quality, multi-dimensional fused conditional sequences generated in the preceding steps, the final output speech not only exhibits clear texture and smooth transitions in the frequency domain but also maintains extremely high fidelity in the time domain. This end-to-end generation path maximizes the reproduction of the subtle emotional granularity in the reference prompt speech while preserving the inherent timbre characteristics of the target speaker, achieving highly realistic speech synthesis.

[0064] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0065] This application also provides a speech synthesis system based on multi-condition fusion, which corresponds one-to-one with a speech synthesis method described in the above embodiments. Please refer to... Figure 3 ,include: The data acquisition module 310 is used to acquire the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; Text encoding module 320 is used to encode the target text and generate a text embedding vector sequence; Speaker processing module 330 is used to generate a corresponding speaker embedding vector based on the target speaker identifier; The prompt processing module 340 is used to extract features and map dimensions of the reference prompt speech to obtain a mapped feature sequence, and to perform temporal alignment of the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence. Conditional fusion module 350 is used to construct a fusion conditional sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; The speech generation module 360 ​​is used to input the fusion condition sequence into a preset acoustic model for prediction, obtain the target Mel spectrum, and generate target speech based on the target Mel spectrum.

[0066] In one embodiment, generating the corresponding speaker embedding vector based on the target speaker identifier includes: Obtain a pre-constructed speaker embedding matrix, wherein the speaker embedding matrix contains a one-to-one mapping relationship between multiple speaker identifiers and embedding vectors; The corresponding feature vector is retrieved from the speaker embedding matrix based on the target speaker identifier, and used as the speaker embedding vector.

[0067] In one embodiment, the step of extracting features and mapping dimensions of the reference prompt speech to obtain a mapped feature sequence includes: Extract the raw Mel-spectral features of the reference prompt speech; A projection layer is constructed, and the original Mel spectral features are linearly mapped using the projection layer to convert the feature dimensions of the original Mel spectral features to be consistent with the feature dimensions of the text embedding vector sequence, thereby obtaining the mapped feature sequence.

[0068] In one embodiment, the step of temporally aligning the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain the cue feature vector sequence includes: Determine the length difference relationship between the length of the mapped feature sequence in the time dimension and the sequence length of the text embedding vector sequence; Based on the length difference relationship, the alignment module is used to adjust the mapping feature sequence to the sequence length to generate the prompt feature vector sequence.

[0069] In one embodiment, adjusting the mapped feature sequence to the sequence length using an alignment module based on the length difference relationship includes: The alignment weight between the mapped feature sequence and the text embedding vector sequence is calculated using an attention pooling mechanism or a segmented averaging mechanism. The mapping feature sequence is aggregated based on the alignment weight to generate the prompt feature vector sequence with the same length as the sequence.

[0070] In one embodiment, constructing a fusion condition sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence includes: Iterate through each time step position in the text embedding vector sequence; Obtain the text embedding vector corresponding to the current time step position and the corresponding prompt feature vector in the prompt feature vector sequence; The text embedding vector, the cue feature vector, and the speaker embedding vector corresponding to the current time step position are added element by element to obtain the fusion condition vector corresponding to the current time step position. The fusion condition sequence is formed based on the fusion condition vectors corresponding to all time step positions.

[0071] In one embodiment, the step of inputting the fusion condition sequence into a preset acoustic model for prediction to obtain a target Mel spectrum, and generating target speech based on the target Mel spectrum, includes: The fusion condition sequence is input into a decoder based on a Transformer structure to predict the target Mel spectrum corresponding to the target text; The target Mel spectrum is input into a vocoder for waveform reconstruction, and the synthesized target speech is output.

[0072] The system provided in this application acquires text, speaker identifiers, and three-dimensional information of prompt speech. Temporal alignment technology is used to adjust the length of the prompt speech features to match the text sequence, and a unified fusion condition sequence is constructed using an element-by-element fusion approach. This scheme ensures identity consistency in long dialogue scenarios through speaker embedding vectors, while supplementing real-time tone and style information using prompt feature vectors. This deep fusion at the feature level allows the model to simultaneously consider identity and style, thereby significantly improving the naturalness and expressiveness of speech synthesis in complex dialogue scenarios.

[0073] For specific limitations regarding the speech synthesis system, please refer to the limitations on speech synthesis methods mentioned above, which will not be repeated here. Each module in the aforementioned speech synthesis system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.

[0074] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. When executed by the processor, the computer program implements the functions of a speech synthesis system or the steps of a speech synthesis method.

[0075] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the functions of a speech synthesis system, or when it executes the program, it implements the steps of a graph-speech synthesis method, to avoid repetition.

[0076] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the functions of a speech synthesis system, or when executed, implements the steps of a speech synthesis method. To avoid repetition, these steps will not be repeated here.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0079] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speech synthesis method characterized by, include: Acquire the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; The target text is encoded to generate a text embedding vector sequence, wherein the text embedding vector sequence consists of multiple text embedding vectors arranged in chronological order; Based on the target speaker identifier, generate the corresponding speaker embedding vector; Feature extraction and dimension mapping are performed on the reference prompt speech to obtain a mapped feature sequence, and the mapped feature sequence is temporally aligned based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence; A fusion condition sequence is constructed based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, wherein the fusion condition sequence is obtained by performing element-wise feature fusion on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; The fusion condition sequence is input into a preset acoustic model for prediction to obtain the target Mel spectrum, and the target speech is generated based on the target Mel spectrum.

2. The method according to claim 1, characterized in that, The step of generating a corresponding speaker embedding vector based on the target speaker identifier includes: Obtain a pre-constructed speaker embedding matrix, wherein the speaker embedding matrix contains a one-to-one mapping relationship between multiple speaker identifiers and embedding vectors; The corresponding feature vector is retrieved from the speaker embedding matrix based on the target speaker identifier, and used as the speaker embedding vector.

3. The method according to claim 1, characterized in that, The step of extracting features and mapping dimensions of the reference prompt speech to obtain a mapped feature sequence includes: Extract the raw Mel-spectral features of the reference prompt speech; A projection layer is constructed, and the original Mel spectral features are linearly mapped using the projection layer to convert the feature dimensions of the original Mel spectral features to be consistent with the feature dimensions of the text embedding vector sequence, thereby obtaining the mapped feature sequence.

4. The method according to claim 1, characterized in that, The step of temporally aligning the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain the cue feature vector sequence includes: Determine the length difference relationship between the length of the mapped feature sequence in the time dimension and the sequence length of the text embedding vector sequence; Based on the length difference relationship, the alignment module is used to adjust the mapping feature sequence to the sequence length to generate the prompt feature vector sequence.

5. The method according to claim 4, characterized in that, The step of adjusting the mapped feature sequence to the sequence length using the alignment module based on the length difference relationship includes: The alignment weight between the mapped feature sequence and the text embedding vector sequence is calculated using an attention pooling mechanism or a segmented averaging mechanism. The mapping feature sequence is aggregated based on the alignment weight to generate the prompt feature vector sequence with the same length as the sequence.

6. The method according to claim 1, characterized in that, The construction of the fusion condition sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence includes: Iterate through each time step position in the text embedding vector sequence; Obtain the text embedding vector corresponding to the current time step position and the corresponding prompt feature vector in the prompt feature vector sequence; The text embedding vector, the cue feature vector, and the speaker embedding vector corresponding to the current time step position are added element by element to obtain the fusion condition vector corresponding to the current time step position. The fusion condition sequence is formed based on the fusion condition vectors corresponding to all time step positions.

7. The method according to any one of claims 1-6, characterized in that, The step of inputting the fusion condition sequence into a preset acoustic model for prediction to obtain the target Mel spectrum, and generating target speech based on the target Mel spectrum, includes: The fusion condition sequence is input into a decoder based on a Transformer structure to predict the target Mel spectrum corresponding to the target text; The target Mel spectrum is input into a vocoder for waveform reconstruction, and the synthesized target speech is output.

8. A speech synthesis system, characterized in that, include: The data acquisition module is used to acquire the target text to be synthesized, the target speaker identifier, and the reference prompt speech containing tone and style information; A text encoding module is used to encode the target text and generate a text embedding vector sequence, wherein the text embedding vector sequence consists of multiple text embedding vectors arranged in a time sequence; The speaker processing module is used to generate a corresponding speaker embedding vector based on the target speaker identifier; The prompt processing module is used to extract features and map dimensions of the reference prompt speech to obtain a mapped feature sequence, and to perform temporal alignment of the mapped feature sequence based on the sequence length of the text embedding vector sequence to obtain a prompt feature vector sequence. The conditional fusion module is used to construct a fusion conditional sequence based on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence, wherein the fusion conditional sequence is obtained by performing element-wise feature fusion on the text embedding vector sequence, the speaker embedding vector, and the cue feature vector sequence; The speech generation module is used to input the fusion condition sequence into a preset acoustic model for prediction, obtain the target Mel spectrum, and generate target speech based on the target Mel spectrum.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images