Speech generation method and device based on double-layer style modulation, equipment and medium

Abstract

The application relates to the technical field of speech synthesis, and discloses a speech generation method and device based on double-layer style modulation, equipment and a medium, which comprises the following steps: receiving an input text and a style description text, standardizing the input text, converting phonemes, and extracting prosodic features; generating phoneme embedding vectors and performing context coding; combining the prosodic features to perform gate modulation to obtain local prosodic features; performing semantic coding on the style description text and decomposing the style description text to obtain local style components and global style components; performing fusion modulation to obtain style modulation features; performing time length prediction and sequence expansion on the style modulation features to obtain expanded features; mapping and decoding the expanded features to obtain acoustic features; and generating a speech waveform based on the acoustic features. The application can be applied to business scenarios such as financial technology and medical health, rhythm features and paralanguage style modeling are separated, fusion modulation and feature constraint processing are performed, and therefore the controllability and stability of speech generation are improved.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech generation method, apparatus, device, and medium based on two-layer style modulation. Background Technology

[0002] With the development of speech synthesis technology, existing methods can generate relatively natural speech output at the levels of language content and basic prosody, but there are still significant shortcomings in terms of style expression. Especially when processing prosodic information and paralinguistic style information simultaneously, existing technologies often rely on a single information source or implicit modeling methods, resulting in coarse-grained style control, insufficient interpretability, and poor stability of the generated results. At the same time, some methods rely on reference audio or large-scale models for style modeling, which faces problems such as high computational cost, limited deployment, and sensitivity to input description in practical applications, making it difficult to maintain consistency and controllability of speech output in different scenarios.

[0003] In the fintech sector, voice interaction is widely used in scenarios such as intelligent customer service, voice broadcasting, investment advisory prompts, risk notifications, and compliance alerts. These scenarios demand high consistency, accuracy, and controllability in voice output style. For example, in wealth management recommendations, risk disclosures, or anti-fraud alerts, different tones and styles of expression are needed depending on the business type. However, existing technologies lack effective separation and coordination mechanisms when processing text content and style information, making the voice style susceptible to the influence of the input text's expression, resulting in inconsistent styles for similar business semantics. Furthermore, relying on large models makes it difficult to meet low-latency requirements in real-time transaction prompts or high-concurrency voice broadcasting scenarios, impacting the overall system response efficiency.

[0004] In the healthcare field, speech synthesis is applied to scenarios such as intelligent consultation, health education, medication reminders, and postoperative follow-up. These scenarios place high demands on the emotional expression, tone control, and stability of information delivery in speech. For example, a caring tone is needed in disease notification or health advice, while clarity and seriousness are required in emergency reminders. Existing technologies are insufficient in the coordinated expression of prosodic features and high-level stylistic information, making it difficult to simultaneously maintain consistency between tonal details and overall style. This results in a lack of stable control over the generated speech under different expressive requirements. Furthermore, some methods are sensitive to subtle changes in the input description, which can lead to inconsistent speech styles under different expressions of the same medical semantics, reducing the reliability and usability of the system in medical scenarios. Summary of the Invention

[0005] The main objective of this invention is to provide a speech generation method, apparatus, device, and storage medium based on two-layer style modulation, aiming to solve the technical problem that existing technologies are unable to achieve decoupled modeling and collaborative control of the prosodic features of text content and the paralinguistic style during speech synthesis, resulting in insufficient controllability, consistency, and stability of generated speech in terms of style expression.

[0006] To achieve the above objectives, the present invention provides a speech generation method based on two-layer style modulation, comprising: The system receives input text and style description text, performs standardization processing on the input text to obtain standardized text, performs phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extracts the prosodic features of the standardized text. A phoneme embedding vector is generated based on the phoneme sequence. Context encoding is performed on the phoneme embedding vector to obtain context dependencies. Gated modulation is then performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features. The style description text is semantically encoded to obtain a global style vector, and the global style vector is decomposed to obtain local style components and global style components. Gated fusion is performed based on the local style components and the local prosodic features to obtain local fusion features, and feature linear modulation is performed on the local fusion features based on the global style components to obtain style modulation features; The style modulation features are subjected to duration prediction to obtain the pronunciation duration, and the style modulation features are sequence extended based on the pronunciation duration to obtain extended features; The extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features; Based on the acoustic features, waveform synthesis processing is performed to obtain a speech waveform.

[0007] Furthermore, to achieve the above objectives, the present invention provides a speech generation apparatus based on dual-layer style modulation, comprising: The text preprocessing and prosody extraction module is used to receive input text and style description text, perform standardization processing on the input text to obtain standardized text, perform phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extract the prosodic features of the standardized text. The local prosodic modeling module is used to generate phoneme embedding vectors based on the phoneme sequence, perform context encoding processing on the phoneme embedding vectors to obtain context dependencies, and perform gating modulation based on the phoneme embedding vectors, the context dependencies, and the prosodic features to obtain local prosodic features. A global style encoding module is used to perform semantic encoding on the style description text to obtain a global style vector, and to decompose the global style vector to obtain local style components and global style components. The style fusion modulation module is used to perform gated fusion based on the local style components and the local prosodic features to obtain local fusion features, and to perform feature linear modulation on the local fusion features based on the global style components to obtain style modulation features; The duration modeling and expansion module is used to predict the duration of the style modulation features to obtain the pronunciation duration, and to expand the style modulation features based on the pronunciation duration to obtain expanded features; An acoustic modeling and decoding module is used to map the extended features to the acoustic latent space to obtain latent space features, and to perform acoustic decoding based on the latent space features to obtain acoustic features; The waveform generation module is used to perform waveform synthesis processing based on the acoustic features to obtain a speech waveform.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a speech generation program based on two-layer style modulation stored in the memory and executable on the processor, wherein when the speech generation program based on two-layer style modulation is executed by the processor, it implements the steps of the speech generation method based on two-layer style modulation as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a speech generation program based on two-layer style modulation, wherein when the speech generation program based on two-layer style modulation is executed by a processor, it implements the steps of the speech generation method based on two-layer style modulation as described above.

[0010] Beneficial Effects: This invention relates to the field of speech synthesis technology and discloses a speech generation method, apparatus, device, and medium based on two-layer style modulation, comprising: receiving input text and style description text; performing standardization processing, phoneme conversion processing, and extracting prosodic features on the input text; generating phoneme embedding vectors based on phoneme sequences and performing context encoding processing; and performing gated modulation combined with prosodic features to obtain local prosodic features; performing semantic encoding on the style description text and decomposing it to obtain local style components and global style components; performing gated fusion processing based on the local style components and local prosodic features; and performing feature linear modulation based on the global style components to obtain style modulation features; performing duration prediction on the style modulation features and performing sequence expansion to obtain extended features; mapping the extended features to the acoustic latent space and performing acoustic decoding to obtain acoustic features; and performing waveform synthesis processing based on the acoustic features to obtain a speech waveform. This invention can be applied to business scenarios such as fintech and healthcare. By separating the prosodic features and the modeling process of paralinguistic style, and introducing local and global modulation mechanisms in the fusion stage, it achieves the collaborative expression of multiple features. At the same time, by combining duration modeling and acoustic latent space mapping, the features are constrained and transformed step by step, thereby improving the style controllability and stability of speech generation. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a speech generation method based on two-layer style modulation according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the speech generation method based on dual-layer style modulation according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech generation device based on dual-layer style modulation of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The speech generation method based on two-layer style modulation provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can receive input text and style description text from the client, perform standardization and phoneme conversion on the input text, extract prosodic features, generate phoneme embedding vectors based on the phoneme sequence, and perform context encoding. Gated modulation is then applied to the prosodic features to obtain local prosodic features. The style description text is semantically encoded and decomposed to obtain local and global style components. Gated fusion processing is performed based on the local style components and local prosodic features, and feature linear modulation is applied to the global style components to obtain style modulation features. Duration prediction and sequence expansion are performed on the style modulation features to obtain extended features. These extended features are mapped to the acoustic latent space and acoustically decoded to obtain acoustic features. Waveform synthesis is then performed based on the acoustic features to obtain the speech waveform. This invention can be applied to business scenarios such as fintech and healthcare. By separating the prosodic features from the modeling process of paralinguistic styles and introducing local and global modulation mechanisms in the fusion stage, it achieves the collaborative expression of multiple types of features. Simultaneously, by combining duration modeling and acoustic latent space mapping to progressively constrain and transform features, it improves the style controllability and stability of speech generation. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the speech generation method based on two-layer style modulation provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the speech generation method based on two-layer style modulation proposed in this invention includes the following steps: S10, receive input text and style description text, perform standardization processing on the input text to obtain standardized text, perform phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extract the prosodic features of the standardized text; In this embodiment, the data input unit receiving input text and style description text synchronously collects two types of semantic information. The input text carries the information of speech content expression, while the style description text carries the information of speech expression form. The input text may originate from transaction broadcast statements, asset reminder statements, or risk disclosure statements in the fintech business system, while the style description text may be text content describing tone or expression style, such as risk warning tone or stable broadcast tone. The two types of text maintain a temporal correspondence when entering the processing unit, providing a unified input basis for subsequent processing.

[0016] Standardization processing of the input text involves unifying the format and standardizing the semantics of the original text to meet the requirements of subsequent phoneme conversion. Standardization includes converting structured content such as numbers, amounts, interest rates, and times, unifying different expressions into readable ones. For example, it converts interest rate and amount expressions in financial text into phonetically friendly formats. Simultaneously, it standardizes punctuation and removes redundant characters. In implementation, standardization can be achieved by combining rule matching and sequence labeling models. Rule matching is used to identify fixed pattern information such as amounts and dates, while sequence labeling models are used to identify complex semantic structures, thereby generating standardized text with consistent structure.

[0017] Phoneme conversion of standardized text maps a text sequence to a phoneme sequence, where each phoneme represents the smallest unit of pronunciation. Phoneme conversion is achieved through a phoneme dictionary or a neural network model. The phoneme dictionary provides the basic mapping relationships, while the neural network model handles complex cases such as polyphonic characters and domain-specific terminology. In financial scenarios, specialized terms such as fund names and financial product names are adapted using an expanded dictionary or domain-specific training data to ensure the stability and accuracy of the phoneme sequences. A one-to-one or one-to-many correspondence is maintained between the phoneme sequences and the text character sequences for subsequent speech generation processing.

[0018] Extracting prosodic features from standardized text involves acquiring information related to speech rhythm and tone from the text structure and semantics. Prosodic features include tone features, stress features, and pause boundary features. Tone features originate from the pitch variation rules of the language itself; stress features are derived from keyword recognition processes, such as assigning higher weights to risk warning words or key indicator words in financial texts; and pause boundary features are derived from punctuation marks and semantic segmentation structure. In implementation, a sequence labeling model can be used to annotate the standardized text character by character or phoneme by phoneme, generating a prosodic label sequence aligned with the phoneme sequence. This label sequence is used to represent speech rhythm and tone variations.

[0019] In one implementation, standardization is achieved through a rule engine, which includes a monetary value recognition module, a time recognition module, and a punctuation processing module to transform structured text. Phoneme conversion is achieved through a dictionary-based mapping method, combined with a statistical model to select polyphonic characters. In another implementation, standardization and phoneme conversion are implemented using an end-to-end model. The model input is the original text sequence, and the output is a phoneme sequence and intermediate normalized representations. The model structure can use an encoding network to extract features from the text and a decoding network to generate phoneme results. Domain word embeddings are also incorporated into the model to enhance its ability to handle financial terminology. In yet another implementation, prosodic feature extraction employs a multi-task learning structure. By sharing an encoding layer, tone labels, stress labels, and pause labels are simultaneously output. The model training data includes labeled prosodic information, and training is performed using a joint loss function to improve the consistency of multi-dimensional prosodic information.

[0020] This embodiment performs standardized processing on the input text and generates a corresponding phoneme sequence. At the same time, it extracts prosodic features such as tone, stress, and pause boundaries from the same text, so that the pronunciation information and rhythm information are consistent in the same time dimension. This reduces the instability of speech output caused by differences in text expression and improves the accuracy and consistency of speech expression.

[0021] S20, a phoneme embedding vector is generated based on the phoneme sequence, context encoding is performed on the phoneme embedding vector to obtain context dependencies, and gating modulation is performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features; In this embodiment, generating phoneme embedding vectors based on phoneme sequences maps discrete pronunciation units to continuous numerical representations for subsequent feature modeling. The phoneme sequence consists of multiple phoneme identifiers, each mapped to a fixed-dimensional vector using a lookup table or embedding matrix, forming a phoneme embedding vector sequence. The embedding matrix is ​​a trainable parameter matrix; its number of rows corresponds to the size of the phoneme set, and its number of columns corresponds to the embedding dimension. During training, gradient updates are used to ensure that similar pronunciation units have similar representations in the vector space. In financial text-to-speech scenarios, phoneme combinations containing product names or technical terms, covered by training data, enable the embedding vectors to express pronunciation differences and semantic relationships, thus providing basic input for subsequent contextual modeling.

[0022] Context encoding of phoneme embedding vectors involves sequence modeling of the phoneme sequence, ensuring that the representation at each position includes not only its own information but also contextual information. Context encoding is implemented using a sequence encoding network, which can employ a multi-layer self-attention structure or a bidirectional recurrent structure to map the input phoneme embedding vector sequence into a sequence of context-encoded representations. During encoding, each layer extracts features from the input through a weight matrix and nonlinear transformations, and incorporates information from preceding and following positions through attention weights or state transfer mechanisms, thus forming contextual dependencies. These dependencies are reflected in the context-encoded representation corresponding to each phoneme position, which includes information from neighboring phonemes and sentence-level structure. In fintech texts, this can demonstrate the semantic relationships between preceding and following phrases, such as in risk warning statements.

[0023] Gated modulation based on phoneme embedding vectors, context dependencies, and prosodic features jointly models pronunciation, context, and rhythm information. The gated modulation process is implemented through a dual-branch feature transformation structure: one branch performs a nonlinear transformation on the input features to generate candidate features, while the other generates gate weights to control the activation level of the candidate features. Specifically, the phoneme embedding vectors, context encoding representations, and prosodic features are concatenated along the feature dimension to form a joint input feature, which is then fed into two parallel linear transformation modules. One module outputs candidate modulation features, and the other outputs gate weight features. The gate weights are mapped to a zero-to-one range through an activation function, and element-wise modulation is applied to the candidate modulation features to obtain local prosodic features. These local prosodic features are aligned with the phoneme sequence along the time dimension, expressing fine-grained rhythmic variations at each phoneme position. In financial business texts, this processing can locally emphasize key indicator words and risk warning words based on prosodic features, enabling semantic differences in speech at the pronunciation level.

[0024] In one implementation, phoneme embedding vectors are generated via an embedding matrix lookup table, with an embedding dimension of 128 or 256. The embedding matrix is ​​trained using backpropagation. Context encoding employs a multi-layer self-attention structure, with each layer containing a query matrix, a key matrix, and a value matrix. Context information aggregation is achieved through attention weight calculation, with four to eight layers to balance modeling capability and computational complexity. In another implementation, context encoding uses a bidirectional recurrent structure, integrating context information through forward and backward state propagation, suitable for deployment environments with limited computational resources. Yet another implementation employs a dual-branch fully connected network structure for gated modulation processing. The candidate feature branch uses a non-linear activation function for mapping, while the gated branch uses an activation function to generate weights. The outputs of the two branches are multiplied element-wise to generate local prosodic features.

[0025] This embodiment jointly models phoneme embedding vectors, contextual dependencies, and prosodic features, and uses gating modulation to weight different features, enabling pronunciation information, semantic information, and rhythmic information to be expressed collaboratively in the same representation, thereby improving the rhythmic control capability and expression stability of speech at a fine-grained level.

[0026] S30, Semantically encode the style description text to obtain a global style vector, and decompose the global style vector to obtain local style components and global style components; In this embodiment, semantic encoding of the style description text maps the textual information describing tone, emotion, or expression into a continuous vector representation. This continuous vector is used to characterize the overall style attribute. Style description text is typically composed of natural language and contains semantic information such as robust expression, risk warning tone, and caring tone. Semantic encoding is implemented through a text encoding network, which consists of a lexical embedding layer, a sequence encoding layer, and a feature aggregation layer. The lexical embedding layer segments the style description text into a sequence of lexical units and maps it to a vector representation. The sequence encoding layer models the context of the lexical sequence, ensuring that each lexical representation contains sentence-level semantic information. The feature aggregation layer compresses the sequence encoding result, converting the variable-length sequence representation into a fixed-dimensional vector, resulting in a global style vector. This global style vector reflects the overall distribution characteristics of the style description text in the semantic space. In fintech scenarios, different expressions such as risk warnings, profit explanations, and asset announcements are distinguished in the vector space, giving the speech expression a clear style differentiation capability.

[0027] Decomposing the global style vector involves splitting the uniformly represented style information into sub-representations at different levels, enabling subsequent modulation of speech features at different granularities. This decomposition is achieved through a multi-branch feature extraction structure. The global style vector is input to multiple parallel mapping units, each extracting feature information of different dimensions through linear or nonlinear transformations. A portion of the mapping result serves as a local style component, representing fine-grained style changes related to local speech units, such as intensity or intonation fluctuations. The other portion serves as a global style component, representing macroscopic attributes of the overall speech expression, such as overall tone type or expressive tendency. While the local and global style components originate from the same global style vector in the numerical space, they have different expressive emphases in the feature space, thus achieving the separate expression of style information at different scales.

[0028] Semantic encoding and decomposition processes form a continuous data transformation relationship. After encoding, the style description text generates a global style vector, which is then decomposed to obtain local and global style components, transforming style information from a single representation into a multi-level representation. In the speech generation of financial business text, this process enables different style descriptions to be stably mapped into structured vectors and further distinguishes between local modulation requirements and overall expression requirements, thereby supporting consistent speech output across different business contexts.

[0029] In one implementation, the semantic encoding network employs a multi-layer self-attention structure with a word embedding dimension of 128 or 256. The sequence encoding layer models semantic relationships between words through attention weight calculation, and the feature aggregation layer generates a global style vector through weighted averaging or pooling operations. In another implementation, the semantic encoding network uses a bidirectional recurrent structure, achieving sentence-level semantic representation through the fusion of forward and backward states, suitable for computationally limited scenarios. Yet another implementation employs a dual-branch mapping structure for decomposition processing. One branch generates local style components through linear transformation, while the other branch generates global style components through nonlinear transformation. Furthermore, constraining the loss function can maintain the differences between the two branches in the feature space, thereby enhancing the decomposition effect.

[0030] This embodiment enhances the precision and consistency of style control by semantically encoding the style description text and splitting the resulting global style vector into local style components and global style components, thereby expressing style information at different granularities.

[0031] S40, Gated fusion is performed based on the local style components and the local prosodic features to obtain local fusion features, and feature linear modulation is performed on the local fusion features based on the global style components to obtain style modulation features; In this embodiment, gating fusion processing based on local style components and local prosodic features jointly models style and prosodic information on the same temporal dimension, enabling speech expression to reflect style changes at the local level. Local style components originate from the decomposition results of style description text, representing fine-grained style control information. Local prosodic features originate from phoneme-level prosodic modeling, representing pronunciation rhythm and stress distribution information. Gating fusion processing is implemented by constructing a dual-branch feature transformation structure, concatenating local style components and local prosodic features along the feature dimension to form a fused input feature. This fused input feature is input to two parallel feature transformation units. One unit generates candidate fused features, and the other unit generates gating weights. The gating weights are constrained within a numerical range by an activation function, used to adjust the response degree of the candidate fused features at different positions. Local fused features are obtained through element-wise operations. The local fused features are aligned with the phoneme sequence in the temporal dimension, ensuring that each pronunciation unit carries both style and prosodic information. In financial business texts, this processing can adjust prosodic features based on local style components, making risk warning statements exhibit more obvious tone changes at key positions while maintaining overall rhythm consistency.

[0032] Based on global style components, linear modulation of local fusion features is applied to adjust the fused local representation at the global level, ensuring the overall speech expression conforms to the expected style. Global style components represent the overall tone or expressive tendency. Through linear modulation, scaling coefficients and offsets of the same dimension as the local fusion features are generated. The scaling coefficients adjust the amplitude of the local fusion features, and the offsets translate the features, thus achieving unified control of the overall style in the feature space. Specifically, the global style components are input to the parameter generation unit, where linear transformations generate scaling and offset parameters. The local fusion features are then multiplied element-wise with the scaling parameters and added element-wise with the offset parameters to obtain the style modulation features. These style modulation features simultaneously contain fine-grained changes at the local level and unified style information at the global level, manifested in the numerical space as amplitude and offset variations of features at different locations.

[0033] In one implementation, the gated fusion processing employs a two-branch fully connected structure. The fused input features are mapped through two independent weight matrices. The candidate fusion feature branch uses a non-linear activation function to enhance expressiveness, while the gated weight branch uses an activation function to generate weight coefficients. The two are then processed element-wise to obtain local fusion features. In another implementation, the gated fusion processing uses a convolutional structure. One-dimensional convolution is used to extract features from the time series, resulting in a smoother fusion relationship between adjacent phonemes, suitable for continuous speech expression scenarios. In yet another implementation, the feature linear modulation processing uses a parameter generation network. Global style components are input to a multi-layer fully connected network to generate scaling and offset parameters. The number of network layers can be set to two or three to balance expressiveness and computational cost.

[0034] This embodiment achieves improved fine-grained adjustment capability and overall consistency of speech expression by gating and fusing local style components and local prosodic features, and combining them with global style components for linear modulation. This allows local speech expression and overall tone control to be synergistically reflected in the same feature representation.

[0035] S50, the style modulation features are predicted for duration to obtain pronunciation duration, and the style modulation features are extended based on the pronunciation duration to obtain extended features; In this embodiment, duration prediction of style modulation features corresponds to generating time distribution information related to pronunciation length based on feature sequences containing style and prosodic information. Pronunciation duration is used to represent the duration of each phoneme on the time axis. Style modulation features are aligned with the phoneme sequence in the time dimension, and the features at each position contain pronunciation content, contextual relationships, and style modulation information. Duration prediction processing is implemented through a duration modeling network, which receives the style modulation feature sequence as input and obtains a sequence of predicted values ​​related to time length through multi-layer feature transformation. After numerical constraint processing, the pronunciation duration is obtained, which is a positive integer or discrete time unit, representing the number of frames the corresponding phoneme lasts in the speech output. In financial business texts, different tone types affect the duration distribution. For example, the duration of key prompts in risk warning statements is relatively long, while the duration of ordinary explanatory statements is relatively uniform. Duration prediction processing learns this distribution difference to adjust the speech rhythm.

[0036] The duration modeling network can consist of a multi-layer feedforward structure or a sequence modeling structure. Input features are processed through linear transformations and non-linear activation functions, and the output is a duration prediction value with the same length as the input sequence. In implementation, the predicted values ​​can be exponentially mapped or truncated to satisfy the non-negativity constraint of the duration. Simultaneously, a pronunciation duration sequence is generated through rounding or discretization operations. The pronunciation duration corresponds one-to-one with the phoneme sequence, forming a time alignment relationship to guide subsequent feature expansion.

[0037] Sequence expansion of style modulation features based on pronunciation duration involves unfolding the original feature sequence along the time dimension, aligning the feature sequence with the time frame number. Sequence expansion is achieved by constructing a mapping relationship that assigns a corresponding number of repetitions to each phoneme position based on pronunciation duration. Each position in the style modulation feature sequence is copied and unfolded according to pronunciation duration, generating a longer expanded feature sequence. The expanded features are represented as a continuous frame sequence along the time dimension, with each frame inheriting the feature information of the corresponding phoneme position. In implementation, feature expansion can be achieved through indexed unfolding or repetition operations. For example, an indexed sequence can be constructed based on pronunciation duration, mapping the original features to new timeline positions to obtain the expanded features.

[0038] In one implementation, duration prediction processing employs a multi-layer feedforward network structure. The input is a style modulation feature sequence, and the output is a predicted value with the same length as the sequence. This is mapped using a non-linear activation function and rounded to obtain the pronunciation duration. In another implementation, duration prediction processing uses a sequence modeling structure, predicting duration based on contextual information to smooth the duration distribution between adjacent phonemes. In yet another implementation, sequence expansion processing is implemented using repeated indexing. An index sequence is generated based on the pronunciation duration, and the original features are rearranged and copied using the index to obtain an expanded feature sequence. Intermediate frame features can also be generated through interpolation to improve temporal resolution.

[0039] This implementation performs duration prediction on style modulation features and performs sequence expansion accordingly, aligning the feature sequence with the speech frame in the time dimension. This allows for adjustment of pronunciation duration based on tone and semantic differences, thereby improving the accuracy and consistency of speech rhythm expression.

[0040] S60, the extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features; In this embodiment, mapping extended features to the acoustic latent space corresponds to converting the time-frame-aligned feature sequence into an implicit representation space suitable for acoustic modeling. The acoustic latent space is used to express the spectral structure and acoustic properties of speech. Extended features include pronunciation content, contextual relationships, style information, and temporal distribution information. This feature sequence is already aligned with speech frames in the temporal dimension, but it is still in the text feature space and needs to be converted into an acoustic representation through a mapping process. The mapping process is implemented through a latent space coding network, which consists of multiple layers of feature transformation units. Each layer transforms the input features through a weight matrix and a non-linear activation function, gradually approximating the acoustic distribution. During the encoding process, the input features are compressed or reconstructed layer by layer, forming latent space features in the feature space. The latent space features have a lower dimension or a constrained distribution, used to represent the acoustic properties of speech.

[0041] Latent space coding networks can be implemented using variational or streaming transform structures. In a variational structure, the coding network outputs latent variable distribution parameters and generates latent space features through sampling operations, ensuring the feature distribution is continuous and sampleable. In a streaming transform structure, a series of invertible mappings transform the input features into latent space representations, ensuring the feature distribution satisfies preset constraints. The latent space features maintain consistency with the extended features in the temporal dimension, with each frame corresponding to a latent space representation used for subsequent acoustic generation.

[0042] Acoustic decoding based on latent space features involves converting the latent space representation into acoustic features. These acoustic features describe the spectral structure of the speech signal, such as the spectral amplitude distribution or acoustic parameter sequences. Acoustic decoding is implemented through a decoding network consisting of multiple layers of sequence modeling units and upsampling units. The sequence modeling units recover the temporal dependencies in the latent space features, while the upsampling units improve the temporal or frequency resolution to ensure the output features meet acoustic representation requirements. During decoding, the latent space features undergo layer-by-layer transformation to generate an acoustic feature sequence. This sequence is consistent with the speech frames in the temporal dimension and contains acoustic information that can be used for waveform generation.

[0043] Acoustic decoding networks can be implemented using self-attention or convolutional structures. Self-attention structures are used to capture long-range dependencies, while convolutional structures are used to enhance local continuity. In fintech voice applications, this processing can convert features containing style and rhythm information into stable acoustic representations, ensuring that risk warning statements and asset announcement statements have consistent expressive characteristics at the acoustic level.

[0044] In one implementation, the latent space encoding network employs a variational structure. The input is an extended feature sequence, and the encoding network outputs latent variable distribution parameters. Latent space features are generated through sampling operations, and the decoding network receives these features and recovers temporal dependencies through a multi-layer self-attention structure, ultimately outputting acoustic features. In another implementation, the latent space encoding uses a streaming transform structure, converting extended features into latent space features through multi-layer invertible mappings. The decoding network uses a convolutional structure for feature recovery to reduce computational complexity. In yet another implementation, the acoustic decoding network employs a hybrid structure, combining self-attention units with convolutional units to simultaneously model long-range dependencies and local continuity.

[0045] This embodiment improves the stability and consistency of acoustic representation by mapping extended features to the acoustic latent space and performing acoustic decoding, thereby converting text features into acoustic features in a constrained representation space.

[0046] S70, Based on the acoustic features, perform waveform synthesis processing to obtain a speech waveform.

[0047] In this embodiment, waveform synthesis processing based on acoustic features converts the spectral representation into a time-domain speech signal, which is then used for direct playback or transmission. The acoustic features are aligned with the speech frames in the time dimension, representing the spectral structure or acoustic parameter information of each frame. This feature sequence includes timbre, frequency distribution, and energy variation information. Waveform synthesis processing is implemented through a vocoder network, which receives the acoustic features as input and generates a one-dimensional time-domain signal through multi-layer feature transformation.

[0048] The vocoder network consists of a feature upsampling unit, a sequence modeling unit, and a signal generation unit. The feature upsampling unit extends the acoustic features in the time dimension, gradually bringing the sampling rate of the feature sequence closer to the target speech sampling rate. The sequence modeling unit models the local and global relationships of the extended features to ensure the continuity of the generated signal in the time dimension. The signal generation unit maps the features into a sequence of waveform amplitude values ​​through nonlinear transformations, forming the speech waveform. In implementation, the upsampling unit can use a deconvolutional or interpolation structure, the sequence modeling unit can use a convolutional or attention structure, and the signal generation unit generates the waveform amplitude by calculating it point-by-point.

[0049] Waveform synthesis processing also includes constraining the generated signal to ensure the output signal meets audio playback requirements. Constraint methods include amplitude range limiting and smoothing to ensure the speech waveform is continuous and abrupt in the time dimension. In fintech voice scenarios, this processing can stably convert acoustic features into speech waveforms, ensuring consistency in sound quality and tone between risk warning statements and asset announcements.

[0050] In one implementation, the vocoder network employs an adversarial training-based structure, comprising a generator network and a discriminator network. The generator network receives acoustic features and outputs a speech waveform, while the discriminator network evaluates the authenticity of the generated waveform. Adversarial training improves waveform quality. In another implementation, the vocoder network uses an autoregressive model, generating waveform sampling points progressively to achieve high-precision speech synthesis, suitable for scenarios with high sound quality requirements. Yet another implementation employs a non-autoregressive structure, improving inference speed through parallel generation, suitable for real-time broadcasting needs in fintech applications.

[0051] This embodiment converts acoustic features into speech waveforms, enabling the continuous expression of spectral information in the time domain, thereby improving the naturalness and stability of speech output.

[0052] In one embodiment, step S10 above includes: S101, Receive input text and style description text; S102, perform text cleaning on the input text to obtain cleaned text, and perform punctuation analysis and sentence segmentation on the cleaned text to obtain sentence segmented text; S103, perform language type identification and annotation on the segmented text to obtain standardized text; S104, Based on the language type corresponding to the standardized text, perform phoneme conversion processing on the standardized text to obtain a phoneme sequence; S105, Based on the language type, extract tone features and stress features from the standardized text; S106, Based on the structure and punctuation information of the standardized text, extract pause boundary features from the standardized text; S107, combine the tone features, the stress features, and the pause boundary features to obtain the prosodic features of the standardized text.

[0053] In this embodiment, two types of text data are received synchronously: input text and style description text. The input text carries the content information of the speech to be generated, while the style description text carries style information such as tone, emotion, and expression. The current processing revolves around the input text, aiming to convert the original text into standardized text, phoneme sequences, and prosodic features that can be directly used in subsequent speech generation. After the input text enters the processing unit, it undergoes text cleaning. The cleaning process is used to eliminate invalid components that affect text standardization and pronunciation mapping. Specifically, it includes removing repeated spaces, abnormal control characters, format marks without pronunciation value, residual symbols from web scraping, non-semantic decorative characters, and noise fragments unrelated to the context, while retaining valid punctuation related to pauses, sentence boundaries, and tone expression. The result of the cleaning process is the cleaned text, which has a more stable character distribution and clearer semantic boundaries compared to the original input, facilitating subsequent sentence segmentation and language recognition. For profit announcement text, risk warning text, and transaction notification text in fintech businesses, the cleaning process can also eliminate invalid symbols interspersed before and after amounts, abnormal intervals in interest rate expressions, and non-standard separators in date fields, making the text format uniform. For health reminder texts, medication notification texts, and indicator interpretation texts in healthcare services, cleaning can remove redundant field markers and irrelevant abbreviations from the data uploaded by the testing equipment, thus avoiding interference with subsequent pronunciation conversion.

[0054] After cleaning, the text undergoes punctuation analysis and sentence segmentation. Punctuation analysis identifies the role of structural symbols such as periods, commas, semicolons, pauses, question marks, exclamation marks, parentheses, and quotation marks in semantic organization. Sentence segmentation is not simply character-by-character cutting, but rather determines sentence boundaries by combining punctuation positions, keyword distribution, and local semantic continuity. This segmented text provides clearer contextual units for subsequent language type identification and annotation. The significance of sentence segmentation lies in dividing the original long text into semantically relatively complete segments, avoiding phoneme conversion and prosodic extraction from crossing semantic boundaries that should not be continuous. In fintech business texts, a complete account announcement may simultaneously include balance reminders, profit descriptions, and risk disclosures; sentence segmentation can distinguish these three types of segments, ensuring each segment maintains an independent semantic scope. In healthcare business texts, health advice, abnormal indicator alerts, and lifestyle tips can also be segmented into independent semantic units, thereby improving the stability of subsequent processing.

[0055] After sentence segmentation, the text enters the language type identification and annotation process. Language type identification determines whether each text segment or its local components belong to Chinese, English, colloquial numerical expressions, financial abbreviations, medical abbreviations, or a mixed expression. Language type annotation adds language category labels to each text unit based on the identification results, enabling subsequent phoneme conversion to call matching pronunciation rules and mapping tables. Language type identification and annotation can be implemented through a rule-based system or a machine learning model. When using a rule-based system, the determination can be based on character sets, case distribution, number formats, and common terminology lists. When using a model, a character-level sequence labeling network can be constructed. The network input is the character sequence of the segmented text. The embedding layer maps characters to vector representations, the encoding layer uses bidirectional recurrent units or self-attention structures to extract context, and the classification layer outputs the language category label for each character or segment. During training, manually labeled language type sequences are used as supervision signals, and parameters are updated using cross-entropy loss. The model training data can come from financial broadcast corpora, medical reminder corpora, and general multilingual texts. The input is a text sequence with sentence structure, and the output is the corresponding language label sequence. Standardized text is obtained after language type identification and annotation. This standardized text is not only the text result after unified formatting, but also contains structured markup information corresponding to the language category, so that the text form and language attributes are stabilized at the same time.

[0056] Based on the language type corresponding to the standardized text, phoneme conversion processing is performed on the standardized text. Phoneme conversion processing transforms the character-level text representation into a phoneme sequence at the pronunciation level. A phoneme sequence consists of a set of minimal speech units capable of distinguishing pronunciation, used to express the pronunciation structure of words in speech. Phoneme conversion processing can be implemented using either regular word mapping or neural networks. Regular word mapping typically includes hierarchical mapping tables from words to pinyin, pinyin to phonemes, and English words to IPA symbols or phoneme symbols, selecting the mapping path based on the language type label. Neural network processing takes standardized text as input and outputs the corresponding phoneme label sequence. The network can employ an encoding-decoding structure or a sequence labeling structure. The embedding layer receives character or sub-word representations, the encoding layer extracts contextual semantics, and the output layer generates phoneme categories. In fintech businesses, product names, institution abbreviations, English codes, and interest rate abbreviations often have specific pronunciations. Phoneme conversion processing can be combined with domain-specific vocabulary lists or domain-specific fine-tuning models to enhance the consistency of terminology pronunciation. In healthcare applications, indicator names, drug names, and test abbreviations often have complex pronunciations. Phoneme conversion processing can be adapted using specialized dictionaries and supervised corpora to avoid homonyms or mispronunciations. After phoneme conversion, a phoneme sequence is generated. This phoneme sequence maintains a sequential mapping relationship with the standardized text, providing a unified unit of pronunciation for subsequent prosodic alignment and speech modeling.

[0057] Prosodic feature extraction revolves around standardized text, extracting features including tone features, stress features, and pause boundary features. Tone features reflect pitch changes in the pronunciation of words or syllables, particularly useful for distinguishing pronunciations in tonal languages; stress features reflect the degree of emphasis of words or syllables within a sentence; pause boundary features reflect the position and intensity of pauses in the speech flow. Tone feature extraction can be based on language type, pinyin mapping results, and dictionary rules, or it can be predicted using annotation models. Stress feature extraction can be determined based on keyword recognition, part-of-speech distribution, syntactic position, and semantic weight. In fintech business texts, keywords such as risk level, return fluctuation, default warning, and credit limit change often carry high stress weight; in healthcare business texts, words such as abnormal indicators, medication frequency, follow-up examination time, and contraindication reminders also require higher emphasis. Pause boundary feature extraction is based on the structure and punctuation information of the standardized text; periods, commas, semicolons, parentheses, and parallel relationships can all form pause boundaries of different levels. The pause boundaries are not determined solely by punctuation; they can also be corrected by considering sentence segmentation and local semantic integrity, thereby avoiding the impact of missing or misused punctuation on speech rhythm.

[0058] To achieve higher accuracy in prosodic feature extraction, a sequence labeling model can be used to uniformly output tone features, stress features, and pause boundary features. The model input consists of standardized text and its language type markers. The character embedding layer maps text units to vectors, and the language type embeddings are concatenated with the character embeddings before entering the shared encoding layer. The shared encoding layer can employ a multi-layer bidirectional gated recurrent network or a multi-layer self-attention network to extract local and global contextual representations from the text. Three parallel output heads are set after the shared encoding layer, corresponding to tone feature prediction, stress feature prediction, and pause boundary feature prediction, respectively. Each output head generates a set of label sequences aligned with the text position. During training, a multi-task loss joint optimization is used, where each task uses labeled data to calculate the classification loss, and parameter updates employ mini-batch gradient descent or an adaptive optimizer. The training data can consist of financial broadcast text and medical reminder text. The input is standardized text and language markers, and the output is three types of prosodic labels. The batch size, learning rate, and training epochs are set according to the corpus size. After model inference or rule system output, tone features, stress features, and pause boundary features are combined into prosodic features of the standardized text. The combination process is not a simple splicing, but rather a unified organization of the three types of features according to text position or phoneme position, forming an integrated prosodic representation with temporal order. The resulting prosodic features maintain an alignment relationship with the phoneme sequence, preserving both pitch variations at the pronunciation level and emphasis and pause information at the word / sentence level.

[0059] This embodiment transforms complex and inconsistent original text into structurally stable standardized text by performing text cleaning, punctuation analysis and sentence segmentation, and language type identification and annotation on the input text. By performing phoneme conversion based on the standardized text, a phoneme sequence corresponding to the text content can be obtained. Furthermore, by extracting and combining tone features, stress features, and pause boundary features from the standardized text, prosodic features are obtained, enabling the simultaneous structured representation of pronunciation and rhythm information. Because the standardized text, phoneme sequence, and prosodic features maintain a consistent order, the pronunciation units and prosodic units in the subsequent speech generation process can be expressed synchronously, thereby improving the consistency of pronunciation, rhythmic stability, and semantic emphasis of text expression in different business contexts.

[0060] In one embodiment, step S20 above includes: S201, Perform embedding mapping processing on the phoneme sequence to obtain a phoneme embedding vector; S202, Perform multi-layer feedforward coding on the phoneme embedding vector to obtain the context dependency; S203, Perform feature concatenation processing on the phoneme embedding vector, the context dependency, and the prosodic features to obtain gated input features; S204, Perform a first feature transformation and a second feature transformation on the gated input features to obtain a first transformed feature and a second transformed feature; S205, perform nonlinear activation processing on the first transformation feature to obtain candidate modulation features; S206, Perform gated activation processing on the second transformation feature to obtain gated weight features; S207, perform element-wise weighted modulation on the candidate modulation features and the gate weight features to obtain the gated output features; S208, perform feature mapping on the gated output features to obtain local prosodic features.

[0061] In this embodiment, after the phoneme sequence enters the embedding mapping process, it is converted into a continuous vector representation suitable for neural network computation. Each phoneme unit in the phoneme sequence corresponds to an index position in the embedding matrix. The embedding matrix maps discrete phoneme identifiers to real-valued vectors of fixed dimensions, and the output constitutes a phoneme embedding vector sequence. The phoneme embedding vectors preserve the differences in pronunciation categories and convert symbol sequences that could not be directly involved in weighted operations into trainable representations. The embedding dimension can be set according to the corpus size, phoneme set size, and deployment resources. In the financial technology voice broadcasting scenario, the pronunciation differences of terms such as interest rates, funds, accounts, and risk control can be stably distinguished by the embedding matrix trained on the domain corpus. In the medical and health voice reminder scenario, the phoneme combinations of indicator names, drug names, and examination items can also be formed into specialized representations in the same way. The phoneme embedding vectors output by the embedding mapping process are strictly aligned with the phoneme sequence in time order, and each position subsequently participates in context encoding and gating modulation operations.

[0062] Multilayer feedforward coding is used to extract contextual dependencies from phoneme embedding vectors. These contextual dependencies are not static representations of individual phonemes, but rather representations of information about the mutual influence between adjacent phonemes and even longer-distance phonemes. In implementation, multilayer feedforward coding can be composed of a combination of multilayer feedforward transformation units and sequence information interaction units. Each layer receives the phoneme embedding vector sequence output from the previous layer, performs linear transformation, normalization, activation, and residual connections to form a new sequence representation, and gradually incorporates context between layers. If an attention-based structure is used, the network calculates the correlation weights between different positions in each layer, enabling the current phoneme position to access the pronunciation environment information of the preceding and following positions; if a feedforward temporal convolutional structure is used, it captures the sequential relationships within the neighborhood by expanding the receptive field. During training, the input is the phoneme embedding vector sequence and its aligned supervised acoustic or prosodic labels, and the output is the contextual dependencies. The loss function can be jointly optimized with subsequent local prosodic feature prediction tasks. The number of layers, hidden dimensions, number of heads, or convolutional kernel range can be adjusted according to the text complexity. In fintech texts, risk warning statements often contain multiple parallel paragraphs and conditional constraints, making the contextual influence between phonemes more pronounced. In healthcare texts, descriptions of medication frequency, timing, and indicator thresholds often contain closely adjacent combinations of numerical values ​​and nouns. Multi-layer feedforward coding can improve the contextual recognition ability of such structures.

[0063] Phoneme embedding vectors, context dependencies, and prosodic features are processed through feature concatenation to form unified gated input features. Phoneme embedding vectors provide the basic representation of the articulatory units, context dependencies provide local and global semantic environment information, and prosodic features provide information related to tone, stress, and pauses. Feature concatenation connects these three types of information along the feature dimension at each time point, generating a new high-dimensional representation. Instead of simple summation, concatenation is used to avoid premature compression of features from different sources before they enter the gating unit, allowing subsequent transformation units to learn the operational patterns of articulatory, contextual, and prosodic information separately. The concatenated gated input features still correspond one-to-one with phoneme positions, maintaining the temporal order. If the training data comes from financial technology voice broadcast text, the gated input features will simultaneously reflect the phonetic representation of terms, business semantic constraints in the context, and stress and pause requirements at risk warning locations; if the training data comes from medical and health reminder text, the gated input features will simultaneously reflect the phonetic representation of drug names, reminder semantics in the context, and rhythmic requirements at medication time or abnormal indicator locations.

[0064] After the gated input features enter the first and second feature transformation processes, they form two parallel branches. The first feature transformation process focuses on generating candidate modulation features, while the second feature transformation process focuses on generating gated weight features. Both branches can be implemented by independent linear transformation layers or one-dimensional convolutional layers, each using a different weight matrix to ensure that the network can learn two different types of projective representations. The first branch focuses on mapping the gated input features to content candidate representations, preserving effective combinations of pronunciation and prosody; the second branch focuses on mapping the gated input features to position-related control weights, enabling the network to determine how much candidate information should be retained at a given time position. The output dimensions of both branches remain consistent for subsequent element-wise modulation. This parallel branch design ensures that the same input undergoes functional separation before entering nonlinear activation and gated activation, thus avoiding a single path simultaneously handling both content representation and weight control tasks and reducing representational interference.

[0065] The first transform feature, after nonlinear activation processing, yields candidate modulation features. The purpose of nonlinear activation is to introduce nonlinear expressive power, enabling the network to learn complex nonlinear combinations of phoneme embedding vectors, contextual dependencies, and prosodic features. Candidate modulation features can be understood as local prosodic candidate representations of the current position before gating suppression, containing comprehensive information such as vocal intensity, rhythmic fluctuations, and local expressive tendencies. The second transform feature, after gating activation processing, yields gating weight features. Gating activation typically constrains the output to the interval between zero and one or other restricted intervals, expressing how much candidate information should be opened or suppressed at the current position. The candidate modulation features and gating weight features are aligned in both the temporal and feature dimensions, facilitating element-wise modulation. At this point, the network does not uniformly weight the entire sequence but generates fine-grained control at each phoneme position and each feature channel. In fintech newsletters, key positions involving amounts, yields, terms, and risk levels will develop higher-response gating weights after training; in healthcare reminders, positions involving dosage, time points, abnormal indicators, and precautions will also form different gating distributions.

[0066] After element-wise weighted modulation of candidate modulation features and gated weight features, gated output features are obtained. Element-wise modulation processing performs multiplicative control at each time position and each feature channel. The gated weight features determine the proportion of candidate modulation features retained, thereby suppressing irrelevant or weakly relevant information and enhancing important pronunciation and rhythmic information. The gated output features then enter feature mapping processing to output local prosodic features. Feature mapping processing is used to transform the gated output features to a unified dimensional space required by subsequent modules, ensuring that the local prosodic features retain both the content after gated modulation and meet the input format requirements of subsequent style fusion units. Feature mapping processing can be implemented using a combination of linear mapping layers, normalization layers, and nonlinear activation layers, or one-dimensional convolutional projection can be used to preserve the smoothness of adjacent time positions. After the local prosodic features are generated, they remain consistent with the phoneme sequence in the temporal dimension, with each position representing the local expression state of the phoneme under the current context and prosodic conditions.

[0067] Phoneme sequences provide phonetic unit indices, phoneme embedding vectors map discrete symbols to continuous representations, multi-layer feedforward encoding extracts contextual dependencies, feature concatenation organizes three types of information into a unified input, bi-branch feature transformation generates candidate content and gating weights respectively, non-linear activation and gating activation form modulated dual-path representations, element-wise modulation controls the degree of information retention at different positions, and feature mapping outputs local prosodic features. If implemented using model training, this part of the network takes phoneme sequences and prosodic features aligned with phoneme positions as input and outputs local prosodic features. The training objective can be jointly calculated with the target prosodic representation, acoustic feature representation, or subsequent task loss. The model can be trained in stages or jointly trained end-to-end with the subsequent acoustic modeling part. Training parameters include embedding dimension, number of encoding layers, hidden unit dimension, gating branch dimension, learning rate, batch size, and iteration rounds. Parameter values ​​are determined by the size of the training corpus, domain complexity, and device computing power constraints.

[0068] For example, the local prosodic coding formula is:

[0069] in, This represents the local prosodic feature corresponding to the position of the i-th phoneme. GTU() represents the gated modulation unit. This represents the phoneme embedding vector of the i-th phoneme. It indicates the contextual dependency relationship formed by the preceding and following phonemes. It represents the phonemes preceding the i-th phoneme. This represents the phonemes following the i-th phoneme. i represents the phoneme position index.

[0070] This embodiment converts phoneme sequences into phoneme embedding vectors, extracts contextual dependencies through multi-layer feedforward coding, and then constructs gated input features by combining prosodic features. This enables the simultaneous expression of content, context, and rhythmic information at the phoneme unit level. By performing bi-branch transformation, nonlinear activation, gated activation, and element-wise modulation on the gated input features, it can finely control the preservation and suppression of local information according to temporal position and feature channel. By performing feature mapping on the gated output features, it can output local prosodic features aligned with phoneme positions. The resulting local prosodic features have stronger fine-grained expressive capabilities, improving the tone discrimination, rhythmic stability, and pronunciation consistency of key word positions in both fintech and healthcare texts.

[0071] In one embodiment, step S30 above includes: S301, perform word segmentation on the style description text to obtain a style word sequence, perform word embedding on the style word sequence to obtain a word representation sequence, and encode the word representation sequence through a pre-trained language model to obtain a sentence-level semantic representation; S302, Perform feature aggregation and normalization on the sentence-level semantic representation to obtain a global style vector; S303, perform first branch feature extraction on the global style vector to obtain the first decomposition sub-vector; S304, perform second branch feature extraction on the global style vector to obtain the second decomposition sub-vector; S305, perform linear mapping processing on the first decomposed subvector to obtain the local style component, and generate a global style component based on the second decomposed subvector.

[0072] In this embodiment, after the style description text enters the semantic encoding process, it no longer participates in subsequent calculations in its original character form. Instead, it first undergoes word segmentation to form a style word sequence. Word segmentation is used to break down continuous text into the smallest computable semantic units. The segmentation granularity can be character-level, sub-word-level, or word-level. Character-level segmentation is suitable for short text style descriptions, sub-word-level segmentation is suitable for mixed Chinese and English expressions, and word-level segmentation is suitable for long text expressions with relatively stable style semantics. In fintech businesses, style description text may contain expressions such as a stable tone, a risk warning tone, a profit announcement tone, or a customer service explanation tone. The segmentation process needs to preserve the combination relationship between financial keywords and tone modifiers. In healthcare businesses, style description text may contain expressions such as a caring tone, a reminder tone, a reassuring tone, or an anomaly notification tone. The segmentation process needs to avoid breaking down descriptive phrases to prevent semantic dilution. After the style word sequence is formed, each word carries a clear positional order and semantic boundary, providing a unified input format for subsequent vectorized representation.

[0073] After entering the lexical embedding process, the style lexical sequence is converted into a lexical representation sequence. The lexical embedding process maps discrete lexical units to continuous vectors using an embedding matrix. The number of rows in the embedding matrix corresponds to the size of the lexical vocabulary, and the number of columns corresponds to the embedding dimension. The embedding dimension can be set to 128, 256, or higher, depending on the corpus size and style expression complexity. During training, the embedding matrix is ​​updated via gradients, ensuring that lexical units with similar semantics and style expressions form a distinguishable yet locally clustered distribution in the vector space. In the fintech field, lexical units such as risk, compliance, return, volatility, and stability can form clear style clusters after training. Similarly, in the healthcare field, lexical units such as reassurance, abnormality, follow-up examination, medication, and monitoring will form stable representations. The lexical representation sequence preserves the sequential information and local semantic differences in the text, upon which subsequent language model encoding processes extract higher-level sentence-level semantic representations.

[0074] Language model encoding processing is used to convert lexical representation sequences into sentence-level semantic representations that include contextual dependencies. Language model encoding processing can employ pre-trained language models or sequence models primarily based on self-attention encoding layers. If a pre-trained language model is used, the encoding network typically consists of embedding layers, positional encoding layers, and multiple encoding blocks. Each encoding block contains multi-head self-attention units, feedforward transformation units, normalization units, and residual connection units. Multi-head self-attention units weight the correlation between lexical units across different attention heads, allowing words like "risk," "warning," and "attention" in risk warning to reinforce each other, and words like "gentle," "relaxed," and "recovering" in reassuring to be grouped together. Feedforward transformation units enhance expressive power through nonlinear mapping, normalization units maintain stable numerical distributions, and residual connections mitigate information decay during deep network training. If a specialized industry model is used, financial technology corpora containing financial management instructions, investment tips, risk disclosures, and transaction notifications can serve as style semantic training samples, while healthcare corpora containing chronic disease reminders, rehabilitation suggestions, follow-up reminders, and indicator explanations can serve as style semantic training samples. During training, the input is the sequence of lexical representations corresponding to the style description text, and the output is the style category label, style similarity label, or style embedding target. The loss function can be cross-entropy loss, contrastive loss, or multi-task joint loss. The optimizer can use adaptive gradient optimization. The batch size, learning rate, and number of training epochs are set according to the data scale and hardware resources. After language model encoding, the sentence-level semantic representation is no longer just a word-by-word local representation, but a high-order representation set that has absorbed the semantics of the entire sentence, tone, and modification relationships.

[0075] After sentence-level semantic representations undergo feature aggregation and normalization, a global style vector is output. Feature aggregation compresses variable-length sentence-level semantic representations into fixed-length vectors, facilitating subsequent unified decomposition. Aggregation methods can include average pooling, weighted pooling, attention pooling, or vector extraction at specific locations. Average pooling is suitable for uniformly distributed style descriptions, weighted pooling for descriptions with core style words, and attention pooling for long sentences with complex style modification relationships. Normalization unifies the scale and calibrates the distribution of the aggregated vectors, ensuring that style vectors from texts of different lengths and scenarios are within a comparable numerical range. The global style vector formed after aggregation and normalization carries overall style semantic information, expressing style categories such as robustness, formality, warning, and care, while also reflecting tone strength, emotional tension, and expressive tendencies. In financial technology voice, the global style vector formed from risk disclosure text will show a clear separation from ordinary broadcast text in the vector space; in medical and health voice, the global style vectors formed from reassuring and reminding tones will also show different clustering regions.

[0076] After the global style vector undergoes decomposition, it is split into local style components and global style components. Instead of simple replication, the decomposition process uses two parallel feature extraction branches to extract style information at different levels. The first branch focuses on preserving fine-grained style differences, outputting the first decomposed sub-vector. This first sub-vector emphasizes local adjustability, suitable for expressing the control needs of stress, local emotional intensity, and intonation fluctuations in local sentences. The second branch focuses on preserving the overall style bias, outputting the second decomposed sub-vector. This second sub-vector emphasizes global consistency, suitable for expressing the overall attributes of the entire speech in terms of formality, friendliness, alertness, and emotional stability. The two branches can use independent linear transformation matrices, or different projectors can be introduced after sharing a base network. During training, additional branch difference constraints can be introduced to maintain a proper separation between the two types of sub-vectors in terms of content, avoiding the overlap of local adjustment information and global control information. Constraints can take the form of orthogonal constraints, mutual information inhibition constraints, or correlation penalty constraints. The resulting two types of decomposed sub-vectors are more conducive to subsequent application to local and global expressions, respectively.

[0077] The first decomposed subvector is processed by linear mapping to obtain local style components. Linear mapping projects the first decomposed subvector into a dimensional space that matches the subsequent local speech unit modulation. The local style components preserve locally controllable style information, such as tension near warning words, softness in caring statements, and smoothness in explanatory statements. The role of linear mapping is not to add new semantics, but to organize the separated local style information into numerical representations that can be directly invoked later. If the input text contains risk warning sentences, the local style components can enhance the local expression differences at key informational locations; if the input text contains health reminder sentences, the local style components can enhance the prominence of the reminder positions.

[0078] The second sub-vector is used to generate global style components. Global style components represent the unified style attributes of the entire speech segment, numerically corresponding to overall tone, emotional tendency, expressive intensity, and style consistency. The generation process can employ identity transitivity or lightweight mapping units for scale calibration or dimensionality adjustment. If mapping units are used, the mapping depth is typically kept shallow to avoid excessive reconstruction of global information. Once formed, the global style components, together with the local style components, constitute a two-layer style representation structure. Local style components express subtle changes at local locations, while global style components control the overall style of the entire speech segment. In fintech, risk disclosure statements, profit explanation statements, and transaction confirmation statements can each form different global style components; in healthcare, reassuring reminder statements, anomaly notification statements, and rehabilitation suggestion statements also form globally style components with clear distinctions. After this processing, the style description text no longer corresponds to a single vector but is split into a two-component structure suitable for different expressive scales, laying the data foundation for subsequent style modulation.

[0079] For example, the formula for global style vector decomposition is:

[0080] in, This represents the global style vector obtained by encoding the style description text. This represents the local style components obtained from the decomposition of the global style vector. This represents the global style component obtained from the decomposition of the global style vector. This indicates a decomposition or splitting relationship. [,] indicates splitting the global style vector into two sub-vectors.

[0081] This embodiment transforms the original style description text into a sentence-level semantic representation containing contextual semantic relationships by performing lexical segmentation, lexical embedding, and language model encoding. Through feature aggregation and normalization of the sentence-level semantic representation, a global style vector at a unified scale is formed. By performing a bi-branch decomposition on the global style vector and generating local and global style components respectively, style information is separated from a single representation into fine-grained control information and overall control information. Thus, the style description text forms a clearly structured and hierarchical representation in the numerical space, enabling subsequent speech expression processes to simultaneously achieve local adjustability and overall consistency control, thereby improving style discriminability, expression stability, and semantic adaptability in financial technology and healthcare speech.

[0082] In one embodiment, step S40 above includes: S401, Perform a first projection mapping on the local style component to obtain style projection features, perform a second projection mapping on the local rhythmic features to obtain a first rhythmic projection feature, and perform a third projection mapping on the local rhythmic features to obtain a second rhythmic projection feature. S402, the style projection feature and the first rhythm projection feature are added together to obtain the superimposed intermediate feature, and the superimposed intermediate feature is activated by the tangent function to obtain the tangent activation feature; S403, perform gate function activation processing on the second prosodic projection feature to obtain gated activation feature, and multiply the tangent activation feature and the gated activation feature element-wise to obtain local fusion feature; S404, Perform scaling parameter generation processing and offset parameter generation processing on the global style component to obtain scaling parameters and offset parameters; S405, perform feature multiplication on the local fusion feature and the scaling parameter to obtain preliminary modulation feature, and add the preliminary modulation feature and the offset parameter to obtain style modulation feature.

[0083] In this embodiment, before the local style components and local prosodic features enter the gating fusion process, projection mapping is performed separately. The first projection mapping process is applied to the local style components, outputting style projection features. The second and third projection mapping processes are applied to the local prosodic features, outputting first and second prosodic projection features, respectively. Using three-way projection instead of directly adding the local style components and local prosodic features aims to first achieve dimensional unification and functional separation of representations from different sources. The local style components carry fine-grained style control information, while the local prosodic features carry phoneme-level rhythm, accent, and pause information. These two types of representations differ in statistical distribution, feature scale, and semantic emphasis; direct mixing would cause interference between features from different sources in the same space. Through the first projection mapping process, the local style components are transformed into a space matching the local fusion computation; through the second projection mapping process, the local prosodic features are mapped to the first prosodic projection features participating in additive fusion; and through the third projection mapping process, the local prosodic features are mapped to the second prosodic projection features participating in gating control. The resulting three-way output maintains consistency with the original sequence in the time dimension and achieves alignment in the feature dimension, providing conditions for subsequent element-wise operations. In fintech voice communication, texts such as risk warnings, profit announcements, and transaction confirmations often require varying intensities of expression at local locations. Projection mapping can transform the risk level, formality, or prompt intensity in local style components into numerical representations that can participate in fusion calculations, while preserving the stress and pause constraints in local prosodic features. In medical and health voice communication, the local style components corresponding to reminder tones, reassuring tones, and informing tones can also be converted into locally controllable representations in the same way and aligned with local prosodic features at locations such as medication frequency, abnormal indicators, and follow-up examination times.

[0084] Style projection features and first prosodic projection features are added together to obtain superimposed intermediate features. Feature addition represents the additive combination of two vector sequences of the same dimension at corresponding positions, used to directly inject local style information into the main prosodic representation. Style projection features provide local style bias, and first prosodic projection features provide local articulation rhythm bias; their addition forms a superimposed intermediate feature containing dual information. The superimposed intermediate feature is then processed by a tangent function activation process, outputting a tangent activation feature. The role of the tangent function activation process is to map the additively combined features to a bounded continuous interval, suppressing extreme values, improving the stability of the fusion process, and preserving the differences in positive and negative directions. The tangent activation feature can be understood as a candidate representation after nonlinear integration of local style information and local prosodic information, where different time positions correspond to different expression intensities. In risk warning texts, when words such as loss, warning, caution, and volatility are involved, the bias introduced by the local style component will enhance the expressive tendency of the corresponding position in the superimposed intermediate features, forming a more stable tangent activation feature after tangent function activation processing. In medical and health texts, when words such as follow-up visit, monitoring, abnormality, and timely medication are involved, the local style component will also make directional corrections to the local prosodic features, so that the tangent activation features show clearer expressive differences in the corresponding positions.

[0085] The second prosodic projection feature enters the gating activation process, outputting a gated activation feature. The gating activation process constrains the input to zero to one or other limited intervals, giving the gated activation feature a control over proportions. The second prosodic projection feature comes from another projection branch of the local prosodic feature; it does not represent candidate content but rather performs weight control. By performing gating activation on the second prosodic projection feature, the network generates gating coefficients for each time position and each feature channel. The gated activation feature is then multiplied element-wise with the tangent activation feature to obtain the local fusion feature. Element-wise multiplication performs fine-grained modulation at corresponding positions; when the gating coefficient is large, relevant information in the tangent activation feature is preserved; when the gating coefficient is small, the corresponding information is suppressed. The resulting local fusion feature is no longer a simple superposition of local style components and local prosodic features, but a local representation after positional and channel-related control. This local fusion feature maintains a correspondence with the phoneme-level sequence in the time dimension and simultaneously reflects local style bias, local rhythmic changes, and gating adjustment results in terms of content. In fintech scenarios, when dealing with high-risk products, fund changes, and maturity reminders, gated activation features can increase the weight of key positions, allowing the local fusion features to retain stronger tonal differences. For ordinary status announcements, gated activation features can appropriately reduce local style bias, resulting in smoother speech. In healthcare scenarios, gated activation features can also create differentiated modulation for abnormal announcements, health reminders, and reassurance messages, giving the local fusion features higher local expression resolution.

[0086] After the local fusion features are formed, they enter the feature linear modulation processing dominated by the global style component. The global style component carries the overall attributes of the entire speech segment in terms of formality, friendliness, alertness, and emotional tendency. The global style component is then processed by scaling parameter generation and offset parameter generation, outputting scaling parameters and offset parameters. The scaling parameters control the amplitude range of the local fusion features in each feature channel, while the offset parameters adjust the center position of the local fusion features in each channel. The scaling parameters and offset parameters are dimensionally aligned with the local fusion features and can be shared across the entire segment or broadcast to local locations according to time position. In implementation, the scaling parameter generation and offset parameter generation processes can be completed by two independent mapping heads. Each mapping head consists of a linear layer or a multilayer perceptron, with the global style component as input and a parameter vector with the same dimension as the local fusion features as output. During training, the two parameter generation branches can share the underlying weights or learn independently to adapt to the modulation requirements of different style granularities.

[0087] The local fusion features are multiplied by the scaling parameters to obtain the preliminary modulation features. This feature multiplication process adjusts the amplitude of each channel proportionally, causing the local fusion features to scale under the constraints of the overall style. If the global style component represents a stronger warning tone, the scaling parameters will increase the response values ​​of channels related to tension and stress; if the global style component represents a softer explanatory tone, the scaling parameters will decrease the response amplitude of the relevant channels. The preliminary modulation features are then added to the offset parameters to output the style modulation features. The offset parameters are used to further correct the overall position after amplitude adjustment, causing the local fusion features to be biased towards the target style region in the numerical space. After these two steps, the style modulation features simultaneously retain the phoneme-level style and prosodic details of the local fusion features, while also exhibiting a unified expressive tendency throughout the entire segment due to the constraints of the global style components. The style modulation features maintain consistency with the original local representation in the temporal dimension, and achieve a two-layer unification in the feature space, from local style control to global style constraints.

[0088] To achieve the above processing, a two-layer style fusion network can be constructed. The network input includes local style components, local prosodic features, and global style components. The local gated fusion part consists of three projection branches, one additive fusion unit, one tangent activation unit, one gate activation unit, and one element-wise multiplication unit. The global linear modulation part consists of two parameter generation branches, one multiplicative modulation unit, and one additive modulation unit. The three projection branches receive local style components and local prosodic features, respectively, and output style projection features, first prosodic projection features, and second prosodic projection features. The additive fusion unit receives style projection features and first prosodic projection features, and outputs a superimposed intermediate feature; the tangent activation unit outputs a tangent activation feature; the gate activation unit outputs a gated activation feature; and the element-wise multiplication unit outputs a local fusion feature. The two parameter generation branches receive global style components and output scaling parameters and offset parameters, respectively, which are then multiplicative and additively modulated to output style modulation features. During training, the input can be set as local style components, local prosodic features, and global style components, and the output is either the precursor features of the target acoustic representation or the intermediate alignment features corresponding to the target speech. The loss function can include feature reconstruction loss, style classification loss, style consistency loss, and parameter regularization terms. The feature reconstruction loss constrains the difference between the style modulation features and the target representation; the style classification loss ensures that the style modulation features retain clear style category distinctions; and the style consistency loss maintains the coordination between local and global representations. Training data can come from fintech broadcast corpora and healthcare reminder corpora. The style description text in the input undergoes preprocessing to obtain local and global style components, and the input text undergoes preprocessing to obtain local prosodic features. The model obtains fusion parameters suitable for different business scenarios through supervised learning. In fintech scenarios, training data can cover text types such as risk disclosure, profit explanation, account reminders, and transaction confirmations, enabling the style modulation features to stably express the formality, alertness level, and broadcast rhythm required by different business texts. In healthcare scenarios, training data can cover text types such as medication reminders, follow-up notices, health advice, and abnormality notifications, enabling style modulation features to express different tones, such as gentle reminders, cautious notifications, and stable explanations.

[0089] For example, the formula for local control fusion is:

[0090] in, This represents the local fusion feature at position i. tanh() represents the tangent function activation. This represents the first weight matrix that acts on the local prosodic features. This represents the local prosodic feature at the i-th position. This represents the second weight matrix that acts on the local style components. Indicates local style components. () indicates that the gate function is activated. This represents the third weight matrix that acts on local prosodic features. This indicates element-wise multiplication.

[0091] The global linear modulation formula is:

[0092] in, This represents the style modulation features after global style modulation. This represents the scaling function or scaling parameter generated from the global style components. This represents the offset function or offset parameter generated by the global style components. Represents the global style component. H represents the local blending feature. ⊙ represents element-wise multiplication.

[0093] This embodiment converts local style and prosodic information into representations suitable for fusion computation by performing a first projection mapping on local style components and a second and third projection mapping on local prosodic features. By adding the style projection features and the first prosodic projection features and applying tangent activation, while simultaneously applying gate activation to the second prosodic projection features, and then multiplying the tangent activation features and gated activation features element-wise, fine-grained control of local style and prosodic characteristics can be achieved at each time point and each feature channel, resulting in local fusion features. Furthermore, by performing scaling and offset parameter generation on the global style components and using these parameters to perform feature multiplication and addition on the local fusion features, the style tendencies of the local expressions can be unified at the overall level, resulting in style modulation features.

[0094] In one embodiment, step S50 above includes: S501, Perform duration feature extraction processing on the style modulation features to obtain duration modeling input features; S502, Perform duration dependency modeling processing on the duration modeling input features to obtain the original duration sequence; S503, Perform duration value normalization on the original duration sequence to obtain a normalized duration sequence; S504, Perform pronunciation duration generation processing on the regularized duration sequence to obtain the pronunciation duration; S505, Perform expansion unit generation processing on the pronunciation duration to obtain an expansion unit sequence; S506, Based on the unfolded unit sequence, perform extended mapping relationship generation processing on the style modulation features to obtain the extended mapping relationship; S507, Perform temporal dimension expansion processing on the style modulation features based on the extended mapping relationship to obtain extended features.

[0095] In this embodiment, after the style modulation features enter the duration feature extraction process, the processing objective shifts from local acoustic expression control to duration modeling. The style modulation features themselves already contain information on pronunciation content, local style changes, and overall style constraints, and still maintain a correspondence with phoneme-level units in the time dimension. The duration feature extraction process is used to extract the part directly related to duration allocation from this type of high-dimensional sequence representation, so that subsequent networks no longer face all the mixed style information, but instead receive duration modeling input features more suitable for duration inference. In implementation, the duration feature extraction process can use linear projection layers, one-dimensional convolutional layers, or feedforward networks with residual connections to perform dimensionality reduction, recombination, and local context fusion on the representation of style modulation features at each time position. If a one-dimensional convolutional structure is used, the convolutional kernel slides along the time axis, which can absorb the influence of adjacent phoneme positions on duration. For example, in financial technology broadcast text, the pauses before and after key fields such as amount, interest rate, and risk level will affect the duration of the corresponding phoneme group; in medical and health reminder text, the emphasis of phrases such as medication frequency, abnormal indicators, and follow-up time will also change the local duration. After the duration modeling input features are output, the sequential indexing relationship remains consistent with the original style modulation features, and each position can be traced back to the corresponding phoneme unit.

[0096] The input features for duration modeling are processed by duration dependency modeling to generate the original duration sequence. Duration dependency modeling characterizes the duration relationships between different time positions, determining not only the duration of a single phoneme but also the influence of neighboring phonemes and sentence structure on the current duration. This can be implemented using a sequence modeling network, whose main body can consist of bidirectional recurrent units, multi-layer self-attention units, or feedforward temporal modules. Bidirectional recurrent units absorb contextual dependencies through forward and backward states, self-attention units characterize long-distance effects through positional correlation weights, and feedforward temporal modules capture duration variation patterns in local segments by expanding the receptive field. The network input is the duration modeling input feature sequence, and the output is a predicted value vector with the same length as the sequence, forming the original duration sequence. The original duration sequence reflects the model's initial estimate of the duration of each phoneme; its numerical form can be continuous real numbers or a constrained representation of intermediate durations. In fintech texts, profit announcements and risk notifications have different rhythmic templates. After training, the model will produce higher original duration responses at positions involving warning words. Similarly, anomaly alerts and caring prompts in healthcare texts will also generate different duration estimation distributions. To train this part of the network, data with phoneme duration annotations can be used. The input is a style modulation feature sequence, and the output is a sequence of true durations. The loss function can be mean squared error loss, absolute error loss, or a joint loss with smoothing constraints. The learning rate, batch size, hidden dimension, and number of layers are set according to the corpus size and real-time requirements. Fintech corpora can be derived from asset announcements, risk disclosures, transaction notifications, and corresponding speech alignment results. Healthcare corpora can be derived from reminder texts, follow-up notices, indicator explanation texts, and corresponding speech alignment results.

[0097] After the original duration sequence undergoes duration value normalization, a normalized duration sequence is obtained. The purpose of duration value normalization is to transform the original predicted values ​​into a usable form that meets the requirements of subsequent expansion. The original duration sequence may contain excessively small values, outliers, or continuous values ​​that do not meet the discrete expansion conditions. Duration value normalization uses calculations such as truncation, smoothing, lower bound constraints, upper bound constraints, and discretization to unify the duration at each position into a canonical representation that can be expanded. If the original duration sequence contains values ​​close to zero, normalization can set a minimum duration unit to prevent the corresponding phoneme from being completely eliminated during expansion. If there are abrupt changes between adjacent positions in the original duration sequence, normalization can weaken abnormal fluctuations through local smoothing, making the speech rhythm more continuous. In financial technology texts, normalization can avoid unreasonable duration breaks between monetary and explanatory fields in the same broadcast sentence; in medical and health texts, normalization can reduce the duration imbalance between long drug names and short reminder words. After the normalized duration sequence is formed, each position corresponds to a constrained time length representation, providing standard input for subsequent pronunciation duration generation processing.

[0098] After the regularized duration sequence enters the pronunciation duration generation process, the pronunciation duration is output. The pronunciation duration represents the discrete length occupied by each phoneme unit on the time axis, serving as the direct basis for subsequent sequence expansion. The pronunciation duration generation process numerically maps the regularized duration sequence to the discrete time allocation result, ensuring each phoneme position has a clearly defined number of duration units. This processing can be implemented using rounding operations, thresholding, rounding strategies, or discrete quantization strategies, aiming to generate discrete time markers usable for index expansion without disrupting the overall rhythmic pattern. The pronunciation duration maintains a one-to-one correspondence with the preceding phoneme-level representation; any subsequent expansion operation is based on this positional correspondence. During the training phase, if supervised duration learning is used, the true duration obtained through forced alignment or statistical alignment can be directly used as a label to drive the model to optimize the pronunciation duration prediction accuracy. If a weakly supervised approach is used, the duration modeling part can be indirectly optimized by combining acoustic reconstruction loss and duration consistency constraints, ensuring that the output pronunciation duration satisfies both acoustic reconstruction requirements and preserves the text's rhythmic characteristics.

[0099] After the pronunciation duration is processed by the unwrap unit generation process, an unwrap unit sequence is obtained. This process converts the discrete duration results into an intermediate representation that can be used for feature replication and timeline expansion. Each pronunciation duration value is resolved into a corresponding number of unwrap units during the unwrap unit generation process. The unwrap unit sequence is organized by these units according to the original phoneme order. The unwrap unit sequence does not change the phoneme order; it only explicitly represents the length that each position will occupy in the future along the time dimension. After this processing, the subsequent generation of extended mapping relationships no longer directly depends on the original values ​​but on the already structured unwrap intermediate sequence, making it easier to construct the correspondence between time frames and phoneme positions in the implementation. In fintech businesses, the pronunciation duration of high-risk notifications, expiration reminders, and key broadcast segments is often longer; the unwrap unit sequence can explicitly encode these differences. In healthcare businesses, key expression positions involving abnormal indicators, follow-up examination times, and medication dosages will also form a denser distribution of unwrap units.

[0100] After performing extended mapping relationship generation processing on style modulation features based on the unfolded unit sequence, the extended mapping relationship is output. The extended mapping relationship is used to characterize the correspondence rules between the original phoneme-level positions and the target time frame positions. The unfolded unit sequence determines how many times each phoneme needs to appear in the time dimension, and the extended mapping relationship generation process uses this to establish a mapping table, mapping matrix, or equivalent control structure from phoneme index to frame index. This processing allows subsequent temporal dimension expansion to be performed under strict correspondence conditions, rather than unconstrained copying. The extended mapping relationship includes not only the number of repetitions but also temporal order and positional attribution information, thus ensuring that the arrangement order of different phoneme positions remains unchanged after unfolding. In fintech broadcasting scenarios, if the amount field and the risk warning field are assigned different pronunciation durations, the extended mapping relationship will accurately reflect this difference in subsequent frame-level feature unfolding; in medical and health reminder scenarios, the differences in the duration of drug names, follow-up times, and precautions positions will also be stably transmitted to the time frame level through the extended mapping relationship.

[0101] After performing temporal dimension expansion processing on style modulation features based on the extended mapping relationship, extended features are obtained. The temporal dimension expansion processing, according to the extended mapping relationship, transforms the original phoneme-aligned style modulation features into a feature sequence aligned to time frames. During execution, the style modulation features at each phoneme position are copied, arranged, and aligned on the time axis according to the extended mapping relationship, forming a longer frame-level extended feature sequence. The extended features retain the content, prosodic, and style information of the corresponding phoneme position in each frame, while globally reflecting the temporal distribution differences caused by pronunciation duration. The length of the extended features is usually significantly larger than the length of the original style modulation features, and is consistent with or approximately consistent with the target acoustic frame number, providing direct input for subsequent acoustic decoding. If the subsequent system requires a fixed frame rate, the temporal dimension expansion processing can also fine-tune the extended mapping relationship in conjunction with frame rate constraints to ensure that the extended features meet the target temporal resolution requirements. The resulting extended features no longer remain at the phoneme abstraction level but have become a temporal unfolding representation oriented towards acoustic generation, accurately reflecting the length distribution and rhythmic structure of different content segments in both fintech and healthcare scenarios.

[0102] This embodiment performs duration feature extraction, duration dependency modeling, duration value normalization, and pronunciation duration generation on style modulation features. This transforms phoneme-level style representations into discrete-time allocation results corresponding to actual pronunciation lengths. Furthermore, by performing expansion unit generation, extended mapping relation generation, and temporal dimension expansion on pronunciation durations, phoneme-level features are stably expanded into temporal frame-level extended features. Consequently, speech rhythm no longer relies solely on a uniform duration distribution but can form duration variations that match business semantics based on style modulation results, thereby improving the performance of fintech and healthcare voice in terms of rhythm control, emphasis expression, and temporal consistency.

[0103] In one embodiment, step S60 above includes: S601, perform latent space condition mapping on the extended features to obtain latent space condition features; S602, perform latent variable distribution generation processing on the latent space condition features to obtain latent variable distribution parameters; S603, perform latent variable sampling processing based on the latent variable distribution parameters to obtain latent variable samples; S604, Perform flow matching transformation on the latent variable samples to obtain latent space features; S605, Perform layered sequence decoding processing on the latent space features to obtain decoded intermediate features; S606, Perform deconvolution upsampling processing on the decoded intermediate features to obtain acoustic features.

[0104] In this embodiment, after the extended features enter the latent space conditional mapping process, the processing target shifts from the temporal frame-level modulation representation to a latent variable conditional representation suitable for acoustic generation. The extended features have already undergone temporal expansion related to pronunciation length, with each time position simultaneously containing text content, local prosody, local style, and global style modulation results. However, this representation remains in a text-driven high-level feature space and has not yet been transformed into an acoustic representation space suitable for spectrum generation. The latent space conditional mapping process is completed through a conditional coding network. The conditional coding network receives the extended feature sequence and performs multi-layer feature transformations, causing the original representation to converge towards the latent variable conditional distribution while maintaining temporal order. The conditional coding network can consist of a linear projection layer, a convolutional feature extraction layer, and a sequence context modeling layer. The linear projection layer unifies the input dimension, the convolutional feature extraction layer enhances local continuity between adjacent time positions, and the sequence context modeling layer preserves structural dependencies over a longer time range. The processed output is the latent space conditional feature, which maintains a temporal correspondence with the extended features but is closer to the constraint form required for acoustic latent variable modeling in the feature space.

[0105] After latent space conditional features enter the latent variable distribution generation process, they are converted into a parameterized distribution representation. The latent variable distribution generation process outputs latent variable distribution parameters based on the conditional representation at each time point. These parameters can include mean parameters, scale parameters, or other sets of parameters characterizing the distribution shape; the form depends on the latent variable modeling method. If a variational structure is used, the network output is typically generated by two parallel branches: one branch outputs the mean sequence, and the other outputs the variance or log-variance sequence. If a more general conditional probability modeling structure is used, distribution control parameters that can be used for sampling can also be output. The purpose of this process is to transform the deterministic input representation into a latent variable expression with probabilistic constraints, ensuring that the subsequent generated results maintain consistency with the input text and style constraints while also possessing a modelable acoustic variation space. The latent variable distribution generation process is connected to the preceding conditional coding network; the latent variable distribution parameters output by the network are aligned with the latent space conditional features in the time dimension, forming a corresponding set of probabilistic descriptions for each time point.

[0106] Latent variable sampling is performed based on the latent variable distribution parameters to obtain latent variable samples. This sampling process transforms the distribution parameters into specific latent variable representations that can participate in subsequent streaming transformations. If a reparameterized sampling method is used, latent variable samples are generated based on the mean parameter, scale parameter, and random noise, allowing the sampling process to participate in gradient backpropagation. The sampled latent variable samples retain their sequential form in the time dimension, with each time position corresponding to a latent variable vector. These latent variable samples are not directly used as the final acoustic representation but rather as intermediate samples within the latent space for subsequent distribution correction and streaming transformations. The purpose of latent variable sampling is to embed the conditional information corresponding to the extended features into a probability-constrained implicit representation space, enabling the model to learn the mapping relationship between text-driven representations and the true acoustic distribution during training.

[0107] After latent variable samples enter the stream matching transform process, latent space features are output. The stream matching transform process performs a learnable distribution transformation on the latent variable samples, making the latent variable representation closer to the target acoustic distribution. The stream matching transform process can be composed of multi-layer reversible transform units, residual stream units, or continuous-time vector field approximation units. Each transform unit receives the latent variable sequence from the previous state and progressively corrects the distribution by incorporating conditional information. If a stream matching-based modeling approach is used, the network learns the trajectory constraints from the simple distribution to the target distribution, achieving distribution renormalization by matching the change patterns between intermediate and target states. If a reversible transform structure is used, the network transforms the latent variable samples to a more suitable latent space region for decoding through a series of reversible mappings. The latent space features obtained after the stream matching transform process are no longer just random sampling results, but rather acoustic latent variable representations that incorporate extended feature conditional information and have undergone distribution correction. The latent space features correspond one-to-one with the latent variable samples in the time dimension, and at the same time, their statistical distribution is closer to the acoustic latent representation of real speech.

[0108] After latent space features enter the stacked sequence decoding process, intermediate decoding features are output. The stacked sequence decoding process is used to progressively recover the implicit acoustic representations in the latent space features into intermediate representations with clear temporal structure and spectral trends. The stacked sequence decoding network can adopt a multi-layered decoding block stacking structure. Each decoding block contains self-attention units, feedforward transform units, normalization units, and residual connection units. Conditional interaction units can also be added to some layers to maintain the coupling between latent space features and temporal sequence information during the decoding process. Self-attention units are used to adjust the relationships between acoustic events over a longer time range, such as the mutual influence of long syllables, pauses, and emphasis intervals. Feedforward transform units are used to enhance feature representation capabilities, gradually expanding latent variables into intermediate representations that can be used for spectrum generation. The intermediate decoding features formed after multi-layer stacking already possess strong continuity and local structure in the temporal direction, but the frequency resolution or temporal resolution is still insufficient to directly serve as the final acoustic features; therefore, further upsampling is required.

[0109] After the decoded intermediate features are processed by deconvolution upsampling, the output acoustic features are obtained. Deconvolution upsampling transforms the decoded intermediate features into acoustic features suitable for use by a vocoder or waveform generation module by progressively expanding the resolution of the time or frequency dimensions. The deconvolution upsampling layer can consist of multiple levels of transposed convolutional units, each responsible for expanding the resolution by a predetermined stride, while simultaneously learning the smooth transition relationships between adjacent positions through the convolutional kernel. The acoustic features obtained after upsampling can be represented as a spectral amplitude sequence, a Mel-spectral representation, or other time-frequency domain acoustic parameter sequences. Deconvolution upsampling is not simple interpolation; rather, it completes acoustic detail enhancement while expanding the resolution, giving the output features both temporal continuity and spectral structure. In fintech scenarios, risk warning voices, profit announcement voices, and transaction notification voices have different requirements for tone stability and acoustic performance at key points; deconvolution upsampling can refine these differences at the spectral level. In healthcare scenarios, reminder tones, reassuring tones, and informing tones also exhibit different characteristics through timbre, energy, and frequency distribution in the acoustic features.

[0110] From a model structure perspective, this processing section can be divided into a conditional coding sub-network, a latent variable distribution generation sub-network, a latent variable sampling unit, a stream matching transform sub-network, a stacked sequence decoding sub-network, and a deconvolutional upsampling sub-network. The conditional coding sub-network receives extended features and outputs latent space conditional features; the latent variable distribution generation sub-network outputs latent variable distribution parameters based on the latent space conditional features; the latent variable sampling unit generates latent variable samples based on the latent variable distribution parameters; the stream matching transform sub-network performs distribution correction on the latent variable samples and outputs latent space features; the stacked sequence decoding sub-network recovers the latent space features layer by layer and outputs decoded intermediate features; the deconvolutional upsampling sub-network converts the decoded intermediate features into acoustic features. Each sub-network maintains alignment in the temporal dimension and shares a unified training objective during parameter updates. The training data consists of input-output pairs composed of extended features and corresponding real acoustic representations. If a training method with supervision based on real acoustic features is adopted, the real Mel spectrum or equivalent acoustic representation can be used as the target value to apply a reconstruction loss to the predicted acoustic features. If variational latent variable learning is adopted, a distribution constraint term can be added to the reconstruction loss to ensure that the latent variable distribution parameters are consistent with the prior distribution. If stream matching learning is adopted, trajectory matching loss can be applied to the intermediate and target states in the transformation process, so that the stream matching transformation process learns a stable distribution transfer law. Training optimization can use an adaptive optimizer, and the batch size, number of encoding layers, number of decoding layers, hidden dimension, latent variable dimension, and upsampling layer step size can be set according to the corpus size, time-frequency resolution, and deployment requirements. The corpus for fintech business can cover the speech samples corresponding to texts such as account notifications, product descriptions, risk disclosures, and asset broadcasts, so that the model can learn the distribution differences of formal broadcasts, warning broadcasts, and explanatory broadcasts in the acoustic space. The corpus for healthcare business can cover speech samples such as follow-up reminders, abnormal indicator prompts, medication notifications, and rehabilitation suggestions, so that the model can learn the different distributions of caring expressions, reminder expressions, and informative expressions in the latent space and acoustic space.

[0111] During the model training phase, after completing the construction of each processing stage—input text processing, phoneme and prosody modeling, style encoding and modulation, duration prediction, latent space mapping, acoustic decoding, and waveform generation—the model training employs a joint loss optimization approach to simultaneously constrain speech intelligibility, naturalness, latent space distribution consistency, duration prediction accuracy, and style expression consistency. The overall training objective is expressed as:

[0112] in, Indicates overall training loss. This represents the acoustic feature reconstruction loss, used to constrain the difference between the predicted acoustic features and the target acoustic features; This represents adversarial loss, used to constrain the generated speech to approximate real speech in the time or frequency domain distribution; This represents the latent variable distribution constraint loss, used to limit the deviation between the latent variable distribution parameters and the target prior distribution; This represents the duration prediction loss, used to constrain the difference between the predicted pronunciation duration and the labeled pronunciation duration; This represents style consistency loss, used to constrain the generated speech to maintain consistency with the target style corresponding to the style description text in terms of style expression; , , as well as These represent the weight coefficients of each loss term, used to balance the influence of different training objectives during the parameter update process. In one set of implementations, The value ranges from 0.5 to 2.0. The value ranges from 0.0001 to 0.1. The value ranges from 0.5 to 5.0. The value ranges from 0.5 to 3.0. If a stronger emphasis is required on vocal style, the value can be increased. If a stronger requirement for duration consistency is needed, it can be improved. If higher stability is required in the latent space, it can be improved. .

[0113] The training input includes input text, style description text, acoustic features corresponding to real speech, real speech waveforms, and real pronunciation duration annotations. The input text undergoes standardization, phoneme conversion, and prosodic feature extraction to form text-side features. The style description text undergoes semantic encoding and decomposition to form local and global style components. The local style components and local prosodic features are then subjected to gating fusion and feature linear modulation to form style modulation features. These style modulation features undergo duration prediction and sequence expansion to form extended features. The extended features undergo latent space mapping and acoustic decoding to output predicted acoustic features. These predicted acoustic features are then processed by waveform synthesis to output predicted speech waveforms. Therefore, reconstruction loss, adversarial loss, duration loss, style loss, and latent variable distribution constraint loss can all be jointly backpropagated in the same parameter update.

[0114] The training network can be divided into a text feature encoding unit, a style encoding unit, a two-layer fusion modulation unit, a duration prediction unit, a latent space mapping unit, an acoustic decoding unit, a waveform generation unit, and a discriminator unit. The text feature encoding unit receives phoneme sequences and prosodic features, and outputs local prosodic features. The style encoding unit receives style description text and outputs local and global style components. The two-layer fusion modulation unit receives local style components, global style components, and local prosodic features, and outputs style modulation features. The duration prediction unit receives style modulation features and outputs pronunciation duration. The latent space mapping unit receives extended features and outputs latent variable distribution parameters and latent space features. The acoustic decoding unit receives latent space features and outputs predicted acoustic features. The waveform generation unit receives predicted acoustic features and outputs predicted speech waveforms. The discriminator unit receives predicted speech waveforms and real speech waveforms and outputs a authenticity judgment result. The units are sequentially connected; the style encoding unit and text feature encoding unit merge at the two-layer fusion modulation unit; the latent space mapping unit and acoustic decoding unit are connected sequentially; and the waveform generation unit and discriminator unit form an adversarial training structure.

[0115] Training parameters can be set according to resource conditions and corpus size. In one implementation, the batch size is set to 16, 32, or 64, the initial learning rate is set to 0.0001 to 0.0005, the optimizer uses adaptive moment estimation, the momentum parameter can be set to 0.9 and 0.999, and the weight decay can be set to 0 to 0.0001. The hidden dimension of the text-side encoding network and the style encoding network can be set to 192, 256, or 384, and the number of layers can be set to 4 to 8. The hidden dimension of the duration prediction unit can be set to 128 to 256, and the number of layers can be set to 2 to 4. The latent variable dimension can be set to 32, 64, or 128. The number of decoding layers of the acoustic decoding unit can be set to 4 to 12, and the number of deconvolution upsampling layers can be set to 2 to 5. Gradient pruning can be used during training to suppress gradient anomalies, and the pruning threshold can be set to 0.5 to 5.0. If mixed precision training is used, the training time can be shortened and the memory usage reduced while maintaining numerical stability.

[0116] Data partitioning during training can be configured as training, validation, and test sets. In one implementation, the training set comprises 80%, the validation set 10%, and the test set 10%. The fintech corpus can include account announcement text, risk warning text, return explanation text, and corresponding voice data. Style description text can be set to a conservative tone, a risk warning tone, a formal notification tone, etc. The healthcare corpus can include medication reminder text, abnormal indicator notification text, follow-up notification text, and corresponding voice data. Style description text can be set to a caring tone, a reminder tone, an informative tone, etc. In the input data, text content, style description text, pronunciation duration annotation, acoustic feature annotation, and speech waveform are aligned at the sample level to ensure that each loss term corresponds to the same speech instance during joint training.

[0117] Training termination conditions can be jointly controlled by round constraints and validation metric constraints. In one implementation, the maximum number of training rounds is set to 200 to 500. Early termination is triggered when the overall loss of the validation set decreases by less than 0.0001 within 10 to 20 consecutive rounds. When the overall loss of the validation set increases for 5 consecutive rounds, the learning rate can be reduced, with a learning rate decay coefficient set to 0.5. Training is terminated when the learning rate decays to one percent of the initial learning rate and the overall loss of the validation set still does not decrease significantly. If an adversarial training structure is used, stability conditions for discriminative and generative losses can be added. When both remain within a preset fluctuation range for several consecutive rounds, the adversarial training is considered to be stable. The preset fluctuation range can be set to no more than 5% above or below the average of the most recent 10 rounds, depending on the corpus size. For the duration prediction part, a pronunciation duration deviation threshold can be added. When the average duration error on the validation set is lower than the set threshold and continues for several rounds, the duration modeling part is considered to have converged. After the termination conditions are met, the model parameters corresponding to the lowest overall loss of the validation set are retained as the final parameters.

[0118] If fintech scenarios emphasize real-time broadcasting, the batch size can be set to 32 or 64, the hidden dimension to 192 or 256, and the number of decoding layers to 4 to 6 to reduce inference overhead. If healthcare scenarios emphasize tone stability and subtlety, the hidden dimension can be increased to 256 or 384, the number of decoding layers to 6 to 10, and the style consistency loss weight can be appropriately increased. In both scenarios, the input data format remains consistent: text content, style description text, real acoustic features, real speech waveforms, and pronunciation duration annotations. The output data consists of predicted acoustic features, predicted speech waveforms, predicted pronunciation durations, and corresponding style representations. The only difference lies in the loss weights and network size, which are adjusted according to business needs. This approach allows for adaptation to both fintech broadcasting and healthcare alert scenarios while maintaining a consistent overall training structure.

[0119] This embodiment transforms the time-expanded text-driven representation into a probability-constrained latent variable representation by performing latent space condition mapping, latent variable distribution generation, and latent variable sampling on the extended features. By performing stream matching transformation on the latent variable samples, the latent variable representation is made to further approximate the target acoustic distribution, yielding latent space features. Furthermore, by performing layered sequence decoding and deconvolution upsampling on the latent space features, the temporal structure is recovered layer by layer, and the acoustic features are output. Thus, the extended features can undergo distribution correction within the constrained latent space and then be stably converted into a time-frequency domain acoustic representation, thereby improving the consistency, smoothness, and controllability of acoustic representation in financial technology and healthcare speech.

[0120] In one embodiment, a speech generation apparatus based on two-layer style modulation is provided, which corresponds one-to-one with the speech generation method based on two-layer style modulation described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech generation device based on dual-layer style modulation of the present invention. The modules include: text preprocessing and prosody extraction module 10, local prosody modeling module 20, global style encoding module 30, style fusion modulation module 40, duration modeling and expansion module 50, acoustic modeling and decoding module 60, and waveform generation module 70. Detailed descriptions of each functional module are as follows: The text preprocessing and prosody extraction module 10 is used to receive input text and style description text, perform standardization processing on the input text to obtain standardized text, perform phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extract the prosodic features of the standardized text. The local prosodic modeling module 20 is used to generate phoneme embedding vectors based on the phoneme sequence, perform context encoding processing on the phoneme embedding vectors to obtain context dependencies, and perform gating modulation based on the phoneme embedding vectors, the context dependencies, and the prosodic features to obtain local prosodic features. The global style encoding module 30 is used to perform semantic encoding on the style description text to obtain a global style vector, and to decompose the global style vector to obtain local style components and global style components. The style fusion modulation module 40 is used to perform gated fusion based on the local style components and the local prosodic features to obtain local fusion features, and to perform feature linear modulation on the local fusion features based on the global style components to obtain style modulation features; The duration modeling and expansion module 50 is used to perform duration prediction on the style modulation features to obtain the pronunciation duration, and to perform sequence expansion on the style modulation features based on the pronunciation duration to obtain expanded features; The acoustic modeling and decoding module 60 is used to map the extended features to the acoustic latent space to obtain latent space features, and perform acoustic decoding based on the latent space features to obtain acoustic features; The waveform generation module 70 is used to perform waveform synthesis processing based on the acoustic features to obtain a speech waveform.

[0121] Specific limitations regarding the speech generation device based on two-layer style modulation can be found in the aforementioned limitations on the speech generation method based on two-layer style modulation, and will not be repeated here. Each module in the aforementioned speech generation device based on two-layer style modulation can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0122] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side speech generation method based on two-layer style modulation.

[0123] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a speech generation method based on two-layer style modulation.

[0124] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The system receives input text and style description text, performs standardization processing on the input text to obtain standardized text, performs phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extracts the prosodic features of the standardized text. A phoneme embedding vector is generated based on the phoneme sequence. Context encoding is performed on the phoneme embedding vector to obtain context dependencies. Gated modulation is then performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features. The style description text is semantically encoded to obtain a global style vector, and the global style vector is decomposed to obtain local style components and global style components. Gated fusion is performed based on the local style components and the local prosodic features to obtain local fusion features, and feature linear modulation is performed on the local fusion features based on the global style components to obtain style modulation features; The style modulation features are subjected to duration prediction to obtain the pronunciation duration, and the style modulation features are sequence extended based on the pronunciation duration to obtain extended features; The extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features; Based on the acoustic features, waveform synthesis processing is performed to obtain a speech waveform.

[0125] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: The system receives input text and style description text, performs standardization processing on the input text to obtain standardized text, performs phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extracts the prosodic features of the standardized text. A phoneme embedding vector is generated based on the phoneme sequence. Context encoding is performed on the phoneme embedding vector to obtain context dependencies. Gated modulation is then performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features. The style description text is semantically encoded to obtain a global style vector, and the global style vector is decomposed to obtain local style components and global style components. Gated fusion is performed based on the local style components and the local prosodic features to obtain local fusion features, and feature linear modulation is performed on the local fusion features based on the global style components to obtain style modulation features; The style modulation features are subjected to duration prediction to obtain the pronunciation duration, and the style modulation features are sequence extended based on the pronunciation duration to obtain extended features; The extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features; Based on the acoustic features, waveform synthesis processing is performed to obtain a speech waveform.

[0126] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech generation method based on two-layer style modulation, characterized in that, Includes the following steps: The system receives input text and style description text, performs standardization processing on the input text to obtain standardized text, performs phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extracts the prosodic features of the standardized text. A phoneme embedding vector is generated based on the phoneme sequence. Context encoding is performed on the phoneme embedding vector to obtain context dependencies. Gated modulation is then performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features. The style description text is semantically encoded to obtain a global style vector, and the global style vector is decomposed to obtain local style components and global style components. Gated fusion is performed based on the local style components and the local prosodic features to obtain local fusion features, and feature linear modulation is performed on the local fusion features based on the global style components to obtain style modulation features; The style modulation features are subjected to duration prediction to obtain the pronunciation duration, and the style modulation features are sequence extended based on the pronunciation duration to obtain extended features; The extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features; Based on the acoustic features, waveform synthesis processing is performed to obtain a speech waveform.

2. The speech generation method based on dual-layer style modulation as described in claim 1, characterized in that, The system receives input text and style description text, performs standardization processing on the input text to obtain standardized text, performs phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extracts the prosodic features of the standardized text, including: Receive input text and style description text; The input text is cleaned to obtain cleaned text. The cleaned text is then analyzed for punctuation and segmented into sentences to obtain segmented text. The language type is identified and labeled in the segmented text to obtain standardized text; Based on the language type corresponding to the standardized text, phoneme conversion processing is performed on the standardized text to obtain a phoneme sequence; Based on the language type, tone features and stress features are extracted from the standardized text; Based on the structure and punctuation information of the standardized text, pause boundary features are extracted from the standardized text. By combining the tone features, the stress features, and the pause boundary features, the prosodic features of the standardized text are obtained.

3. The speech generation method based on two-layer style modulation as described in claim 1, characterized in that, A phoneme embedding vector is generated based on the phoneme sequence. Context encoding is performed on the phoneme embedding vector to obtain context dependencies. Gated modulation is then performed based on the phoneme embedding vector, the context dependencies, and the prosodic features to obtain local prosodic features, including: The phoneme sequence is subjected to embedding mapping processing to obtain a phoneme embedding vector; Perform multi-layer feedforward coding on the phoneme embedding vector to obtain the context dependency; Perform feature concatenation processing on the phoneme embedding vector, the context dependency, and the prosodic features to obtain gated input features; Perform a first feature transformation and a second feature transformation on the gated input features to obtain a first transformed feature and a second transformed feature; Perform nonlinear activation processing on the first transformation feature to obtain candidate modulation features; The second transformation feature is subjected to gating activation processing to obtain gating weight features; The candidate modulation features and the gate weight features are subjected to element-wise weighted modulation to obtain the gated output features; The gated output features are subjected to feature mapping to obtain local prosodic features.

4. The speech generation method based on dual-layer style modulation as described in claim 1, characterized in that, The style description text is semantically encoded to obtain a global style vector. The global style vector is then decomposed to obtain local style components and global style components, including: The style description text is segmented into lexical units to obtain a style lexical sequence. The style lexical sequence is then embedded into lexical units to obtain a lexical representation sequence. The lexical representation sequence is then encoded using a pre-trained language model to obtain a sentence-level semantic representation. The sentence-level semantic representation is subjected to feature aggregation and normalization to obtain a global style vector; The first branch feature extraction is performed on the global style vector to obtain the first decomposition sub-vector; The global style vector is subjected to second branch feature extraction to obtain the second decomposition sub-vector; Perform linear mapping on the first decomposed subvector to obtain the local style component, and generate the global style component based on the second decomposed subvector.

5. The speech generation method based on dual-layer style modulation as described in claim 1, characterized in that, Gated fusion is performed based on the local style components and the local prosodic features to obtain local fused features, and feature linear modulation is performed on the local fused features based on the global style components to obtain style-modulated features, including: A first projection mapping is performed on the local style component to obtain style projection features; a second projection mapping is performed on the local rhythmic features to obtain first rhythmic projection features; and a third projection mapping is performed on the local rhythmic features to obtain second rhythmic projection features. The style projection feature and the first prosody projection feature are added together to obtain a superimposed intermediate feature. The superimposed intermediate feature is then activated by a tangent function to obtain a tangent activated feature. The second prosodic projection feature is subjected to a gated activation process to obtain a gated activation feature. The tangent activation feature and the gated activation feature are multiplied element-wise to obtain a local fusion feature. Perform scaling parameter generation and offset parameter generation processes on the global style components to obtain scaling parameters and offset parameters; The local fusion features and the scaling parameters are multiplied together to obtain preliminary modulation features. The preliminary modulation features and the offset parameters are then added together to obtain style modulation features.

6. The speech generation method based on dual-layer style modulation as described in claim 1, characterized in that, The style modulation features are subjected to duration prediction to obtain the pronunciation duration, and the style modulation features are then extended based on the pronunciation duration to obtain extended features, including: The style modulation features are subjected to duration feature extraction processing to obtain duration modeling input features; Perform duration dependency modeling processing on the duration modeling input features to obtain the original duration sequence; The original duration sequence is subjected to duration value normalization to obtain a normalized duration sequence; The regularized duration sequence is processed to generate the pronunciation duration, thus obtaining the pronunciation duration. The pronunciation duration is processed to generate expansion units, resulting in an expansion unit sequence; Based on the unfolded unit sequence, the style modulation features are subjected to an extended mapping relationship generation process to obtain the extended mapping relationship; Based on the extended mapping relationship, the style modulation features are subjected to temporal dimension extension processing to obtain extended features.

7. The speech generation method based on dual-layer style modulation as described in claim 1, characterized in that, The extended features are mapped to the acoustic latent space to obtain latent space features, and acoustic decoding is performed based on the latent space features to obtain acoustic features, including: The extended features are subjected to latent space condition mapping to obtain latent space condition features; The latent space condition features are subjected to latent variable distribution generation processing to obtain latent variable distribution parameters; Latent variable sampling is performed based on the latent variable distribution parameters to obtain latent variable samples; Perform flow matching transformation on the latent variable samples to obtain latent space features; Perform a layered sequence decoding process on the latent space features to obtain the decoded intermediate features; The intermediate decoded features are subjected to deconvolution upsampling to obtain acoustic features.

8. A speech generation device based on dual-layer style modulation, characterized in that, The speech generation device based on dual-layer style modulation includes: The text preprocessing and prosody extraction module is used to receive input text and style description text, perform standardization processing on the input text to obtain standardized text, perform phoneme conversion processing on the standardized text to obtain a phoneme sequence, and extract the prosodic features of the standardized text. The local prosodic modeling module is used to generate phoneme embedding vectors based on the phoneme sequence, perform context encoding processing on the phoneme embedding vectors to obtain context dependencies, and perform gating modulation based on the phoneme embedding vectors, the context dependencies, and the prosodic features to obtain local prosodic features. A global style encoding module is used to perform semantic encoding on the style description text to obtain a global style vector, and to decompose the global style vector to obtain local style components and global style components. The style fusion modulation module is used to perform gated fusion based on the local style components and the local prosodic features to obtain local fusion features, and to perform feature linear modulation on the local fusion features based on the global style components to obtain style modulation features; The duration modeling and expansion module is used to predict the duration of the style modulation features to obtain the pronunciation duration, and to expand the style modulation features based on the pronunciation duration to obtain expanded features; An acoustic modeling and decoding module is used to map the extended features to the acoustic latent space to obtain latent space features, and to perform acoustic decoding based on the latent space features to obtain acoustic features; The waveform generation module is used to perform waveform synthesis processing based on the acoustic features to obtain a speech waveform.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a speech generation program based on two-layer style modulation stored in the memory and executable on the processor. When executed by the processor, the speech generation program based on two-layer style modulation implements the steps of the speech generation method based on two-layer style modulation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a speech generation program based on two-layer style modulation, which, when executed by a processor, implements the steps of the speech generation method based on two-layer style modulation as described in any one of claims 1-7.

Images