An adaptive speech synthesis method and system based on artificial intelligence

Abstract

This application provides an artificial intelligence-based adaptive speech synthesis method and system. It involves semantically parsing the text data to be synthesized to obtain semantic anchor feature vectors and emotional intensity temporal vectors. Based on these, a prosodic prediction model for emotional modulation is constructed, outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with changes in text emotionality. The pitch trajectory parameter sequence, relative duration parameter sequence, and semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. An acoustic parameter prediction model is constructed based on this two-layer intermediate acoustic representation tensor, and the generated predicted acoustic parameter sequence is input into a speech waveform synthesis network to obtain the final speech synthesis result. Using the scheme of this application, refined prosodic modulation that is updated synchronously with changes in text emotionality can be achieved while maintaining the accuracy and clarity of semantic content.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and more specifically, to an adaptive speech synthesis method and system based on artificial intelligence. Background Technology

[0002] Speech synthesis technology is the process of converting text information into audible speech signals through mechanical, electronic, or deep learning-based methods. Its goal is to generate synthesized speech with high naturalness, high intelligibility, and conformity to human hearing habits. With the rapid development of artificial intelligence technology, end-to-end speech synthesis systems based on deep neural networks have gradually become mainstream, capable of generating natural speech that is close to the level of human pronunciation.

[0003] Emotional speech synthesis is an important direction for further development based on traditional speech synthesis. It aims to enable synthesized speech to express specific emotional nuances, such as happiness, sadness, and anger, thereby enhancing the naturalness and friendliness of human-computer interaction. Existing technologies typically employ emotion labeling or emotion category embedding to control the synthesis process. This involves introducing emotion classification loss during model training or concatenating emotion encoding vectors into the model input, allowing the model to learn the acoustic feature mapping relationships under different emotional states. However, existing methods often treat emotion as a global, static control signal, applying a uniform emotional intensity to the entire text, making it difficult to finely characterize the internal nuances of the text. The dynamic fluctuations of emotion that change with semantic content—for example, the transition from a gentle emotion in the first half of a sentence to a strong emotion in the second half—are a challenge that current technologies cannot replicate in sync with the text's semantics. Furthermore, existing methods often employ simple feature concatenation or addition in fusing emotional and semantic features, leading to mutual coupling and interference between them. On one hand, the intervention of emotional information may affect the clarity of the semantic content; on the other hand, changes in semantic content may blur the accuracy of emotional expression. Ultimately, synthesized speech may either exhibit stiff, rigid emotional expression lacking depth, or suffer from decreased semantic clarity, negatively impacting the listening experience. Therefore, achieving refined prosodic control that updates in sync with the text's emotional changes while maintaining accurate and clear semantic content, enabling synthesized speech to possess both accurate conceptual content and natural, nuanced emotional expression, has become a significant challenge for the industry. Summary of the Invention

[0004] This application provides an artificial intelligence-based adaptive speech synthesis method and system that can achieve refined prosodic control that updates synchronously with changes in the emotional content of the text while maintaining the accuracy and clarity of the semantic content.

[0005] In a first aspect, this application provides an adaptive speech synthesis method based on artificial intelligence, comprising the following steps: Semantic hierarchical parsing is performed on the synthetic text data to obtain semantic anchor feature vectors and sentiment intensity temporal vectors; Based on the semantic anchor feature vector and the emotional intensity temporal vector, a prosodic prediction model for emotional regulation is constructed. The emotional intensity temporal vector is mapped to a time weight control matrix, and the time weight control matrix is ​​used to dynamically gating the pitch prediction channel and the speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion. The pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantic stability components and prosodic modulation components; An acoustic parameter prediction model is constructed based on the two-layer intermediate acoustic representation tensor, and an emotion consistency control mechanism constrained by the emotion intensity temporal vector is introduced to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into a speech waveform synthesis network to obtain the final speech synthesis result.

[0006] In conjunction with the first aspect, one possible implementation involves performing semantic-level parsing on the text data to be synthesized to obtain semantic anchor feature vectors and sentiment intensity temporal vectors, specifically including: The synthetic text data is parsed to separate the semantic anchor components and sentiment expression components. Deep semantic encoding is performed on the semantic anchor components, and the core semantic representations that do not change with emotion are extracted as semantic anchor feature vectors; A word-by-word sentiment analysis is performed on the emotional expression components, and the sentiment intensity value at each time step is extracted to form a temporal vector of sentiment intensity.

[0007] In conjunction with the first aspect, in one possible implementation, a prosodic prediction model for emotion regulation is constructed based on the semantic anchor feature vector and the emotion intensity temporal vector. Specifically, this involves mapping the emotion intensity temporal vector to a time weight control matrix and using the time weight control matrix to dynamically gate the pitch prediction channel and speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with changes in text emotion. The emotional intensity time-series vector is mapped to generate a time weight control matrix for regulating pitch and speech rate; Based on the semantic anchor feature vector, a prosodic prediction model including a pitch prediction channel and a speech rate prediction channel is constructed. The time weight control matrix is ​​used as a dynamic gating signal to adjust the pitch prediction channel and speech rate prediction channel, and outputs a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text sentiment.

[0008] In conjunction with the first aspect, in one possible implementation, the pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. Specifically, this includes: Temporal extension is performed on the semantic anchor feature vector to generate a semantically stable temporal basis that is aligned with the pitch trajectory parameter sequence and the relative duration parameter sequence in the time dimension; The pitch trajectory parameter sequence and the relative duration parameter sequence are jointly encoded, and the joint prosodic fluctuation features of the two are extracted as prosodic modulation components. The semantically stable temporal basis and the prosodic modulation component are layered and superimposed and their dimensions are reorganized to construct a two-layer intermediate acoustic representation tensor containing the semantically stable component and the prosodic modulation component.

[0009] In conjunction with the first aspect, in one possible implementation, constructing an acoustic parameter prediction model based on the two-layer intermediate acoustic representation tensor specifically includes: Initialize an acoustic parameter prediction network infrastructure with a two-layer intermediate acoustic representation tensor as input, wherein the acoustic parameter prediction network infrastructure includes an encoding module, a timing processing module, and a decoding module; The two-layer intermediate acoustic representation tensor is used as the input of training samples to perform iterative parameter training on the acoustic parameter prediction network infrastructure until the loss function between the predicted acoustic parameters output by the network and the real acoustic parameters converges. The trained acoustic parameter prediction network infrastructure and its optimized parameter set are defined as the acoustic parameter prediction model.

[0010] In conjunction with the first aspect, one possible implementation involves introducing an emotion consistency control mechanism constrained by an emotion intensity temporal vector to generate a predictive acoustic parameter sequence. This predictive acoustic parameter sequence is then input into a speech waveform synthesis network to obtain the final speech synthesis result. Specifically, this includes: The acoustic parameter prediction model is constructed by inputting the two-layer intermediate acoustic representation tensor to obtain the predicted acoustic parameter sequence; Using the emotional intensity time-series vector as an emotional constraint, an emotional consistency control mechanism is constructed to dynamically modulate the predicted acoustic parameter sequence, generating a modulated acoustic parameter sequence that conforms to emotional characteristics. The modulated acoustic parameter sequence is input into a speech waveform synthesis network to generate an initial speech waveform signal. The initial speech waveform signal is enhanced using an adaptive post-processing network, and the optimized speech waveform signal is output as the final speech synthesis result.

[0011] In conjunction with the first aspect, in one possible implementation, the dual-layer intermediate acoustic representation tensor refers to a three-dimensional tensor structure constructed by layering and recombining semantically stable temporal base and prosodic modulation components. This structure simultaneously contains semantically stable information at the bottom layer that does not change with emotion and prosodic modulation information at the top layer that changes with emotion, while the two components remain structurally separated.

[0012] Secondly, this application provides an artificial intelligence-based adaptive speech synthesis system, comprising: The parsing module is used to perform semantic-level parsing on the text data to be synthesized, and obtain semantic anchor feature vectors and sentiment intensity temporal vectors. The processing module is used to construct a prosodic prediction model for emotion regulation based on the semantic anchor feature vector and the emotion intensity temporal vector. The model maps the emotion intensity temporal vector to a time weight control matrix and uses the time weight control matrix to dynamically gating the pitch prediction channel and speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion. The processing module is further configured to structurally fuse the pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. The execution module is used to construct an acoustic parameter prediction model based on the two-layer intermediate acoustic representation tensor, and introduce an emotion consistency control mechanism constrained by the emotion intensity temporal vector to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into the speech waveform synthesis network to obtain the final speech synthesis result.

[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device executes the above-described artificial intelligence-based adaptive speech synthesis method.

[0014] Fourthly, this application provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the aforementioned artificial intelligence-based adaptive speech synthesis method.

[0015] The technical solution provided in this application has the following beneficial effects: This application's approach first involves semantic-level parsing of the text data to be synthesized, extracting semantic anchor feature vectors and sentiment intensity temporal vectors separately. This allows for the separation and expression of semantic content and sentiment change information at the data structure level. The semantic anchor feature vectors represent the core semantic content of the text that remains unchanged despite sentiment variations, preventing semantic distortion caused by emotional factors during subsequent prosodic adjustment. The sentiment intensity temporal vectors, on the other hand, depict the trend of sentiment changes over time, providing a control basis for the dynamic adjustment of subsequent prosodic parameters, thus ensuring semantic integrity from the outset. Under the premise of accurate and clear content, emotion regulation capabilities are introduced; secondly, by mapping the temporal vector of emotion intensity into a time weight control matrix, and using this matrix to dynamically gate and adjust the pitch prediction channel and speech rate prediction channel, the prediction process of prosodic parameters is transformed from static calculation to dynamic adjustment according to changes in text emotion. This allows the pitch trajectory parameter sequence and relative duration parameter sequence to be updated synchronously with emotion changes. This mechanism gives prosodic regulation temporal continuity and emotional specificity, achieving fine-grained control over the amplitude and rhythm of prosodic changes without altering the semantic content prediction path; then, by... The pitch trajectory parameter sequence, relative duration parameter sequence, and semantic anchor feature vector are structurally fused to form a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. This allows the acoustic modeling process to simultaneously possess information channels with two different attributes: a content stability layer and an emotion change layer. This two-layer structure can maintain the consistency of semantic content during speech generation while accurately reflecting the prosodic fluctuations driven by emotion, providing a stable data foundation for achieving the unity of semantic accuracy and fine prosodic control. Finally, based on the acoustic parameter prediction model constructed on the two-layer intermediate acoustic representation tensor, an emotion-based prediction model is introduced. The emotion consistency control mechanism, constrained by the temporal direction of the intensity of the emotion, establishes a synergistic constraint relationship between the acoustic continuity and the trend of emotional evolution in the generated speech waveform, thereby reducing emotion drift and prosodic collapse. Simultaneously, it enhances the sound quality of the generated speech, improving the overall listening experience and achieving a refined prosodic control effect that updates synchronously with the emotional changes in the text. In summary, this scheme can achieve refined prosodic control that updates synchronously with the emotional changes in the text while maintaining the accuracy and clarity of the semantic content, enabling the synthesized speech to possess both accurate conceptual content and natural, nuanced emotional expression. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an exemplary flowchart of an artificial intelligence-based adaptive speech synthesis method according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the implementation of semantic hierarchy parsing according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of a two-layer intermediate acoustic representation tensor according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an artificial intelligence-based adaptive speech synthesis system according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device implementing an artificial intelligence-based adaptive speech synthesis method according to some embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] refer to Figure 1 The figure is an exemplary flowchart of an AI-based adaptive speech synthesis method according to some embodiments of this application. The AI-based adaptive speech synthesis method mainly includes the following steps: In step 101, the text data to be synthesized is subjected to semantic hierarchical parsing to obtain semantic anchor feature vectors and sentiment intensity temporal vectors.

[0020] In some embodiments, reference Figure 2 As shown, this diagram is an exemplary flowchart of semantic hierarchical parsing in some embodiments of this application. Semantic hierarchical parsing of the text data to be synthesized to obtain semantic anchor feature vectors and sentiment intensity temporal vectors can be achieved through the following steps: The synthetic text data is parsed to separate the semantic anchor components and sentiment expression components. Deep semantic encoding is performed on the semantic anchor components, and the core semantic representations that do not change with emotion are extracted as semantic anchor feature vectors; A word-by-word sentiment analysis is performed on the emotional expression components, and the sentiment intensity value at each time step is extracted to form a temporal vector of sentiment intensity.

[0021] In practice, the syntactic structure parsing of the text data to be synthesized, separating the semantic anchor components and sentiment expression components, can be achieved in the following way: First, the text data to be synthesized is input into a pre-trained dependency parser. This parser identifies the dependency relationships and parent-child node connections of each word in the text, and then extracts the core predicates and their main arguments as semantic anchor components, which may include necessary syntactic components such as subjects, predicates, and objects. At the same time, all words containing emotional coloring, adverbs, interjections, and various clauses expressing emotions are taken as sentiment expression components. Subsequently, a sequence labeling model based on conditional random fields can be used to perform secondary verification and boundary correction on the dependency analysis results to ensure that the division of semantic anchor components and sentiment expression components conforms to language habits and eliminates ambiguity. Finally, for complex texts with parallel structures or compound sentences, a recursive syntactic tree traversal method can be used to peel off the nested structure layer by layer until a complete set of semantic anchor components and the corresponding set of sentiment expression components are separated.

[0022] In specific implementation, deep semantic encoding of the semantic anchor components and extraction of core semantic representations that do not change with emotion as semantic anchor feature vectors can be achieved in the following way: First, the separated semantic anchor components are input into a pre-trained deep semantic encoding model. This model can use a multi-layer transformer structure to perform context-aware semantic mapping on the semantic anchor components, focusing on the logical relationship between words within the main body rather than emotional coloring through a self-attention mechanism, thereby suppressing the encoding weight of emotion-related information. Next, the hidden state vectors corresponding to special markers or all words in the output of the last layer of the model are subjected to mean pooling or max pooling to obtain a dense vector of fixed dimension as the initial semantic representation. Then, a preset semantically stable projection matrix is ​​applied to the initial semantic representation. This matrix can be obtained by training on a large-scale general corpus without emotion annotation, which can map the semantics to an emotion-insensitive orthogonal subspace, further removing residual emotional interference. Finally, the vector after projection transformation is output as the semantic anchor feature vector. This vector only carries the conceptual content of the text and remains constant in the subsequent speech synthesis process, providing a stable content skeleton for speech.

[0023] In specific implementation, the emotional expression components are analyzed word by word to extract the emotional intensity value at each time step, forming an emotional intensity temporal vector. This can be achieved in the following way: First, the separated emotional expression component sequence is input into a pre-trained emotional intensity prediction model. This model can use a bidirectional long short-term memory network structure to temporally encode the word sequence, capture the intensity changes and polarity shifts of emotional words in the context, and output the emotional intensity value corresponding to each word, with a value range of [0, 1], where 0 represents no emotion and 1 represents the strongest emotion. Then, based on the word segmentation results of the text and the preset frame shift parameters, such as 10ms / frame, the word-level emotional intensity values ​​are interpolated to frame-level emotional intensity values ​​to form an emotional intensity temporal vector aligned with the text length. Finally, the vector is smoothed and filtered to eliminate drastic jumps that may occur between adjacent time steps, ensuring the continuity and naturalness of emotional changes, thus obtaining the final emotional intensity temporal vector used for dynamic prosody control. Other methods can also be used in other embodiments, which are not limited here.

[0024] It should be noted that, in this application, the semantic anchor component refers to the core syntactic components separated from the text to be synthesized, used to carry the basic conceptual content of the text; the emotional expression component refers to the modifying words attached to the semantic anchor component separated from the text to be synthesized, used to carry the emotional coloring and modifying information of the text; the core semantic representation that does not change with emotion refers to the intermediate semantic representation obtained after deep semantic encoding of the semantic anchor component, used to remove the interference of emotional information on semantics, focus on the text content itself, and provide pure semantic material for generating semantic anchor feature vectors; the semantic anchor feature vector refers to the final feature vector obtained after further projection transformation of the core semantic representation that does not change with emotion, used to provide a constant content skeleton in the speech synthesis process, ensuring that the semantic accuracy of the synthesized speech is not affected by emotional fluctuations; the emotional intensity temporal vector refers to the frame-level emotional intensity sequence obtained by interpolation after word-by-word emotional intensity analysis of the text, used to dynamically adjust the pitch and duration in the speech synthesis process, so that the synthesized speech has the prosodic expressiveness synchronized with the emotional changes of the text.

[0025] In step 102, a prosodic prediction model for emotion regulation is constructed based on the semantic anchor feature vector and the emotion intensity temporal vector. The emotion intensity temporal vector is mapped to a time weight control matrix, and the time weight control matrix is ​​used to dynamically gating the pitch prediction channel and the speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion.

[0026] In some embodiments, a prosodic prediction model for emotion regulation is constructed based on the semantic anchor feature vector and the emotion intensity temporal vector. The model is implemented by mapping the emotion intensity temporal vector to a time weight control matrix and using the time weight control matrix to dynamically gate the pitch prediction channel and speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with changes in text emotion. This can be achieved through the following steps: The emotional intensity time-series vector is mapped to generate a time weight control matrix for regulating pitch and speech rate; Based on the semantic anchor feature vector, a prosodic prediction model including a pitch prediction channel and a speech rate prediction channel is constructed. The time weight control matrix is ​​used as a dynamic gating signal to adjust the pitch prediction channel and speech rate prediction channel, and outputs a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text sentiment.

[0027] In specific implementation, mapping the emotional intensity temporal vector to generate a time weight control matrix for regulating pitch and speech rate can be achieved in the following way: First, the emotional intensity temporal vector is input into a fully connected mapping network containing two parallel sub-network branches, corresponding to the pitch control branch and the speech rate control branch, respectively; then, the pitch control branch maps the emotional intensity temporal vector into a time weight sequence corresponding to the duration dimension of the synthesized speech through two fully connected layers with non-linear activation functions. Each weight value in this sequence is used to characterize the modulation amplitude of the emotional intensity at the corresponding moment on pitch rise or fall; simultaneously... The speech rate modulation branch maps the temporal vector of emotional intensity to another temporal weight sequence through a similar structure, which is used to characterize the modulation magnitude of emotional intensity on speech rate acceleration or deceleration at each moment. Then, the two temporal weight sequences are concatenated or stacked along the channel dimension to form a two-dimensional temporal weight control matrix. The rows of this matrix correspond to the two modulation dimensions of pitch and speech rate, and the columns correspond to the time steps of speech synthesis. Finally, the matrix is ​​smoothed and filtered to eliminate possible abrupt changes between adjacent time steps, ensuring the continuity and naturalness of emotional modulation, thus obtaining the final temporal weight control matrix used for dynamic gating of pitch and speech rate.

[0028] In specific implementation, the prosodic prediction model, which includes a pitch prediction channel and a speech rate prediction channel, can be constructed based on the semantic anchor feature vector as follows: First, a duration prediction sub-network is constructed, which uses the semantic anchor feature vector as input to predict the preliminary duration distribution of the speech to be synthesized (i.e., the preliminary duration corresponding to each phoneme), and obtains the total number of frames T; then, an encoder structure is constructed, which uses the semantic anchor feature vector and the preliminary duration distribution as input. This encoder uses a multi-layer one-dimensional convolutional network or a bidirectional long short-term memory network to perform temporal expansion and contextual modeling on the semantically stable features, and outputs a semantic hidden state sequence aligned with the preliminary duration T; then, a pitch prediction channel and a speech rate prediction channel are built after this encoder, wherein the pitch prediction channel consists of two layers with residual connections. The system consists of fully connected layers, taking the semantic hidden state sequence as input and outputting the fundamental frequency logarithmic domain prediction value corresponding to each time step, forming a preliminary pitch trajectory parameter sequence. The speech rate prediction channel consists of a gated recurrent unit network and a fully connected layer, taking the semantic hidden state sequence as input and outputting the speech rate factor prediction value corresponding to each time step, correcting the preliminary duration, and forming the final relative duration parameter sequence. Finally, the initial parameters of the encoder, pitch prediction channel and speech rate prediction channel are initialized through pre-training on a large-scale neutral sentiment speech dataset to obtain a basic model that can generate basic prosodic parameters based on pure semantic content. This model is the initial prosodic prediction model before sentiment modulation is applied, waiting for subsequent dynamic gating adjustment through a time weight control matrix.

[0029] In specific implementation, the time weight control matrix is ​​used as a dynamic gating signal to adjust the pitch prediction channel and the speech rate prediction channel. The output pitch trajectory parameter sequence and relative duration parameter sequence, which are updated synchronously with the text sentiment changes, can be implemented in the following way: First, the time weight control matrix is ​​decoupled into a pitch weight vector and a speech rate weight vector, corresponding to the first and second rows of the matrix, respectively; then, in the pitch prediction channel, the pitch weight vector is applied to the intermediate layer output or the final output of the pitch prediction channel in an element-wise multiplication manner. Specifically, the pitch weight vector is multiplied by the fundamental frequency prediction value of the last fully connected layer output of the pitch prediction channel using a Hadamard product operation, thereby dynamically raising or lowering the pitch value at each time step to obtain the sentiment-modulated pitch trajectory parameter sequence; simultaneously, in the In the speech rate prediction channel, the speech rate weight vector is input into a speech rate adapter. This adapter converts the speech rate weight vector into a time stretching factor through cumulative summation and interpolation operations, thereby recalibrating the initial speech rate factor output by the speech rate prediction channel. This ensures that regions with strong emotions correspond to faster speech rates, and regions with milder emotions correspond to slower speech rates, resulting in a sentiment-modulated relative duration parameter sequence. Finally, the modulated pitch trajectory parameter sequence and the relative duration parameter sequence are temporally aligned to ensure strict synchronization in the time dimension. The final pitch trajectory parameter sequence and relative duration parameter sequence, updated synchronously with the text's sentiment changes, are output. These two sequences are the sentiment prosodic parameters used for structured fusion in subsequent steps. Other methods can also be used in other embodiments, which are not limited here.

[0030] It should be noted that the time weight control matrix in this application refers to a two-dimensional matrix structure generated by mapping the emotional intensity temporal vector, used as a dynamic gating signal to carry emotional modulation information for both pitch and speech rate dimensions. The rows of the matrix correspond to the two control dimensions of pitch and speech rate, and the columns correspond to the time steps of speech synthesis. The prosodic prediction model in this application refers to a basic prediction model constructed using semantic anchor feature vectors as input, used to generate a preliminary pitch trajectory parameter sequence and a relative duration parameter sequence based on pure semantic content without applying emotional influence. The pitch trajectory parameter sequence in this application refers to... The set of temporal parameters output after dynamic gating adjustment by the time weight control matrix is ​​used to characterize the trajectory of fundamental frequency change that is updated synchronously with the emotional changes in the text. As one of the emotional prosodic parameters for subsequent structured fusion, it ensures that the pitch fluctuations of the synthesized speech can accurately reflect the emotional fluctuations of the text. The relative duration parameter sequence in this application refers to the set of temporal parameters output after dynamic gating adjustment by the time weight control matrix. It is used to characterize the trajectory of speech rate change that is updated synchronously with the emotional changes in the text. As one of the emotional prosodic parameters for subsequent structured fusion, it ensures that the speed and rhythm of the synthesized speech can accurately reflect the emotional fluctuations of the text.

[0031] In step 103, the pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components.

[0032] In some embodiments, reference Figure 3 As shown, this figure is an exemplary flowchart for determining a two-layer intermediate acoustic representation tensor in some embodiments of this application. In this embodiment, the pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. This can be achieved by the following steps: In step 1031, the semantic anchor feature vector is temporally extended to generate a semantically stable temporal basis that is aligned with the pitch trajectory parameter sequence and the relative duration parameter sequence in the time dimension; In step 1032, the pitch trajectory parameter sequence and the relative duration parameter sequence are jointly encoded, and the joint prosodic fluctuation features of the two are extracted as prosodic modulation components. In step 1033, the semantically stable temporal basis and the prosodic modulation component are layered and recombined to construct a two-layer intermediate acoustic representation tensor containing the semantically stable component and the prosodic modulation component.

[0033] In specific implementation, temporal extension of the semantic anchor feature vector to generate a semantically stable temporal basis aligned with the pitch trajectory parameter sequence and the relative duration parameter sequence in the time dimension can be achieved in the following way: First, obtain the length of the pitch trajectory parameter sequence as the target total number of time steps, which corresponds to the total number of frames of the speech to be synthesized; then, input the semantic anchor feature vector into a temporal extension network, which can adopt a recurrent neural network decoder structure based on an attention mechanism, using the semantic anchor feature vector as the initial hidden state, and the pitch trajectory parameter sequence and the relative duration parameter sequence as auxiliary inputs for conditional decoding, outputting a semantic extension vector corresponding to the current time step at each time step; or, in another implementation, phoneme-level semantic coding and global semantic coding can be used. An extended method for semantic anchor combination: First, phoneme segmentation is performed on the text to be synthesized to obtain the local semantic code corresponding to each phoneme; then, the global semantic anchor feature vector is concatenated with the local semantic code of each phoneme to obtain a phoneme-level semantic feature sequence; finally, the phoneme-level semantic feature sequence is interpolated into a temporal sequence aligned with the total number of target time steps based on the relative duration parameter sequence, and then input into a temporal convolutional network for context smoothing, so that the semantic vector of each time step contains both the global core semantics and reflects the semantic differences of local phonemes; finally, the temporal vector sequence obtained after decoding or smoothing is output as a semantically stable temporal basis, which corresponds one-to-one with the pitch trajectory parameter sequence and the relative duration parameter sequence in the time dimension, so as to provide semantically stable information that is strictly aligned in time for subsequent layer superposition.

[0034] In specific implementation, the joint encoding of the pitch trajectory parameter sequence and the relative duration parameter sequence, and the extraction of their joint prosodic fluctuation features as prosodic modulation components, can be achieved in the following way: First, the pitch trajectory parameter sequence and the relative duration parameter sequence are concatenated along the feature dimension to form a two-dimensional joint prosodic parameter matrix. The rows of this matrix correspond to time steps, and the columns correspond to pitch values ​​and speech rate factors, respectively. Then, this joint prosodic parameter matrix is ​​input into a bidirectional long short-term memory network encoder. The encoder's temporal modeling capability captures the cooperative relationship and mutual influence between pitch and speech rate over time, and outputs each time step. The hidden state vector corresponding to each time step is used; then, the sequence of hidden state vectors corresponding to each time step output by the bidirectional long short-term memory network encoder is used as the joint prosodic fluctuation feature; or, in another implementation, the joint prosodic parameter matrix is ​​input into a two-dimensional convolutional neural network, and the local prosodic change pattern is extracted by sliding the convolution kernel in both time and feature dimensions, and the temporal feature sequence with the same length as the input time step is output; finally, the temporal feature sequence is output as the prosodic modulation component, which summarizes the dynamic prosodic information jointly expressed by pitch and speech rate step by step, and is used as upper-layer information for hierarchical fusion with the semantically stable temporal base.

[0035] In specific implementation, the semantically stable temporal base and the prosodic modulation component are layered and their dimensions reorganized to construct a two-layer intermediate acoustic representation tensor containing the semantically stable component and the prosodic modulation component. This can be achieved in the following way: First, ensure that the semantically stable temporal base and the prosodic modulation component have the same time step length and feature dimension; then, construct a three-dimensional tensor structure, setting the first layer (i.e., the first channel of the depth dimension) of this three-dimensional tensor as the semantically stable temporal base to carry the underlying semantically stable information that does not change with emotion, and setting the second layer (i.e., the second channel of the depth dimension) of this three-dimensional tensor as the prosodic modulation component. The modulation component carries the prosodic modulation information of the upper layer that changes with emotion, thus forming a three-dimensional tensor of shape [T, D, 2], where T is the total number of time steps and D is the feature dimension. Then, the three-dimensional tensor is subjected to layer normalization to eliminate possible magnitude differences between different layers. Finally, the normalized tensor is output as the final two-layer intermediate acoustic representation tensor, which simultaneously contains a semantically stable component to ensure the accuracy of the content and a prosodic modulation component to impart emotional expressiveness to the speech, and the two components are structurally separated. Other methods can also be used in other embodiments, which are not limited here.

[0036] It should be noted that the semantically stable temporal basis in this application refers to a temporal vector sequence that is aligned in the time dimension with the pitch trajectory parameter sequence and the relative duration parameter sequence after temporal expansion of the semantic anchor feature vector. This provides strictly temporally aligned semantically stable information for layered overlay, ensuring that speech generation at each time step is supported by corresponding core semantic content. The joint prosodic fluctuation feature in this application refers to a global feature representation extracted after joint encoding of the pitch trajectory parameter sequence and the relative duration parameter sequence. This feature is used to summarize the dynamic prosodic information jointly expressed by pitch and speech rate, capturing the two... The collaborative relationship between the components changes over time; the prosodic modulation component in this application refers to the feature representation used to carry the prosodic modulation information of the upper layer that changes with emotion, and is used as the upper layer information to perform layered fusion with the semantically stable temporal base, giving the subsequent acoustic representation tensor an emotion-driven prosodic expressiveness; the double-layer intermediate acoustic representation tensor in this application refers to the three-dimensional tensor structure constructed by layering and reorganizing the semantically stable temporal base and the prosodic modulation component, which is used to simultaneously contain the semantically stable information of the lower layer that does not change with emotion and the prosodic modulation information of the upper layer that changes with emotion, and the two components are structurally separated.

[0037] In step 104, an acoustic parameter prediction model is constructed based on the two-layer intermediate acoustic representation tensor, and an emotion consistency control mechanism constrained by the emotion intensity temporal vector is introduced to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into the speech waveform synthesis network to obtain the final speech synthesis result.

[0038] In some embodiments, constructing an acoustic parameter prediction model based on the two-layer intermediate acoustic representation tensor can be achieved using the following steps: Initialize an acoustic parameter prediction network infrastructure with a two-layer intermediate acoustic representation tensor as input, wherein the acoustic parameter prediction network infrastructure includes an encoding module, a timing processing module, and a decoding module; The two-layer intermediate acoustic representation tensor is used as the input of training samples to perform iterative parameter training on the acoustic parameter prediction network infrastructure until the loss function between the predicted acoustic parameters output by the network and the real acoustic parameters converges. The trained acoustic parameter prediction network infrastructure and its optimized parameter set are defined as the acoustic parameter prediction model.

[0039] In specific implementation, an acoustic parameter prediction network infrastructure is initialized, taking a two-layer intermediate acoustic representation tensor as input. This infrastructure includes an encoding module, a temporal processing module, and a decoding module, which can be implemented as follows: First, an encoding module is constructed. This module can employ a 3-layer 2D convolutional neural network structure with a kernel size of (3,3), a stride of (1,1), and padding of (1,1). Using the two-layer intermediate acoustic representation tensor as input, it gradually extracts high-level abstract features by sliding the convolutional kernel across the time and feature dimensions, while preserving the separation structure between semantically stable components and prosodic modulation components in the tensor, outputting a deeply encoded hidden feature tensor. Subsequently, a temporal processing module is built. This module can employ a 4-layer transformer structure to perform temporal context modeling on the hidden feature tensor output by the encoding module, capturing speech frames. The system identifies long-range dependencies between components and outputs a context-aware feature sequence enhanced by time. Then, a decoding module is constructed, which can employ a two-layer transposed convolutional network structure. Taking the feature sequence output from the time-series processing module as input, it progressively decodes the feature sequence into an initial acoustic feature sequence matching the target acoustic parameters (i.e., Mel spectrum, dimension 80). Finally, the encoding, time-series processing, and decoding modules are sequentially connected to form a complete acoustic parameter prediction network infrastructure. The network weights of each module are randomly initialized, awaiting subsequent parameter optimization using training data.

[0040] In specific implementation, the two-layer intermediate acoustic representation tensor is used as the input training sample to iteratively train the acoustic parameter prediction network infrastructure until the loss function between the predicted acoustic parameters and the real acoustic parameters output by the network converges. This can be achieved in the following way: First, obtain the corresponding two-layer intermediate acoustic representation tensors from the training dataset as input samples, and the corresponding real Mel spectrum sequences as supervision labels; then, feed the input samples into the acoustic parameter prediction network infrastructure for forward propagation, passing through the encoding module, temporal processing module, and decoding module in sequence to obtain the predicted Mel spectrum sequence output by the network; then, calculate the predicted Mel spectrum sequence using a preset loss function. The difference between the spectrum and the real Mel spectrum is analyzed. The loss function uses mean squared error as the main loss term and introduces structural similarity loss as an auxiliary constraint to improve the naturalness and detail fidelity of the generated acoustic parameters. Next, based on the calculated loss value, the gradient of each network layer parameter is calculated using the backpropagation algorithm, and the network parameters are updated using the adaptive moment estimation optimizer with a learning rate set to 0.0001. Finally, the above forward propagation, loss calculation and parameter update process is repeated to iteratively process multiple batches of samples in the training dataset until the loss function value tends to stabilize and no longer decreases or reaches the preset upper limit of the number of iterations, such as 100,000 times. At this time, the network parameter training is considered to have converged.

[0041] In specific implementation, the trained acoustic parameter prediction network infrastructure and its optimized parameter set can be defined as the acoustic parameter prediction model in the following way: First, save the final weight parameters and network structure configuration information of each module in the acoustic parameter prediction network infrastructure after training convergence, namely the encoding module, the timing processing module, and the decoding module; then, load the saved parameter set back into the network infrastructure to form a complete network instance with definite parameter values ​​that can be deployed and used; finally, encapsulate the network instance and its associated preprocessing parameters and postprocessing parameters together into an acoustic parameter prediction model. This model can receive a new two-layer intermediate acoustic representation tensor as input and directly output the corresponding predicted acoustic parameter sequence for use in subsequent speech waveform synthesis steps. Other methods can also be used in other embodiments, which are not limited here.

[0042] It should be noted that the acoustic parameter prediction network infrastructure in this application refers to an initial network structure consisting of an encoding module, a temporal processing module, and a decoding module connected sequentially. This structure serves as the model framework to be trained, receiving a two-layer intermediate acoustic representation tensor as input and outputting initial predicted acoustic parameters after being processed step-by-step by each module. The encoding module extracts high-level abstract features from the input two-layer intermediate acoustic representation tensor while maintaining the separation of semantics and prosody. The temporal processing module performs temporal context modeling on the encoded features to capture long-range dependencies between speech frames. The decoding module progressively converts the temporally enhanced features into an initial acoustic feature sequence that matches the dimension of the target acoustic parameters. The acoustic parameter prediction model in this application refers to a complete network instance formed by encapsulating the trained and converged acoustic parameter prediction network infrastructure and its optimized final weight parameters, along with relevant preprocessing and post-processing parameters, as a deployable acoustic parameter prediction tool.

[0043] In some embodiments, an emotion consistency control mechanism constrained by an emotion intensity temporal vector is introduced to generate a predictive acoustic parameter sequence. The predictive acoustic parameter sequence is then input into a speech waveform synthesis network to obtain the final speech synthesis result. This can be achieved through the following steps: The acoustic parameter prediction model is constructed by inputting the two-layer intermediate acoustic representation tensor to obtain the predicted acoustic parameter sequence; Using the emotional intensity time-series vector as an emotional constraint, an emotional consistency control mechanism is constructed to dynamically modulate the predicted acoustic parameter sequence, generating a modulated acoustic parameter sequence that conforms to emotional characteristics. The modulated acoustic parameter sequence is input into a speech waveform synthesis network to generate an initial speech waveform signal. The initial speech waveform signal is enhanced using an adaptive post-processing network, and the optimized speech waveform signal is output as the final speech synthesis result.

[0044] In specific implementation, the predicted acoustic parameter sequence can be obtained by inputting the two-layer intermediate acoustic representation tensor into the constructed acoustic parameter prediction model in the following manner: First, load the final weight parameters determined after training convergence from the acoustic parameter prediction model and deploy the model in forward inference mode; then, feed the two-layer intermediate acoustic representation tensor as input data into the acoustic parameter prediction model, and perform layer-by-layer calculations through the encoding module, temporal processing module, and decoding module in the model. The encoding module performs high-level abstract feature extraction on the input tensor and preserves the separation structure of semantics and prosody; the temporal processing module... The processing module models the long-range dependencies of the extracted features, and the decoding module gradually maps the temporally enhanced features to outputs that match the dimensions of the target acoustic parameters. Then, the original predicted values ​​of the last layer output of the model decoding module are obtained. These predicted values ​​are the acoustic feature sequences corresponding to the time steps of the input tensor, which may include acoustic parameters such as Mel spectral coefficients or linear spectral coefficients. Finally, the original predicted values ​​are optionally inversely normalized to map them from the normalized space used during model training back to the real acoustic feature space, resulting in the final predicted acoustic parameter sequence, which is the input material for subsequent emotion consistency modulation.

[0045] In specific implementation, using the emotional intensity time-series vector as the emotional constraint, an emotional consistency control mechanism is constructed to dynamically modulate the predicted acoustic parameter sequence, generating a modulated acoustic parameter sequence that conforms to emotional characteristics. This can be achieved in the following way: First, an emotional adaptation network is constructed, which adopts a 3-layer fully connected structure with an input dimension of 1 (emotional intensity value) and an output dimension of 80 (Mel spectrum dimension), used to generate the acoustic parameter modulation coefficients for each time step; then, each element in the emotional intensity time-series vector is input into the emotional adaptation network to obtain the 80-dimensional modulation coefficient vector corresponding to each time step; then, the initial predicted acoustic parameter sequence is multiplied element-wise with the modulation coefficient vector, that is, the Mel spectrum coefficient of each time step is equal to the initial predicted value multiplied by the modulation coefficient of the corresponding time step, thereby dynamically adjusting the original acoustic parameters in an emotionally driven manner at each time step; finally, the modulated acoustic parameter sequence is output, which retains the acoustic quality of the original predicted acoustic parameters and injects emotional features that match the emotional intensity time-series vector; other methods can also be used in other embodiments, which are not limited here.

[0046] In specific implementation, the modulation of the acoustic parameter sequence is input into the speech waveform synthesis network to generate the initial speech waveform signal. This can be achieved in the following way: First, a pre-trained speech waveform synthesis network is loaded. This network can adopt a neural network vocoder structure, such as the HiFi-GAN vocoder model, and can be built based on a generative adversarial network or a streaming generation model. Its function is to convert the acoustic parameter sequence into a playable original speech waveform. Then, the modulation of the acoustic parameter sequence is input into the speech waveform synthesis network frame by frame in chronological order. The network generates a corresponding waveform sampling point for each input frame through upsampling and sample-by-sample prediction. Then, the vocoder module in the network performs phase reconstruction and waveform splicing on the generated waveform sampling points to ensure smooth and natural waveform transitions between adjacent frames and avoid audible splicing traces. Finally, the complete waveform data output by the network is obtained as the initial speech waveform signal, which already has basic speech intelligibility and naturalness.

[0047] In specific implementation, the initial speech waveform signal is enhanced using an adaptive post-processing network, and the optimized speech waveform signal is output as the final speech synthesis result. This can be achieved as follows: First, an adaptive post-processing network based on a temporal convolutional network is constructed. This network contains 5 layers of gated linear units, with a 16kHz sampling rate speech waveform as input and an enhanced speech waveform as output. Then, the initial speech waveform signal is windowed and segmented before being input into the adaptive post-processing network, with a frame length of 20ms and a frame shift of 10ms. Next, the network performs noise suppression and high-frequency enhancement processing on each waveform frame. Specifically, the gated linear units learn the separation features between noise and speech, suppressing background noise while enhancing the energy of the high-frequency components, making the speech clearer and brighter. Then, the processed waveform frames are reconstructed using an overlap-addition method to form a complete optimized speech waveform signal. Finally, the optimized speech waveform signal is verified against the corresponding semantic anchor feature vector to ensure that the waveform content is consistent with the semantics of the original text. Once the verification is passed, it is output as the final speech synthesis result. Other methods can also be used in other embodiments, which are not limited here.

[0048] It should be noted that the predicted acoustic parameter sequence in this application refers to the acoustic feature sequence corresponding to the time step of the input tensor after the acoustic parameter prediction model is constructed by inputting a two-layer intermediate acoustic representation tensor, and is used as input material for subsequent emotion consistency modulation. The emotion consistency control mechanism in this application refers to a processing mechanism that dynamically modulates the predicted acoustic parameter sequence using the emotion intensity time-series vector as an emotion constraint. It is used to inject the emotion features represented by the prosodic modulation component into the acoustic parameter sequence, ensuring that the modulated acoustic parameter sequence is consistent with the emotion of the original text in terms of emotion expression. The modulated acoustic parameter sequence in this application refers to the acoustic parameter sequence output after the predicted acoustic parameter sequence is dynamically modulated by the emotion consistency control mechanism. This sequence retains the original acoustic quality and is injected with emotion features that match the emotion intensity time-series vector. It is used as input to the speech waveform synthesis network to drive... This application generates emotionally expressive speech waveforms. The speech waveform synthesis network in this application refers to a generation network that takes a modulated acoustic parameter sequence as input and outputs an original speech waveform, used to convert the acoustic parameter sequence into a playable time-domain waveform signal. The initial speech waveform signal in this application refers to the complete waveform data output after the modulated acoustic parameter sequence is processed by the speech waveform synthesis network, used as an intermediate speech result with basic speech intelligibility and naturalness. The sound quality enhancement processing in this application refers to noise suppression and high-frequency enhancement processing performed on the initial speech waveform signal, used to improve the overall listening quality of the synthesized speech. The final speech synthesis result in this application refers to the optimized speech waveform signal output after the initial speech waveform signal undergoes sound quality enhancement processing by an adaptive post-processing network, used as the final output result of the entire speech synthesis process. This result, based on semantic accuracy, possesses prosodic expressiveness synchronized with changes in the emotional content of the text.

[0049] Furthermore, in another aspect of this application, in some embodiments, this application provides an artificial intelligence-based adaptive speech synthesis system, with reference to... Figure 4 The figure is a schematic diagram of the structure of an AI-based adaptive speech synthesis system 400 according to some embodiments of this application. The AI-based adaptive speech synthesis system 400 includes: a parsing module 401, a processing module 402, and an execution module 403, which are described below: The parsing module 401 in this application is mainly used to perform semantic hierarchical parsing on the text data to be synthesized, and to obtain semantic anchor feature vectors and sentiment intensity temporal vectors. Processing module 402, in this application, is mainly used to construct a prosodic prediction model for emotion regulation based on the semantic anchor feature vector and the emotion intensity temporal vector. Specifically, by mapping the emotion intensity temporal vector to a time weight control matrix, and using the time weight control matrix to dynamically gating the pitch prediction channel and speech rate prediction channel, a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion are output. The processing module 402 described in this application is further used to structurally fuse the pitch trajectory parameter sequence, the relative duration parameter sequence and the semantic anchor feature vector to obtain a two-layer intermediate acoustic representation tensor containing semantic stable components and prosodic modulation components; The execution module 403 in this application is mainly used to construct an acoustic parameter prediction model based on the two-layer intermediate acoustic representation tensor, and introduce an emotion consistency control mechanism constrained by the emotion intensity temporal vector to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into the speech waveform synthesis network to obtain the final speech synthesis result.

[0050] The foregoing has detailed examples of the AI-based adaptive speech synthesis method and system provided in the embodiments of this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0051] In some embodiments, this application also provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device performs the above-described artificial intelligence-based adaptive speech synthesis method.

[0052] In some embodiments, reference Figure 5 The dashed lines in the figure indicate that the unit or module is optional. This figure is a schematic diagram of the structure of a computer device implementing the AI-based adaptive speech synthesis method of this application. The AI-based adaptive speech synthesis method in the above embodiments can... Figure 5The computer device 500 shown is used to implement this, and the computer device 500 includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device 500 may be a terminal device, a server or a chip.

[0053] The processor 501 can be a general-purpose processor or a special-purpose processor. For example, the processor 501 can be a central processing unit (CPU). The CPU can be used to control the computer device 500, execute software programs, and process data from the software programs. The computer device 500 may also include a communication unit 505 for inputting (receiving) and outputting (transmitting) signals.

[0054] For example, computer device 500 may be a chip, communication unit 505 may be the input and / or output circuit of the chip, or communication unit 505 may be the communication interface of the chip, and the chip may be a component of terminal device, network device or other device.

[0055] For example, computer device 500 may be a terminal device or a server, and communication unit 505 may be a transceiver of the terminal device or the server, or communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0056] The computer device 500 may include one or more memories 502 storing a program 504. The program 504 can be executed by a processor 501 to generate instructions 503, causing the processor 501 to perform the methods described in the above method embodiments according to the instructions 503. Optionally, the memory 502 may also store data (such as a target audit model). Optionally, the processor 501 may also read data stored in the memory 502, which may be stored at the same storage address as the program 504, or the data may be stored at a different storage address than the program 504.

[0057] The processor 501 and memory 502 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0058] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in the processor 501. The processor 501 can be a CPU, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] For example, in some embodiments, this application also provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the above-described artificial intelligence-based adaptive speech synthesis method.

[0061] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0062] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An adaptive speech synthesis method based on artificial intelligence, characterized in that, Includes the following steps: Semantic hierarchical parsing is performed on the synthetic text data to obtain semantic anchor feature vectors and sentiment intensity temporal vectors; Based on the semantic anchor feature vector and the emotional intensity temporal vector, a prosodic prediction model for emotional regulation is constructed. The emotional intensity temporal vector is mapped to a time weight control matrix, and the time weight control matrix is ​​used to dynamically gating the pitch prediction channel and the speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion. The pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantic stability components and prosodic modulation components. An acoustic parameter prediction model is constructed based on the two-layer intermediate acoustic representation tensor, and an emotion consistency control mechanism constrained by the emotion intensity temporal vector is introduced to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into a speech waveform synthesis network to obtain the final speech synthesis result.

2. The method as described in claim 1, characterized in that, Semantic hierarchical parsing of the synthesized text data yields semantic anchor feature vectors and sentiment intensity temporal vectors, specifically including: The synthetic text data is parsed to separate the semantic anchor components and sentiment expression components. Deep semantic encoding is performed on the semantic anchor components, and the core semantic representations that do not change with emotion are extracted as semantic anchor feature vectors; A word-by-word sentiment analysis is performed on the emotional expression components, and the sentiment intensity value at each time step is extracted to form a temporal vector of sentiment intensity.

3. The method as described in claim 1, characterized in that, A prosodic prediction model for emotion regulation is constructed based on the semantic anchor feature vector and the emotion intensity temporal vector. Specifically, by mapping the emotion intensity temporal vector to a time weight control matrix, and using this time weight control matrix to dynamically gate the pitch prediction channel and speech rate prediction channel, a sequence of pitch trajectory parameters and a sequence of relative duration parameters updated synchronously with changes in text emotion are output. The emotional intensity time-series vector is mapped to generate a time weight control matrix for regulating pitch and speech rate; Based on the semantic anchor feature vector, a prosodic prediction model including a pitch prediction channel and a speech rate prediction channel is constructed. The time weight control matrix is ​​used as a dynamic gating signal to adjust the pitch prediction channel and speech rate prediction channel, and outputs a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text sentiment.

4. The method as described in claim 1, characterized in that, The pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector are structurally fused to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. Specifically, this includes: Temporal extension is performed on the semantic anchor feature vector to generate a semantically stable temporal basis that is aligned with the pitch trajectory parameter sequence and the relative duration parameter sequence in the time dimension; The pitch trajectory parameter sequence and the relative duration parameter sequence are jointly encoded, and the joint prosodic fluctuation features of the two are extracted as prosodic modulation components. The semantically stable temporal basis and the prosodic modulation component are layered and superimposed and their dimensions are reorganized to construct a two-layer intermediate acoustic representation tensor containing the semantically stable component and the prosodic modulation component.

5. The method as described in claim 1, characterized in that, The construction of an acoustic parameter prediction model based on the aforementioned two-layer intermediate acoustic representation tensor specifically includes: Initialize an acoustic parameter prediction network infrastructure with a two-layer intermediate acoustic representation tensor as input, wherein the acoustic parameter prediction network infrastructure includes an encoding module, a timing processing module, and a decoding module; The two-layer intermediate acoustic representation tensor is used as the input of training samples to perform iterative parameter training on the acoustic parameter prediction network infrastructure until the loss function between the predicted acoustic parameters output by the network and the real acoustic parameters converges. The trained acoustic parameter prediction network infrastructure and its optimized parameter set are defined as the acoustic parameter prediction model.

6. The method as described in claim 1, characterized in that, An emotion consistency control mechanism constrained by the temporal vector of emotion intensity is introduced to generate a predictive acoustic parameter sequence. This predictive acoustic parameter sequence is then input into a speech waveform synthesis network to obtain the final speech synthesis result, specifically including: The acoustic parameter prediction model is constructed by inputting the two-layer intermediate acoustic representation tensor to obtain the predicted acoustic parameter sequence; Using the emotional intensity time-series vector as an emotional constraint, an emotional consistency control mechanism is constructed to dynamically modulate the predicted acoustic parameter sequence, generating a modulated acoustic parameter sequence that conforms to emotional characteristics. The modulated acoustic parameter sequence is input into a speech waveform synthesis network to generate an initial speech waveform signal. The initial speech waveform signal is enhanced using an adaptive post-processing network, and the optimized speech waveform signal is output as the final speech synthesis result.

7. The method as described in claim 1, characterized in that, The aforementioned dual-layer intermediate acoustic representation tensor refers to a three-dimensional tensor structure constructed by layering and reorganizing semantically stable temporal base and prosodic modulation components. It is used to simultaneously contain semantically stable information at the bottom layer that does not change with emotion and prosodic modulation information at the top layer that changes with emotion, while the two components remain structurally separated.

8. An adaptive speech synthesis system based on artificial intelligence, characterized in that, include: The parsing module is used to perform semantic-level parsing on the text data to be synthesized, and obtain semantic anchor feature vectors and sentiment intensity temporal vectors. The processing module is used to construct a prosodic prediction model for emotion regulation based on the semantic anchor feature vector and the emotion intensity temporal vector. The model maps the emotion intensity temporal vector to a time weight control matrix and uses the time weight control matrix to dynamically gating the pitch prediction channel and speech rate prediction channel, thereby outputting a pitch trajectory parameter sequence and a relative duration parameter sequence that are updated synchronously with the changes in text emotion. The processing module is further configured to structurally fuse the pitch trajectory parameter sequence, the relative duration parameter sequence, and the semantic anchor feature vector to obtain a two-layer intermediate acoustic representation tensor containing semantically stable components and prosodic modulation components. The execution module is used to construct an acoustic parameter prediction model based on the two-layer intermediate acoustic representation tensor, and introduce an emotion consistency control mechanism constrained by the emotion intensity temporal vector to generate a predicted acoustic parameter sequence. The predicted acoustic parameter sequence is then input into the speech waveform synthesis network to obtain the final speech synthesis result.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory being used to store computer programs, and the processor being used to retrieve and run the computer programs from the memory, causing the computer device to perform the artificial intelligence-based adaptive speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions or code that, when executed on a computer, cause the computer to implement the artificial intelligence-based adaptive speech synthesis method as described in any one of claims 1 to 7.

Images