Emotional dialogue speech synthesis method and system based on heterogeneous subgraph comparative learning

Abstract

The invention belongs to the technical field of voice data processing, and particularly discloses an emotional dialogue voice synthesis method and system based on heterogeneous subgraph comparative learning. The method comprises the following steps: obtaining a dialogue voice data set and carrying out cleaning preprocessing, carrying out characterization processing on dialogue voice data, constructing a heterogeneous dialogue graph, carrying out context emotion modeling based on heterogeneous subgraph comparison learning, and carrying out voice synthesis based on emotion perception. According to the method, through heterogeneous subgraph comparative learning, context emotion clues can be focused, and the problem that emotion and context are disjointed in a traditional method is solved; continuous emotion intensity and dimension changes are supported through spherical coordinate coding, and the limitation of discrete emotion labels is broken through; the heterogeneous graph can dynamically integrate multiple rounds of dialogue information and can adapt to emotional evolution of a long dialogue scene; the emotion parameters and the voice acoustic features are directly mapped, so that mismatching of the emotion and the voice features is effectively avoided.

Description

Technical Field

[0001] This invention belongs to the field of speech data processing technology, and specifically relates to an emotional dialogue speech synthesis method and system based on heterogeneous subgraph contrastive learning, which is particularly suitable for human-computer interaction scenarios such as human-computer emotional dialogue. Background Technology

[0002] In practical human-computer emotional dialogue applications, users often expect the system to respond coherently, naturally, and in accordance with the context and emotional direction of the conversation, just like a human. For example, when a user expresses anxiety about something and continues to ask related questions, the system should continue to understand the user's anxiety and show appropriate concern and reassurance in its response; and when the topic shifts to lighter content, the system's emotional expression should also be adjusted accordingly.

[0003] However, existing human-computer emotion dialogue systems face the following three main technical challenges in achieving this goal: I. The expression of emotional modalities is too simplistic and fails to capture the subtle changes and differences in intensity of emotions.

[0004] Existing emotion recognition systems generally use discrete emotion tags as the output form of emotion modalities, such as simply classifying user emotions into a limited number of categories like "happy," "angry," and "surprised." These methods are mostly based on Ekman's six basic emotion models or Plutchik's emotion wheel theory. Although they have certain advantages in terms of standardization and interpretability, they are essentially coarse-grained and low-dimensional emotion representation methods. Their emotion modality expression is coarse and lacks the ability to describe fine-grained emotions.

[0005] II. The system's insufficient modeling of contextual relationships leads to a lack of coherence in emotional expression.

[0006] In a system, user emotions are not generated in isolation, but rather are the result of gradual accumulation, evolution, and transfer within a continuous interactive context. However, existing emotional dialogue systems have weak contextual connections, relying only on text or speech from adjacent turns, ignoring the accumulation of emotions and semantic connections in long dialogues, resulting in problems such as narrow modeling scope and localized dependencies in context modeling.

[0007] III. Failure to fully consider the moderating effect of contextual factors on emotions resulted in a disconnect between emotional responses and actual interaction scenarios.

[0008] In real-world conversations, a user's current emotional state depends not only on the current input and context, but also on the relevance of the context. However, existing emotional dialogue systems generally employ strategies such as "equal-weighted input throughout history" or "simple time decay," failing to filter historical information for semantic relevance. This results in a disconnect between emotion and context, an inability to distinguish the relevance of emotional context, and susceptibility to interference from irrelevant conversational emotions (e.g., negative emotions from past conversations influencing current positive responses).

[0009] The existence of these problems has severely limited the user experience and practical value of human-computer emotional dialogue systems. Summary of the Invention

[0010] The purpose of this invention is to propose an emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning. This method uses spherical coordinate representation to enhance emotional expression in the acquisition of emotional features, and models multimodal context through heterogeneous subgraphs. It uses subgraph contrastive learning to focus on relevant emotions, thereby facilitating the accurate matching of emotions and context and improving the quality of speech synthesis.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: An emotion-based dialogue speech synthesis method based on heterogeneous subgraph contrastive learning includes the following steps: Step 1. Obtain the dialogue speech dataset and perform audio and text cleaning and preprocessing; Step 2. Perform feature processing on the preprocessed dialogue speech data; Feature processing includes text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition, resulting in multimodal data containing five modalities. In the acquisition of sentiment features, a spherical coordinate representation is used to map discrete sentiment categories to a continuous high-dimensional semantic space; Step 3. Construct a heterogeneous dialogue graph based on the obtained multimodal data to model the complex semantic and emotional dependencies in the dialogue history; the heterogeneous dialogue graph contains five types of nodes and different types of edges; The heterogeneous dialogue graph has five types of nodes, which correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities. Step 4. Based on the constructed heterogeneous dialogue graph, construct positive and sample heterogeneous subgraphs by random masking, and use the heterogeneous subgraphs for comparative learning to extract the most relevant sentiment cues to the current discourse from the dialogue history. Step 5. Based on the contrastive learning-enhanced graph representation, predict the sentiment, intensity, and prosodic features of the current utterance, and input them along with the text content and speaker identity into the acoustic decoder to generate the final emotional speech waveform.

[0012] Furthermore, based on the aforementioned emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning, this invention also proposes a corresponding emotional dialogue speech synthesis system based on heterogeneous subgraph contrastive learning, the technical solution of which is as follows: An emotion-based dialogue speech synthesis system based on heterogeneous subgraph contrastive learning includes the following modules: The dialogue speech data preprocessing module is used to acquire dialogue speech datasets and perform audio and text cleaning and preprocessing. The feature extraction module is used to perform feature processing on the preprocessed dialogue speech data; Feature processing includes text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition, resulting in multimodal data containing five modalities. In the acquisition of sentiment features, a spherical coordinate representation is used to map discrete sentiment categories to a continuous high-dimensional semantic space; The heterogeneous dialogue graph construction module is used to build heterogeneous dialogue graphs based on multimodal data to model complex semantic and emotional dependencies in dialogue history; the heterogeneous dialogue graph contains five types of nodes and different types of edges; The heterogeneous dialogue graph has five types of nodes, which correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities. The contrastive learning module, based on the constructed heterogeneous dialogue graph, constructs positive and sample heterogeneous subgraphs through random masking, and uses the heterogeneous subgraphs for contrastive learning to extract the most relevant emotional cues to the current discourse from the dialogue history. The system also includes an emotion-aware speech synthesis module that predicts the emotion, intensity, and prosodic features of the current utterance based on the contrastive learning-enhanced graph representation. These features are then input into the acoustic decoder along with the text content and speaker identity to generate the final emotional speech waveform.

[0013] The present invention has the following advantages: As described above, this invention discloses an emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning. This method employs spherical coordinate representation when acquiring emotional feature representations, mapping discrete emotional categories to a continuous high-dimensional semantic space. This supports continuous changes in emotional intensity and dimensionality, overcoming the limitations of discrete emotional labels and enriching emotional expression. After feature processing to obtain multimodal data, a heterogeneous dialogue graph is further constructed to model complex semantic and emotional dependencies in the dialogue history. Because the heterogeneous graph dynamically integrates multi-turn dialogue information, it can adapt to the emotional evolution in long dialogue scenarios, exhibiting strong contextual adaptability. Furthermore, positive and sample heterogeneous subgraphs are constructed using random masks, and contrastive learning is performed on these subgraphs. Heterogeneous subgraph contrastive learning can focus on contextual emotional cues, solving the problem of emotional disconnect from context in traditional methods, resulting in high emotional fit. Additionally, when generating the final emotional speech waveform, this invention uses a direct mapping method between emotional parameters and speech acoustic features, avoiding mismatch between emotion and speech features, resulting in a highly natural synthesized emotional speech waveform. Attached Figure Description

[0014] Figure 1 This is a flowchart of the emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This embodiment describes an emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning, in order to solve the problems of simple emotional modality expression, weak contextual correlation, and disconnect between emotion and context in the existing human-computer emotional dialogue process.

[0016] like Figure 1 As shown, the emotion-based dialogue speech synthesis method based on heterogeneous subgraph contrastive learning includes the following steps: Step 1. Obtain the dialogue speech dataset and perform audio and text cleaning and preprocessing.

[0017] In this embodiment, the DailyTalk two-person dialogue voice dataset was obtained through publicly available channels.

[0018] The DailyTalk dataset contains 23,773 audio clips with a total duration of approximately 20 hours. The data format consists of audio files (sampling rate 44.1kHz, bit depth 16bit) and corresponding text transcriptions, resulting in a dialogue speech dataset.

[0019] Data cleaning tools are used to clean the audio and text in the dialogue speech dataset, removing data with noise interference and mismatch between text and audio timing, resulting in a high-quality incremental dialogue speech dataset.

[0020] Step 2. Perform feature processing on the preprocessed dialogue speech data.

[0021] Step 2.1. Definition of dialogue voice data.

[0022] Before performing feature processing on the dialogue speech data, the preprocessed dialogue speech data is defined, including speech waveform data, text transcription data, emotion annotation data, and speaker information.

[0023] The speech waveform data consists of raw audio signals containing multiple rounds of dialogue.

[0024] The text transcription data consists of sentence-by-sentence text content corresponding to the speech waveform data, including speaker identifiers.

[0025] The sentiment labeling data includes the sentiment category and sentiment intensity of each sentence. The sentiment category of each sentence includes, for example, happiness, sadness, anger, disgust, fear, surprise, and neutrality, and the sentiment intensity includes, for example, weak, medium, and strong.

[0026] Speaker information is a unique identifier for each speaker.

[0027] Step 2.2. Feature processing.

[0028] Feature processing includes five parts: text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition. Each feature processing step will be introduced in detail below.

[0029] I. The text feature extraction process is as follows: Text fronting is used to convert text into phoneme sequences, and pre-trained models (such as BERT) are used to extract text embeddings. The formula is expressed as follows: ; in This represents the phoneme sequence corresponding to the text. This indicates the extraction of text embedding representation.

[0030] II. The audio feature extraction process is as follows: Based on the audio sampling rate, professional audio processing tools are used to process the cleaned audio to obtain the corresponding mel spectrum; then, the alignment relationship between the speech mel spectrum segment and the phoneme sequence obtained in the above process is obtained using the Montreal Forced Aligner tool (Textgrid).

[0031] Then, the preprocessed mel spectrum is received using a Global Style Encoder (GST), and the mel spectrum is encoded according to the alignment relationship of the TextGrid to generate the final audio feature representation. The formula is expressed as follows: ; in This represents the mel spectrum sequence corresponding to the audio. This indicates encoding processing.

[0032] III. The process of acquiring emotional features is as follows: A spherical coordinate representation is used to map discrete emotion categories to a continuous high-dimensional semantic space. Specifically, the seven emotion categories are mapped to points on a unit sphere, making semantically similar emotions appear closer together in space.

[0033] Different emotions are encoded using a spherical emotion encoder to obtain emotion embedding vectors. : .

[0034] in, This indicates the emotion category corresponding to the above audio. This indicates that a spherical emotion encoder is used to encode different emotions.

[0035] Specifically, the emotion embedding vector The process of obtaining the data using spherical coordinates is as follows: Step 2.1. Represent the audio features of each batch. enter The encoding module (such as a speech emotion recognition model based on wav2vec2.0) performs linear mapping and outputs three-dimensional Cartesian coordinates. .

[0036] in Indicates wakefulness. Indicates valence, Indicates dominance, with each value range being... .

[0037] Step 2.2. The pre-processed... Enter a neutral emotional audio clip. The encoding module obtains its three-dimensional Cartesian coordinates and calculates the arithmetic mean to generate neutral center coordinates. ;in It is a natural number.

[0038] Step 2.3. Combine all the results obtained in Step 2.1 coordinates minus neutral center coordinates The coordinates after translation with the neutral center as the origin are obtained. .

[0039] Step 2.4. Regarding the above... The spherical transformation is performed as follows: First calculate the radial distance Then calculate the pitch angle. With azimuth Ultimately, a spherical orientation vector reflecting the emotional style is generated. .

[0040] Step 2.5. The projection layer projects the spherical direction, i.e., the spherical direction vector. The mapping yields the emotion projection vector. Then through After activation and layer normalization, the output is the sentiment feature vector, which is the sentiment embedding vector. .

[0041] This invention uses a spherical coordinate representation to map discrete emotion categories to a continuous high-dimensional semantic space, supporting continuous changes in emotion intensity and dimension. This breaks through the limitations of discrete emotion tags and makes emotion expression richer.

[0042] IV. The process of obtaining intensity features is as follows: Assign corresponding trainable high-dimensional embedding vectors to the three intensity labels (strong, weak, and medium), and encode them using an intensity encoder to obtain the sentiment intensity embedding features. The formula is expressed as follows: ; in This indicates that the encoding process is performed using an emotion intensity encoder. It represents the intensity of emotion and is also included in the dataset; it is a modality or a type of node.

[0043] V. The speaker embedding feature acquisition process is as follows: Assign trainable high-dimensional embedding vectors to the two speaker labels (0,1), and encode them using a speaker encoder to obtain speaker embedding features. The formula is expressed as follows: ; The speaker is the person speaking in the aforementioned audio. This indicates that the high-dimensional embedding vector corresponding to the speaker in the above audio is encoded using a speaker encoder.

[0044] After the above feature processing, multimodal data containing five modalities is obtained, including text embedding representation. Audio feature representation Emotional embedding vector Emotional intensity embedding features and speaker embedding features .

[0045] Step 3. Construction of Heterogeneous Conversational Graph (HCG).

[0046] Based on the multimodal data obtained in step 2 above, this invention constructs a heterogeneous dialogue graph to model the complex semantic and emotional dependencies in the dialogue history. The heterogeneous dialogue graph is expressed as follows: .

[0047] in It is a heterogeneous graph. For the set of all nodes, Then it is the set of all edges.

[0048] It contains five types of nodes and different types of edges.

[0049] The five types of nodes correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities.

[0050] Specifically, heterogeneous graphs The construction process is as follows: Step 3.1. Define heterogeneous graph nodes.

[0051] Heterogeneous graphs contain five types of nodes, corresponding to five modalities, namely text nodes. Audio nodes Speaker node Emotional nodes and strength nodes .

[0052] In summary, the node set ={ , , , , }

[0053] Step 3.2. Define the edges of the heterogeneous graph.

[0054] The heterogeneous graph contains 10 types of edges forming an edge set E, including the interaction relationships between any two modalities, such as... Represents the edge between text and speech nodes; Represents the edge between emotion and intensity nodes.

[0055] This invention defines 10 different types of edges to connect nodes of different modes.

[0056] Step 3. Initialize the nodes using data features from different modalities, as follows: = , = , = , = , = .

[0057] Step 4. Based on the constructed heterogeneous dialogue graph, construct positive and sample heterogeneous subgraphs by random masking, and use the heterogeneous subgraphs for comparative learning to extract the most relevant emotional cues from the dialogue history to the current discourse.

[0058] Heterogeneous graph constructed based on step 3 By employing heterogeneous subgraph contrastive learning, we can accurately extract the most relevant emotional cues from the dialogue history, thereby achieving contextual sentiment modeling based on heterogeneous subgraph contrastive learning.

[0059] The heterogeneous subgraph is a subgraph obtained from the above heterogeneous graph by setting a mask rate such as 20% and consisting of only 80% of the nodes.

[0060] Specifically, step 4 is as follows: Step 4.1. Constructing heterogeneous subgraphs for positive and negative samples.

[0061] This invention is based on the constructed heterogeneous graph Heterogeneous subgraphs of positive and negative samples are constructed using random masks.

[0062] Since the dataset is processed in batches, if there are 100 data points, that is, 100 dialogues, these 100 dialogues correspond to 100 heterogeneous graphs. Generally, a batch has 4 dialogues, that is, 4 heterogeneous graphs.

[0063] A heterogeneous graph is constructed from five modalities of a dataset as nodes. Positive samples are 20% of the nodes in the current heterogeneous graph that are masked, since nodes are generally vectors and masking is to set all these vectors to 0.

[0064] The subsequent negative sample operations are the same, but the targets of the operations are other heterogeneous graphs in this batch.

[0065] Regarding the current dialogue Randomly mask 20% of the nodes (e.g., pad some emotion or speech nodes with zeros) to obtain positive samples. Negative samples are generated by randomly sampling from other dialogues and masking 20% ​​of the nodes in the same way. .

[0066] Step 4.2. Use heterogeneous subgraphs for comparative learning.

[0067] The objective of contrastive learning is to extract separately using a graph encoder. (Full image) , Graph-level representation , , And by comparing losses Bring together graph-level features of the same category and , push away and .

[0068] In this embodiment, the graph encoder is, for example, an HGT (Heterogeneous Graph Transformer) graph encoder.

[0069] Heterogeneous graphs, positive sample heterogeneous subgraphs, and negative sample heterogeneous subgraphs are input into the HGT. The graph encoder uses a multi-layer Transformer stack and is based on attention mechanisms and message passing and aggregation to obtain an enhanced graph representation.

[0070] Then, subgraph contrastive learning is used to accurately extract the most relevant emotional cues to the current discourse from the dialogue history.

[0071] Comparative loss The formula is as follows: ; in, For cosine similarity, For temperature coefficient, This represents the number of negative samples.

[0072] By minimizing The forced graph-level representation indicates that, under the meaning of cosine similarity, similar emotions are closer and dissimilar emotions are farther apart.

[0073] Step 4.3. Sensing emotional cues.

[0074] Through contrastive learning, graph encoders can recover the most relevant emotional context to the target discourse from complete heterogeneous graphs. That is, contrastive learning-enhanced graph feature representations to improve the ability to perceive contextual emotions.

[0075] Step 5. Based on the contrastive learning-enhanced graph representation, predict the sentiment, intensity, and prosodic features of the current utterance, and input them along with the text content and speaker identity into the acoustic decoder to generate the final emotional speech waveform.

[0076] Step 5.1. Represent the contrastive learning-enhanced graph. The data is input into the sentiment renderer to predict the sentiment of the current utterance. ,strength and rhythmic features The formula is expressed as follows: , , ( ); in This is the enhanced representation of the graph node corresponding to the current discourse.

[0077] Step 5.2. Express the emotion in the current statement. ,strength and rhythmic features The text content and speaker identity are input into the acoustic decoder to generate the final emotional speech waveform. The formula is expressed as follows: ; in This indicates that the acoustic decoder is processing... For text content, The speaker's identity.

[0078] This invention employs a direct mapping method between emotional parameters and speech acoustic features, effectively avoiding the mismatch between emotion and speech features, thereby achieving audio synthesis of the target speech, and the emotion of the audio is more consistent with the dialogue context.

[0079] In addition, to verify the effectiveness of the method of the present invention, the following specific experiments are also given: The experiment included 8 rounds of open-domain dialogue, with 200 participants in a blind test. The mismatch rate was determined by manual labeling.

[0080] Experiments have shown that the audio generated using the emotional dialogue speech synthesis method of this invention can reduce the "emotion-semantic mismatch rate" from 12%–18% in traditional speech synthesis to less than 3%, thus verifying the effectiveness of the method of this invention.

[0081] Example 2 This embodiment 2 describes an emotional dialogue speech synthesis system based on heterogeneous subgraph contrastive learning, which is based on the same inventive concept as the emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning in embodiment 1 above.

[0082] An emotion-based dialogue speech synthesis system based on heterogeneous subgraph contrastive learning includes the following modules: The dialogue speech data preprocessing module is used to acquire dialogue speech datasets and perform audio and text cleaning and preprocessing. The feature extraction module is used to perform feature processing on the preprocessed dialogue speech data; Feature processing includes text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition, resulting in multimodal data containing five modalities. In the acquisition of sentiment features, a spherical coordinate representation is used to map discrete sentiment categories to a continuous high-dimensional semantic space; The heterogeneous dialogue graph construction module is used to build heterogeneous dialogue graphs based on multimodal data to model complex semantic and emotional dependencies in dialogue history; the heterogeneous dialogue graph contains five types of nodes and different types of edges; The heterogeneous dialogue graph has five types of nodes, which correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities. The contrastive learning module, based on the constructed heterogeneous dialogue graph, constructs positive and sample heterogeneous subgraphs through random masking, and uses the heterogeneous subgraphs for contrastive learning to extract the most relevant emotional cues to the current discourse from the dialogue history. The system also includes an emotion-aware speech synthesis module that predicts the emotion, intensity, and prosodic features of the current utterance based on the contrastive learning-enhanced graph representation. These features are then input into the acoustic decoder along with the text content and speaker identity to generate the final emotional speech waveform.

[0083] It should be noted that any content not mentioned in the above functional modules of this embodiment 2 can be referred to the step description of the emotional dialogue speech synthesis method based on heterogeneous subgraph contrast learning in embodiment 1 above, and will not be repeated in detail here.

[0084] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for synthesizing emotional dialogue speech based on heterogeneous subgraph contrastive learning, characterized in that, Includes the following steps: Step 1. Obtain the dialogue speech dataset and perform audio and text cleaning and preprocessing; Step 2. Perform feature processing on the preprocessed dialogue speech data; Feature processing includes text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition, resulting in multimodal data containing five modalities. In the acquisition of emotional features, a spherical coordinate representation is used to map discrete emotional categories to a continuous high-dimensional semantic space; Step 3. Construct a heterogeneous dialogue graph based on the obtained multimodal data to model the complex semantic and emotional dependencies in the dialogue history; The heterogeneous dialogue graph contains five types of nodes and different types of edges; The heterogeneous dialogue graph has five types of nodes, which correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities. Step 4. Based on the constructed heterogeneous dialogue graph, construct positive and sample heterogeneous subgraphs by random masking, and use the heterogeneous subgraphs for comparative learning to extract the most relevant sentiment cues to the current discourse from the dialogue history. Step 5. Based on the contrastive learning-enhanced graph representation, predict the sentiment, intensity, and prosodic features of the current utterance, and input them along with the text content and speaker identity into the acoustic decoder to generate the final emotional speech waveform.

2. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 1, characterized in that, In step 1, the obtained dialogue speech dataset contains several audio segments; the data format of each audio segment is an audio file and the corresponding text transcription content of the audio file.

3. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 1, characterized in that, In step 2, before performing feature processing on the dialogue speech data, the preprocessed dialogue speech data is first defined. The defined data includes speech waveform data, text transcription data, emotion annotation data, and speaker information. The speech waveform data consists of raw audio signals containing multiple rounds of dialogue; The text transcription data consists of sentence-by-sentence text content corresponding to the speech waveform data, including speaker identifiers; the sentiment annotation data consists of the sentiment category and sentiment intensity of each sentence; and the speaker information is a unique identifier for each speaker.

4. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 1, characterized in that, In step 2, the multimodal data obtained after feature processing includes text embedding representations. Audio feature representation Emotional embedding vector Emotional intensity embedding features and speaker embedding features .

5. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 4, characterized in that, In step 2, the emotion embedding vector The spherical coordinate representation method is used to obtain the data, and the process is as follows: Step 2.

1. Represent the audio features of each batch. enter The encoding module performs a linear mapping and outputs three-dimensional Cartesian coordinates. , For wakefulness, For valence, For dominance, the value ranges are: ; Step 2.

2. The pre-processed... Enter a neutral emotional audio clip. The encoding module obtains its three-dimensional Cartesian coordinates and calculates the arithmetic mean to generate neutral center coordinates. ;in It is a natural number; Step 2.

3. Combine all the results obtained in Step 2.1 coordinates minus neutral center coordinates The coordinates after translation with the neutral center as the origin are obtained. ; Step 2.

4. Regarding the above... The spherical transformation is performed as follows: First calculate the radial distance Then calculate the pitch angle. With azimuth Ultimately, a spherical orientation vector reflecting the emotional style is generated. ; Step 2.

5. The projection layer projects the spherical direction, i.e., the spherical direction vector. The mapping yields the emotion projection vector. Then through After activation and layer normalization, the output is the sentiment feature vector, which is the sentiment embedding vector. .

6. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 4, characterized in that, In step 3, the heterogeneous dialogue graph is expressed in the following form: ; in It is a heterogeneous graph. For the set of all nodes, Then it is the set of all edges; Heterogeneous graph The construction process is as follows: Step 3.

1. Define the heterogeneous graph nodes; Heterogeneous graphs contain five types of nodes, corresponding to five modalities, namely text nodes. Audio nodes Speaker node Emotional nodes and strength nodes ; In summary, the node set ={ , , , , }; Step 3.

2. Define the edges of the heterogeneous graph; The heterogeneous graph consists of 10 types of edges that form the edge set E, including the interaction relationships between any two modalities; Step 3. Initialize the nodes using data features from different modalities, as follows: = 、 = 、 = 、 = 、 = 。 7. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 1, characterized in that, Step 4 specifically involves: Step 4.

1. Based on the constructed heterogeneous graph Heterogeneous subgraphs of positive and negative samples are constructed using random masks; Regarding the current dialogue Randomly masking 20% ​​of the nodes yields a heterogeneous subgraph of positive samples. Randomly sample 20% of nodes from other dialogues and mask them in the same way to form a negative sample heterogeneous subgraph. ; Step 4.

2. Use heterogeneous subgraphs for comparative learning; The contrastive learning objective is to extract the graph encoder separately. , , Graph-level representation , , And by comparing losses Bring together graph-level features of the same category and , push away and ; Step 4.

3. Sensing Emotional Cues: Through contrastive learning, the graph encoder is able to recover the most relevant emotional context to the target discourse from a complete heterogeneous graph. That is, contrastive learning-enhanced graph feature representations to improve the ability to perceive contextual emotions.

8. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 7, characterized in that, In step 4.2, the comparison loss is performed. The formula is as follows: ； in, For cosine similarity, For temperature coefficient, This represents the number of negative samples.

9. The emotional dialogue speech synthesis method based on heterogeneous subgraph contrastive learning according to claim 1, characterized in that, Step 5 specifically involves: Step 5.

1. Represent the contrastive learning-enhanced graph. The data is input into the sentiment renderer to predict the sentiment of the current utterance. ,strength and rhythmic features The formula is expressed as follows: , , ( )； in This is the enhanced representation of the graph node corresponding to the current discourse. Step 5.

2. Express the emotion in the current statement. ,strength and rhythmic features The text content and speaker identity are input into the acoustic decoder to generate the final emotional speech waveform. The formula is expressed as follows: ； in This indicates that the acoustic decoder is processing... For text content, The speaker's identity.

10. An emotion-based dialogue speech synthesis system based on heterogeneous subgraph contrastive learning, characterized in that, Includes the following modules: The dialogue speech data preprocessing module is used to acquire dialogue speech datasets and perform audio and text cleaning and preprocessing. The feature extraction module is used to perform feature processing on the preprocessed dialogue speech data; Feature processing includes text feature extraction, audio feature extraction, sentiment feature acquisition, intensity feature acquisition, and speaker embedding feature acquisition, resulting in multimodal data containing five modalities. In the acquisition of emotional features, a spherical coordinate representation is used to map discrete emotional categories to a continuous high-dimensional semantic space; The heterogeneous dialogue graph construction module is used to build heterogeneous dialogue graphs based on multimodal data to model the complex semantic and emotional dependencies in the dialogue history. The heterogeneous dialogue graph contains five types of nodes and different types of edges; The heterogeneous dialogue graph has five types of nodes, which correspond to five modal information types: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; different types of edges are used to connect nodes of different modalities. The contrastive learning module, based on the constructed heterogeneous dialogue graph, constructs positive and sample heterogeneous subgraphs through random masking, and uses the heterogeneous subgraphs for contrastive learning to extract the most relevant emotional cues to the current discourse from the dialogue history. The system also includes an emotion-aware speech synthesis module that predicts the emotion, intensity, and prosodic features of the current utterance based on the contrastive learning-enhanced graph representation. These features are then input into the acoustic decoder along with the text content and speaker identity to generate the final emotional speech waveform.

Images