Heterogeneous graph context modeling-based dialogue type speech synthesis emotion rendering method and device

Abstract

The invention relates to the technical field of emotion recognition, and discloses an emotion rendering method and device for dialogue speech synthesis based on heterogeneous graph context modeling, equipment and a medium, and the method comprises the steps: receiving multi-source information; arranging each node set and an edge set of the relationship between the two nodes, and constructing an emotion perception graph based on the node sets and the edge sets; extracting a target node in the emotion perception graph and all neighbor source nodes of the target node, mapping the target node into a query vector based on a heterogeneous mutual attention mechanism, mapping the neighbor source nodes into a key vector, and calculating a dot product of the query vector and the key vector as attention; the method comprises the following steps: calculating dependency information transmitted from a neighbor source node to a target node, projecting a feature vector of the neighbor source node into a message vector by using linear projection, integrating edge dependencies of the neighbor source node and the target node based on the dependency information, combining the edge dependencies into a matrix, and connecting all preset number of message vectors. The above technical means enhances emotion understanding.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to an emotion rendering method, device, equipment and medium for conversational speech synthesis based on heterogeneous graph context modeling. Background Art

[0002] Conversational speech synthesis aims to accurately represent speech in a conversational environment, including appropriate prosody and emotional coloring. In human-computer dialogue, modeling emotional expressions in the context of dialogue is crucial for speech synthesis systems to generate speech with appropriate emotional states and improve the speech-based interactive experience. Although the importance of this task has been recognized, previous studies have been largely lacking in emotion understanding and emotion rendering due to the scarcity of emotional dialogue datasets and the difficulty of state emotion modeling. Summary of the invention

[0003] The present invention provides a method, device, computer equipment and medium for emotion rendering of conversational speech synthesis based on heterogeneous graph context modeling, so as to solve the technical problems of scarcity of emotional dialogue data sets and difficulty in state emotion modeling, and large deficiencies in emotion understanding and emotion rendering.

[0004] In a first aspect, a method for emotion rendering of conversational speech synthesis based on heterogeneous graph context modeling is provided, comprising:

[0005] Receiving multi-source information, wherein the multi-source information includes five types of nodes: a text node, an audio node, a speaker node, a sentiment node, and an intensity node;

[0006] Arrange each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set;

[0007] Extract the target node and all neighbor source nodes of the target node in the sentiment perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0008] Calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node;

[0009] Aggregate the calculated attention scores and message vectors, use the attention scores as weights, average the corresponding messages of all neighbor source nodes, and obtain the sentiment enhancement vector of the target node;

[0010] The emotion and emotion intensity feature representation of the current utterance are predicted based on the emotion enhancement vector of the target node.

[0011] In a second aspect, an emotional rendering device for conversational speech synthesis based on heterogeneous graph context modeling is provided, comprising:

[0012] A receiving module, used for receiving multi-source information, wherein the multi-source information includes five types of nodes: text node, audio node, speaker node, emotion node and intensity node;

[0013] A sorting module is used to sort each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set;

[0014] The extraction module is used to extract the target node and all the neighbor source nodes of the target node in the sentiment perception graph, map the target node into a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node into a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0015] A calculation module, used to calculate the dependency information transmitted from the neighbor source node to the target node, project the feature vector of the neighbor source node into a message vector using linear projection, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node;

[0016] Aggregation module, used to aggregate the calculated attention scores and message vectors, taking the attention scores as weights, averaging the corresponding messages of all neighbor source nodes, and obtaining the sentiment enhancement vector of the target node;

[0017] The prediction module is used to predict the emotion and emotion intensity feature representation of the current utterance based on the emotion enhancement vector of the target node.

[0018] According to a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling are implemented.

[0019] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the emotion rendering method for conversational speech synthesis based on heterogeneous graph context modeling are implemented.

[0020] In the scheme implemented by the above-mentioned emotional rendering method, device, computer equipment and storage medium for conversational speech synthesis based on heterogeneous graph context modeling, multi-source information can be received, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes and intensity nodes; each node set and the edge set of the relationship between two nodes are sorted, and an emotional perception graph is constructed based on the node set and the edge set; the target node and all neighbor source nodes of the target node in the emotional perception graph are extracted, and the target node is mapped to a query vector based on the heterogeneous mutual attention mechanism, and the neighbor source node is mapped to a key vector, and the calculation is performed. The dot product of the query vector and the key vector is used as attention; the dependency information passed from the neighbor source node to the target node is calculated, and the feature vector of the neighbor source node is projected into a message vector using linear projection. The edge dependencies between the neighbor source node and the target node are integrated based on the dependency information and combined into a matrix, and all the preset number of message vectors are connected to obtain the final message vector of each node; the calculated attention scores and message vectors are aggregated, and the attention scores are used as weights to average the corresponding messages of all neighbor source nodes, and the target node's emotion enhancement vector is obtained; the emotion and emotion intensity feature representation of the current utterance are predicted based on the emotion enhancement vector of the target node. Through the above technical means, multi-source conversation history is used as input to model the conversation context and learn emotional clues from the context to enhance emotion understanding; an emotion renderer module based on contrastive learning is adopted to infer the accurate emotional style of the target utterance and realize emotion rendering; emotion labels by category and intensity are created, and additional emotion information is annotated on the existing conversation dataset; the problem of data scarcity is solved; both objective and subjective evaluations show that the model outperforms the baseline model in understanding and rendering emotions. These evaluations also emphasize the importance of comprehensive emotion annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 It is a schematic diagram of an application environment of an emotion rendering method for conversational speech synthesis based on heterogeneous graph context modeling in one embodiment of the present invention;

[0023] Figure 2 It is a flow chart of a method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling in one embodiment of the present invention;

[0024] Figure 3It is a structural schematic diagram of an emotion rendering device for conversational speech synthesis based on heterogeneous graph context modeling in one embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can receive multi-source information, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes; organize each node set and the edge set of the relationship between two nodes, and construct an emotion perception graph based on the node set and the edge set; extract the target node and all neighbor source nodes of the target node in the emotion perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention; calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependency of the neighbor source node and the target node based on the dependency information and combine them into a matrix, connect all the preset number of message vectors to obtain the final message vector of each node; aggregate the calculated attention score and the message vector, use the attention score as a weight, average the corresponding messages of all neighbor source nodes, and obtain the emotion enhancement vector of the target node; predict the emotion and emotion intensity feature representation of the current discourse based on the emotion enhancement vector of the target node. Through the above-mentioned technical means, multi-source conversation history is used as input to model the conversation context and learn emotional clues from the context to enhance emotional understanding; an emotional renderer module based on contrastive learning is adopted to infer the accurate emotional style of the target discourse and realize emotional rendering; emotional labels by category and intensity are created, and additional emotional information is annotated on the existing conversation data set; the problem of data scarcity is solved; both objective and subjective evaluations show that the model is superior to the baseline model in understanding and rendering emotions. These evaluations also emphasize the importance of comprehensive emotional labeling. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0029] See also Figure 2 As shown, Figure 2 A flow chart of a method for emotion rendering of conversational speech synthesis based on heterogeneous graph context modeling provided by an embodiment of the present invention includes the following steps:

[0030] S10: receiving multi-source information, wherein the multi-source information includes five types of nodes: a text node, an audio node, a speaker node, a sentiment node, and an intensity node;

[0031] A dialogue can be defined as a series of utterances (utt1, utt2, ..., uttJ, uttC), where {utt1, utt2, ..., uttJ} is the dialogue history up to the Jth round, and uttC represents the current utterance to be synthesized. The task of affective conversational speech synthesis aims to synthesize audio a given uttC and the dialogue history {utt1, utt2, ..., uttJ}. c For multimodal contexts, each utterance uttj(j∈[1,J]) in the conversation history can be represented by a five-tuple, such as<textj,speakerj,audioj,emotionj,emotion intensityj> , referred to as<uj,sj,aj,ej,ij> Note that uttC can be represented by only two tuples, such as<textC,speakerC> , referred to as<uC,sC> , because the sentiment and intensity information needs to be generated by the model.

[0032] Among them, the speaker node, audio node and text node introduce basic dialogue attributes respectively; the emotion node and intensity node introduce dynamic emotion features and connect the emotional interactions between remote discourses.

[0033] We introduce multi-source knowledge, including 5 types of nodes: text fu, audio fa, speaker fs, emotion fe, and intensity fi, and construct an emotional conversation graph or ECG G = (N, E), where N represents a set of nodes and E represents a set of edges representing the relationship between two nodes. Note that the speaker, audio, and text nodes are designed to introduce basic conversation properties, while the emotion and intensity nodes can introduce dynamic emotional features and connect emotional interactions between remote utterances.

[0034] Considering the multi-source knowledge, 14 different types of edges are created, however, due to space limitations, not all nodes’ edges are depicted. In short, these 14 edges connect 1) text with every other node, 2) audio with speaker nodes, 3) sentiment with speaker, sentiment intensity, audio nodes, and 4) sentiment intensity with speaker, audio nodes. Note that all edges include past-to-future connections and future-to-past connections to simulate bidirectional relationships.

[0035] S20: sorting each node set and the edge set of the relationship between two nodes, and constructing a sentiment perception graph based on the node set and the edge set;

[0036] Initialize all nodes and extract text nodes, speaker nodes, audio nodes, emotion nodes, and intensity nodes from multi-source information based on encoders. In order to achieve meaningful heterogeneous graph encoding, all nodes need to be initialized with their feature representations. Taking multi-round dialogue as an example, we use various encoders to obtain text nodes fuj, speaker nodes fsj, audio nodes faj, emotion nodes fej, and intensity nodes fij (j∈[1,4]) for text, speaker, audio, emotion, and intensity nodes.

[0037] Text nodes. We use the pre-trained BERT model to extract language features: fuj = BERT(uj).

[0038] Audio nodes. We use the Global Style Token (GST) module, which consists of a reference encoder and style token layer, as an audio encoder to extract the acoustic features contained in each audio a: faj = GST(aj).

[0039] Speaker, emotion, and emotion intensity nodes. The speaker, emotion, and intensity encoders are used to define three randomly initialized trainable parameter matrices fsj, fej, and fij to learn two speaker identity features, seven emotion label (happy, sad, angry, disgusted, fearful, surprised, neutral) features, and three emotion intensity label (weak, medium, strong) features.

[0040] Note that the text and speaker nodes fuC, fiC of the current utterance are initialized in the same way as the nodes fuj and fsj in the dialogue history.

[0041] S30: extract the target node and all neighbor source nodes of the target node in the sentiment perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0042] After initializing the constructed graph, a heterogeneous graph encoding module is used to encode the emotional clues in the conversation context to obtain a graph-enhanced representation of each node. Inspired by the heterogeneous graph transformer (HGT), we also adopt a three-stage emotional HGT network containing heterogeneous mutual attention (HMA), heterogeneous message passing (HMP), and emotional knowledge aggregation (EKA) operations to model the dependencies in emotional conversations. Assuming that a node in the heterogeneous ECG is a target node, any type of node can be regarded as a source node. The goal of HGT is to aggregate the information of the source nodes to obtain a contextualized representation of the target node.

[0043] Among them, a preset number of attentions are connected together to obtain the attention vector of each node pair; for each target node, all attention vectors of its neighboring source nodes are collected, and the final attention score is obtained based on softmax.

[0044] Given a target node and all its neighbor source nodes The heterogeneous mutual attention (HMA) mechanism will Mapped into a query vector, is mapped to a key vector and their dot product is calculated as the attention, which indicates the importance of each source node to the target node. Then we connect the h attention heads together to obtain the attention vector for each node pair ( express and For each target node We collect its neighbors All attention vectors of are combined and softmax is performed to obtain the final attention score.

[0045] S40: Calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, connect all the preset number of message vectors to obtain the final message vector of each node;

[0046] Heterogeneous Message Passing (HMP) parallel computing to transfer dependency information from Deliver to Specifically, for a pair of nodes HMP uses linear projection to The feature vector of is projected into the message vector, and then through a matrix To integrate the edge dependencies between neighbor source nodes and target nodes. The last step is to concatenate all h message headers to obtain the final message vector for each node pair.

[0047] S50: Aggregate the calculated attention score and message vector, use the attention score as a weight, average the corresponding messages of all neighbor source nodes, and obtain the emotion enhancement vector of the target node;

[0048] Emotional Knowledge Aggregation (EKA) aims to aggregate the calculated attention scores and message vectors. We use the attention scores as weights and average the The corresponding message of the target node Emotion Enhancement Vector It is worth noting that all ECG nodes, including emotion and intensity nodes, can be regarded as source nodes or target nodes to learn their high-level contextual information. Therefore, the EKA operation eventually integrates the emotion information in the conversation history into all ECG nodes. The graph-enhanced feature representation of each node incorporates the emotion clues in the context.

[0049] In this way, the ECG encoding module can obtain the final emotion perception graph enhanced feature representation f′ of all text, speaker, audio, emotion, and intensity nodes. u , f′ s , f′ a , f′ e and f′ i , these can be fed into subsequent models to provide sentiment information.

[0050] Integrate the emotional information in the conversation history into all nodes of the emotional perception graph. The intensity feature of each node combines the emotional clues in the context. Obtain the enhanced features of the final emotional perception graph of all nodes. The final enhanced features of all nodes are input into the subsequent model to provide emotional information. The ECG encoding module can obtain the final emotional perception graph enhanced feature representation f′ of all text, speakers, audio, emotion and intensity nodes. u , f′ s , f′ a , f′ e and f′ i , these can be fed into subsequent models to provide sentiment information.

[0051] S60: Predict the emotion and emotion intensity feature representation of the current utterance based on the emotion enhancement vector of the target node.

[0052] The emotional conversational speech synthesizer consists of four components: text encoder, speaker encoder, emotion renderer, and acoustic decoder. The text and speaker encoders are designed to encode the current utterance. and speaker identity Note that the sentiment renderer attempts to predict the sentiment of the current utterance. strength and rhythmic features Use graph-enhanced node features. In order to obtain a robust feature representation of the current utterance, the feature aggregator module sets a set of trainable weight parameters for the above five features and then outputs the final mixed feature For the acoustic decoder, we use FastSpeech2 as the backbone, including the variational adapter, mel decoder, and vocoder. to predict duration, energy, and pitch. The Mel decoder aims to predict the Mel spectrum features. Finally, the well-trained HiFi-GAN is used as a vocoder to generate a speech waveform with the desired emotional style. c .

[0053] Note that to achieve emotion rendering, the emotion renderer extracts encoded node features from ECG and predicts the emotion and emotion intensity feature representations of the current utterance while performing prosody prediction. More importantly, to achieve accurate emotion category and emotion intensity feature representation prediction, a contrastive learning-based emotion and emotion intensity loss function is proposed.

[0054] The emotion renderer consists of emotion, intensity, and prosody predictors. The prosody predictor uses a multi-head attention layer to infer the speaking prosody information of the current utterance from the feature representations of text nodes in the dialogue history. The features of audio nodes are not used because we believe that audio and text modalities have interacted during ECG encoding, so text nodes already contain audio information. We use mean squared error (MSE) loss to constrain the training of the prosody predictor, with the target obtained from the GST-based prosody extractor. Next, the emotion and intensity predictors.

[0055] Sentiment predictor, which uses the encoded features of sentiment nodes in the dialogue history to infer the sentiment representation of the current sentence It consists of two convolutional layers, a bidirectional LSTM layer, and two fully connected layers.

[0056]

[0057] Here, f′ e Represents the feature representation of all emotion type nodes in the conversation history after ECG encoding.

[0058] The intensity predictor uses the encoded features of the intensity nodes in the dialogue history to infer the sentiment intensity representation of the current sentence. It consists of two convolutional layers, a bidirectional LSTM layer, two fully connected layers, and a mean pooling layer.

[0059]

[0060] Here, f′ i Represents the common representation of all emotion intensity nodes in the conversation history after ECG encoding.

[0061] Contrastive learning training criteria, the above criteria are designed for the emotion and intensity predictors of the emotional renderer, the purpose is to use the contrastive learning loss function supervised by emotion To motivate the emotional renderer to better distinguish different emotion categories and intensity levels. Specifically, the loss function of contrastive learning is divided into emotion categories and emotional intensity The core idea of ​​this training method is to regard all samples with the same emotion category or intensity label in a batch of data as positive samples, and samples with different labels as negative samples.

[0062] For emotional features A batch containing K sentiment representations is represented as Contrastive learning loss function The calculation method is as follows:

[0063]

[0064] where sim(·,·) is a cosine similarity function. is a scalar temperature parameter. Contains all except External There are the same sentiment labels in a batch of data All positive samples of . Similarly, The loss function as follows:

[0065]

[0066] Finally, the overall loss function is:

[0067]

[0068] in, Refers to the acoustic feature loss of traditional FastSpeech2, including pitch, energy, period and mel spectrum. Refers to the MSE loss of the prosody predictor.

[0069] It can be seen that in the above scheme, for emotion recognition, multi-source information is first received, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes and intensity nodes; each node set and the edge set of the relationship between two nodes are sorted, and an emotion perception graph is constructed based on the node set and the edge set; the target node and all neighbor source nodes of the target node in the emotion perception graph are extracted, and the target node is mapped to a query vector based on the heterogeneous mutual attention mechanism, and the neighbor source node is mapped to a key vector, and the dot product of the query vector and the key vector is calculated as attention; the dependency information transmitted from the neighbor source node to the target node is calculated, and the feature vector of the neighbor source node is projected into a message vector using linear projection, and the edge dependency of the neighbor source node and the target node is integrated and combined into a matrix based on the dependency information, and all the preset number of message vectors are connected to obtain the final message vector of each node; the calculated attention score and message vector are aggregated, and the attention score is used as a weight to average the corresponding messages of all neighbor source nodes, and the emotion enhancement vector of the target node is obtained; the emotion and emotion intensity feature representation of the current discourse are predicted based on the emotion enhancement vector of the target node. Through the above technical means, multi-source conversation history is used as input to model the conversation context and learn emotional clues from the context to enhance emotional understanding; a contrastive learning-based emotional renderer module is used to infer the accurate emotional style of the target discourse and achieve emotional rendering; emotional labels by category and intensity are created, and additional emotional information is annotated on the existing conversation dataset; the problem of data scarcity is solved; both objective and subjective evaluations show that the model is superior to the baseline model in understanding and rendering emotions. These evaluations also emphasize the importance of comprehensive emotional annotation.

[0070] The non-Company software tools or components appearing in the embodiments of the present invention are merely examples and do not represent actual use.

[0071] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0072] In one embodiment, a device for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling is provided, and the device for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling corresponds one-to-one to the method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling in the above embodiment. Figure 3 As shown, the emotional rendering device for conversational speech synthesis based on heterogeneous graph context modeling includes a receiving module 101, a sorting module 102, an extraction module 103, a calculation module 104, an aggregation module 105, and a prediction module 106. The functional modules are described in detail as follows:

[0073] The receiving module 101 is used to receive multi-source information, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes;

[0074] The sorting module 102 is used to sort each node set and the edge set of the relationship between two nodes, and construct a sentiment perception graph based on the node set and the edge set;

[0075] An extraction module 103 is used to extract a target node and all neighbor source nodes of the target node in the emotion perception graph, map the target node to a query vector based on a heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0076] A calculation module 104 is used to calculate the dependency information transmitted from the neighbor source node to the target node, project the feature vector of the neighbor source node into a message vector using linear projection, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node;

[0077] Aggregation module 105, used to aggregate the calculated attention score and message vector, use the attention score as a weight, average the corresponding messages of all neighbor source nodes, and obtain the emotion enhancement vector of the target node;

[0078] The prediction module 106 is used to predict the emotion and emotion intensity feature representation of the current speech based on the emotion enhancement vector of the target node.

[0079] In one embodiment, the receiving module 101 is specifically configured to:

[0080] The speaker node, audio node, and text node introduce basic conversation attributes respectively;

[0081] Emotion nodes and intensity nodes introduce dynamic emotion features and connect the emotional interactions between remote utterances.

[0082] In one embodiment, the sorting module 102 is specifically used for:

[0083] Initialize all nodes and extract text nodes, speaker nodes, audio nodes, emotion nodes, and intensity nodes from multi-source information based on the encoder.

[0084] In one embodiment, the extraction module 103 is used to:

[0085] Concatenate a preset number of attentions together to obtain an attention vector for each node pair;

[0086] For each target node, all attention vectors of its neighbor source nodes are collected and the final attention score is obtained based on softmax.

[0087] In one embodiment, the calculation module 104 is specifically configured to:

[0088] Concatenate all h message headers to obtain the final message vector for each node pair.

[0089] In one embodiment, the aggregation module 105 is specifically configured to:

[0090] Integrate the emotional information in the conversation history into all nodes of the emotion perception graph. The intensity feature of each node combines the emotional clues in the context.

[0091] The enhanced features of the final emotion perception graph of all nodes are obtained, and the final enhanced features of all nodes are input into the subsequent model to provide emotion information.

[0092] In one embodiment, the device is further configured to:

[0093] The node loss function is calculated based on the mean square error loss, and the node loss function is used to distinguish the emotion category and intensity.

[0094] The present invention provides an emotion rendering device for conversational speech synthesis based on heterogeneous graph context modeling. The device first receives multi-source information, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes and intensity nodes; organizes each node set and the edge set of the relationship between two nodes, and constructs an emotion perception graph based on the node set and the edge set; extracts the target node and all neighbor source nodes of the target node in the emotion perception graph, maps the target node to a query vector based on a heterogeneous mutual attention mechanism, maps the neighbor source node to a key vector, and calculates the dot product of the query vector and the key vector as attention; calculates the dependency information transmitted from the neighbor source node to the target node, projects the feature vector of the neighbor source node into a message vector using linear projection, integrates the edge dependency of the neighbor source node and the target node based on the dependency information and combines them into a matrix, connects all the preset number of message vectors to obtain the final message vector of each node; aggregates the calculated attention score and the message vector, uses the attention score as a weight, averages the corresponding messages of all neighbor source nodes, and obtains the emotion enhancement vector of the target node; predicts the emotion and emotion intensity feature representation of the current discourse based on the emotion enhancement vector of the target node. Through the above technical means, multi-source conversation history is used as input to model the conversation context and learn emotional clues from the context to enhance emotional understanding; a contrastive learning-based emotional renderer module is used to infer the accurate emotional style of the target discourse and achieve emotional rendering; emotional labels by category and intensity are created, and additional emotional information is annotated on the existing conversation dataset; the problem of data scarcity is solved; both objective and subjective evaluations show that the model is superior to the baseline model in understanding and rendering emotions. These evaluations also emphasize the importance of comprehensive emotional annotation.

[0095] For the specific definition of the emotion rendering device for conversational speech synthesis based on heterogeneous graph context modeling, please refer to the definition of the emotion rendering method for conversational speech synthesis based on heterogeneous graph context modeling above, which will not be repeated here. Each module in the above-mentioned emotion rendering device for conversational speech synthesis based on heterogeneous graph context modeling can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0096] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps of the server side of a method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling.

[0097] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling are implemented.

[0098] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0099] Receiving multi-source information, wherein the multi-source information includes five types of nodes: a text node, an audio node, a speaker node, a sentiment node, and an intensity node;

[0100] Arrange each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set;

[0101] Extract the target node and all neighbor source nodes of the target node in the sentiment perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0102] Calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node;

[0103] Aggregate the calculated attention scores and message vectors, use the attention scores as weights, average the corresponding messages of all neighbor source nodes, and obtain the sentiment enhancement vector of the target node;

[0104] The emotion and emotion intensity feature representation of the current utterance are predicted based on the emotion enhancement vector of the target node.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0106] Receiving multi-source information, wherein the multi-source information includes five types of nodes: a text node, an audio node, a speaker node, a sentiment node, and an intensity node;

[0107] Arrange each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set;

[0108] Extract the target node and all neighbor source nodes of the target node in the sentiment perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention;

[0109] Calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node;

[0110] Aggregate the calculated attention scores and message vectors, use the attention scores as weights, average the corresponding messages of all neighbor source nodes, and obtain the sentiment enhancement vector of the target node;

[0111] The emotion and emotion intensity feature representation of the current utterance are predicted based on the emotion enhancement vector of the target node.

[0112] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling, characterized in that: include: Receiving multi-source information, wherein the multi-source information includes five types of nodes: a text node, an audio node, a speaker node, a sentiment node, and an intensity node; Arrange each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set; Extract the target node and all neighbor source nodes of the target node in the sentiment perception graph, map the target node to a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node to a key vector, and calculate the dot product of the query vector and the key vector as attention; Calculate the dependency information transmitted from the neighbor source node to the target node, use linear projection to project the feature vector of the neighbor source node into a message vector, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node; Aggregate the calculated attention scores and message vectors, use the attention scores as weights, average the corresponding messages of all neighbor source nodes, and obtain the sentiment enhancement vector of the target node; The emotion and emotion intensity feature representation of the current utterance are predicted based on the emotion enhancement vector of the target node.

2. The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling according to claim 1, characterized in that: The step of receiving multi-source information, wherein the multi-source information includes five types of nodes: text nodes, audio nodes, speaker nodes, emotion nodes, and intensity nodes, comprises: The speaker node, audio node, and text node introduce basic conversation attributes respectively; Emotion nodes and intensity nodes introduce dynamic emotion features and connect the emotional interactions between remote utterances.

3. The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling as claimed in claim 2, characterized in that: Before the step of arranging each node set and the edge set of the relationship between two nodes and constructing an emotional dialogue graph based on the node set and the edge set, the method further includes: Initialize all nodes and extract text nodes, speaker nodes, audio nodes, emotion nodes, and intensity nodes from multi-source information based on the encoder.

4. The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling as claimed in claim 1, characterized in that: The steps of extracting the target node and all neighbor source nodes of the target node in the emotional dialogue graph, mapping the target node to a query vector based on the heterogeneous mutual attention mechanism, mapping the neighbor source node to a key vector, and calculating the dot product of the query vector and the key vector as the attention include: Concatenate a preset number of attentions together to obtain an attention vector for each node pair; For each target node, all attention vectors of its neighbor source nodes are collected and the final attention score is obtained based on softmax.

5. The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling as claimed in claim 1, characterized in that: After the steps of calculating dependency information transmitted from the neighbor source node to the target node, projecting the feature vector of the neighbor source node into a message vector using linear projection, integrating the edge dependencies between the neighbor source node and the target node based on the dependency information and combining them into a matrix, and connecting all the preset number of message vectors to obtain the final message vector of each node, the method further includes: Concatenate all h message headers to obtain the final message vector for each node pair.

6. The emotional rendering method for conversational speech synthesis based on heterogeneous graph context modeling as claimed in claim 1, characterized in that: After the step of aggregating the calculated attention scores and message vectors, taking the attention scores as weights, averaging the corresponding messages of all neighbor source nodes, and obtaining the emotion enhancement vector of the target node, the method further includes: Integrate the emotional information in the conversation history into all nodes of the emotion perception graph. The intensity feature of each node combines the emotional clues in the context. The enhanced features of the final emotion perception graph of all nodes are obtained, and the final enhanced features of all nodes are input into the subsequent model to provide emotion information.

7. The method for emotional rendering of conversational speech synthesis based on heterogeneous graph context modeling according to claim 1, characterized in that: After the step of predicting the emotion and emotion intensity feature representation of the current speech based on the emotion enhancement vector of the target node, the method further includes: The node loss function is calculated based on the mean square error loss, and the node loss function is used to distinguish the emotion category and intensity.

8. An emotional rendering device for conversational speech synthesis based on heterogeneous graph context modeling, characterized in that: include: A receiving module, used for receiving multi-source information, wherein the multi-source information includes five types of nodes: text node, audio node, speaker node, emotion node and intensity node; A sorting module is used to sort each node set and the edge set of the relationship between two nodes, and build a sentiment perception graph based on the node set and the edge set; The extraction module is used to extract the target node and all the neighbor source nodes of the target node in the sentiment perception graph, map the target node into a query vector based on the heterogeneous mutual attention mechanism, map the neighbor source node into a key vector, and calculate the dot product of the query vector and the key vector as attention; A calculation module, used to calculate the dependency information transmitted from the neighbor source node to the target node, project the feature vector of the neighbor source node into a message vector using linear projection, integrate the edge dependencies between the neighbor source node and the target node based on the dependency information and combine them into a matrix, and connect all the preset number of message vectors to obtain the final message vector of each node; Aggregation module, used to aggregate the calculated attention scores and message vectors, taking the attention scores as weights, averaging the corresponding messages of all neighbor source nodes, and obtaining the sentiment enhancement vector of the target node; The prediction module is used to predict the emotion and emotion intensity feature representation of the current utterance based on the emotion enhancement vector of the target node.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for emotion rendering of conversational speech synthesis based on heterogeneous graph context modeling as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for emotion rendering of conversational speech synthesis based on heterogeneous graph context modeling as described in any one of claims 1 to 7 are implemented.

Images