Method for multi-modal dialogue sentiment recognition based on progressive attention and loop-free graph

Abstract

This invention provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, belonging to the field of emotion recognition technology. Based on heterogeneous relational directed acyclic layers, this invention explicitly models the diverse dependencies between utterances, thereby more accurately capturing the dynamic evolution of emotions in dialogue and distinguishing different types of contextual relationships. It solves the technical problem of existing multimodal dialogue emotion recognition methods based on graph neural networks lacking the ability to distinguish dialogue dynamics and different types of contextual relationships, significantly improving the accuracy and robustness of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs. Background Technology

[0002] Multimodal dialogue emotion recognition is an intelligent technology that integrates multi-dimensional information and is widely used in various scenarios such as human-computer interaction, customer service, and mental health monitoring. Its core requirement is to accurately capture the complex and diverse emotional expressions in real human interactions. By integrating multimodal information such as voice, text, and facial expressions in dialogues, it provides reliable emotional basis for subsequent intelligent response generation and decision support, helping to improve the interactive experience and service quality in various scenarios.

[0003] Currently, in the field of multimodal dialogue emotion recognition, common technical approaches mainly rely on graph neural networks. When dealing with dialogue context relationships, these methods usually adopt a unified approach, treating all context relationships in the dialogue as homogeneous information. By constructing a fixed graph structure, they achieve the transmission and aggregation of context information, thereby completing the emotion recognition task.

[0004] However, existing multimodal dialogue emotion recognition methods based on graph neural networks lack the ability to distinguish dialogue dynamics and different contextual relationship types.

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, which solves the technical problem that existing multimodal dialogue emotion recognition methods based on graph neural networks lack the ability to distinguish dialogue dynamics and different types of contextual relationships.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs. Emotion recognition is performed using a pre-trained emotion recognition model based on progressive attention and acyclic graphs. The emotion recognition model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation-oriented acyclic graph layer, a relation-aware graph attention layer, and a classification layer. The multimodal dialogue emotion recognition method includes: Acquire text, speech, and video modal data from the dialogue to be emotion-recognized, and segment the text, speech, and video modal data by conversation to form multiple utterances; Multiple utterances are preprocessed and cross-modal aligned to obtain text data, speech data, and video data of the multiple utterances; The text data, speech data, and video data are extracted using a single-modal feature extraction layer to obtain three single-modal features for each utterance. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector for each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; By leveraging the relation-aware graph attention layer and fusion feature vectors at the discourse level, Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance is then gated and updated with the corresponding context message to obtain the node representation of each utterance containing context dependencies. By processing the context-dependent node representations of each utterance through a classification layer, the probability distribution of the sentiment category of each utterance is obtained. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

[0007] Preferably, the relationship types include temporal continuation, semantic similarity, semantic transition, causal relationship, and same or different speakers.

[0008] Preferably, the step of fusing the three single-modal features through a hierarchical fusion layer to obtain the discourse-level fusion feature vector for each discourse includes: The hierarchical fusion layer based on progressive attention includes L stacked progressive fusion units. Each progressive fusion unit contains two stages of attention interaction and feature update process. Taking visual features as an example, it includes: The first stage employs visual-text cross-modal attention interaction, using visual features as queries and text features as keys and values. A multi-head attention mechanism is used to compute the information supplementation of visual expression by textual semantics, thereby obtaining the discourse. Text-enhanced visual representation ; The second stage utilizes visual-audio cross-modal attention interaction, building upon the first stage by enhancing the visual features of the text. As a new query, with audio features To obtain a second interaction, we need to get the dialogue. The dual-enhanced visual representation ; Will The input is transformed into a feedforward neural network, and the visual output of the current progressive attention layer is obtained through residual connections. : Text features, like audio features, employ a two-stage cross-modal attention interaction, updated synchronously with visual features, to obtain the first... The words output by the layer Text output and audio output ; Output the audio from layer L. Text output and visual output By piecing them together, we can obtain the words. discourse-level fusion feature vector .

[0009] Preferably, the symphony-level fusion feature vector pair based on each utterance is obtained through the relation-aware graph attention layer. Each node in the process performs context information aggregation to obtain the context message vector for each utterance, including: Calculate the directed acyclic graph of heterogeneous relationships The attention weight of each historical node to the current node is calculated above. The formula is as follows: in, Expression of speech Dialogue Attention coefficient; Indicates from discourse Pointing words The edge properties; Expression of speech The predecessor neighborhood; The context message vector is obtained by aggregating the attention weights and the utterance-level fusion feature vector, and its expression is as follows: in, Expression of speech The context message vector; Represents edge attributes The associated learnable weight matrix; Expression of speech The discourse-level fusion feature vector.

[0010] Preferably, the classification layer includes a feedforward neural network classifier, which comprises two layers of feedforward networks. The processing method of the feedforward neural network classifier includes: in, Expression of speech The probability distribution of sentiment categories; , This represents the learnable weight matrix for the first and second layers of the feedforward network; , These are the learnable bias vectors for the first and second layers of the feedforward network; Expression of speech The node representation includes context dependencies.

[0011] Preferably, the preprocessing and cross-modal alignment of multiple utterances to obtain text data, speech data, and video data of the multiple utterances includes: The text of each utterance is cleaned and segmented into words and sub-words to obtain text segments; the speech is subjected to endpoint detection, silence truncation and uniform sampling rate to obtain speech segments; the video is subjected to frame extraction, face detection and alignment to obtain video frame sequences. Alignment relationships are established based on timestamps to align text segments, audio segments, and video frame sequences of the same discourse in time, resulting in text data, audio data, and video data.

[0012] Preferably, the loss function of the pre-trained emotion recognition model based on progressive attention and acyclic graphs during the training process includes: in, Hyperparameters are used to balance various losses; Represents cross-entropy loss; This indicates the sample-weighted focus contrast loss; This represents a cross-modal consistency regularization term.

[0013] Secondly, the present invention provides a multimodal dialogue emotion recognition system based on progressive attention and acyclic graphs, characterized in that emotion recognition is performed through a pre-trained emotion recognition model based on progressive attention and acyclic graphs. The emotion recognition model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation directed acyclic graph layer, a relation-aware graph attention layer, and a classification layer. The multimodal dialogue emotion recognition system includes: The data acquisition module is used to acquire text, voice, and video modal data from the dialogue to be emotion-recognized, and to segment the text, voice, and video modal data by conversation to form multiple utterances; The data processing module is used to preprocess and perform cross-modal alignment on multiple utterances to obtain text data, speech data, and video data of the multiple utterances. The model processing module is used to perform the following operations: The text data, speech data, and video data are extracted using a single-modal feature extraction layer to obtain three single-modal features for each utterance. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector for each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; The attention layer of the relation-aware graph is based on discourse-level fusion feature vector pairs. Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance and the corresponding context message are then gated and updated to obtain the node representation of each utterance containing context dependencies. By processing the context-dependent node representations of each utterance through a classification layer, the probability distribution of the sentiment category of each utterance is obtained. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

[0014] Thirdly, the present invention provides a computer-readable storage medium storing a computer program for multimodal dialogue emotion recognition based on progressive attention and acyclic graphs, wherein the computer program causes a computer to execute the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs as described above.

[0015] Fourthly, the present invention provides an electronic device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing multimodal dialogue emotion recognition based on progressive attention and acyclic graphs as described above.

[0016] (III) Beneficial Effects This invention provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs. Compared with existing technologies, it has the following advantages: This invention, based on heterogeneous relational directed acyclic layers, explicitly models the diverse dependencies between discourses, thereby more accurately capturing the dynamic evolution of emotions in dialogue and distinguishing different types of contextual relationships. It solves the technical problem of existing multimodal dialogue emotion recognition methods based on graph neural networks lacking the ability to distinguish dialogue dynamics and different types of contextual relationships, and significantly improves the accuracy and robustness of emotion recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a block diagram of a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This application provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, which solves the technical problem that existing multimodal dialogue emotion recognition methods based on graph neural networks lack the ability to distinguish dialogue dynamics and different types of contextual relationships. It realizes a directed acyclic layer based on heterogeneous relationships, which explicitly models the diverse dependencies between utterances, thereby more accurately capturing the dynamic evolution of emotions in dialogue.

[0021] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: Existing multimodal dialogue sentiment recognition methods based on graph neural networks have significant technical limitations. The core problem lies in treating all contextual relationships as homogeneous and transmitting information through a fixed graph structure, lacking the ability to distinguish between dialogue dynamics and different types of contextual relationships. Because dialogue itself is dynamic, different contextual relationships (such as dependencies between adjacent statements, sentiment transitions, and topical connections) vary significantly. Fixed graph structures and homogeneous information processing methods cannot effectively distinguish these different types of contextual relationships and are ill-suited to adapting to dynamic changes during dialogue, thus affecting the accuracy and flexibility of sentiment recognition.

[0022] To address the aforementioned issues, this invention proposes a multimodal dialogue sentiment recognition method based on progressive attention and acyclic graphs. This method explicitly models diverse dependencies between utterances using heterogeneous relational directed acyclic layers, thereby more accurately capturing the dynamic evolution of sentiment in dialogue. Simultaneously, a progressive attention mechanism is employed to perform layer-by-layer alignment and adaptive integration of text, speech, and visual cues to improve the utilization of complementary information and mitigate the influence of noisy modalities. Furthermore, a training scheduling mechanism based on progressively increasing sample difficulty is introduced, allowing the model to optimize parameters from easy to difficult, thus alleviating the bias caused by long-tail imbalance and improving the robustness of judgment for a few sentiment categories and overall prediction performance.

[0023] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0024] This invention provides a multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs. The method performs emotion recognition using a pre-trained emotion recognition model based on progressive attention and acyclic graphs. The emotion recognition model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation-oriented acyclic graph layer, a relation-aware graph attention layer, and a classification layer. This multimodal dialogue emotion recognition method is as follows: Figure 1 As shown, it specifically includes: S1. Obtain text, speech, and video modal data from the dialogue to be emotion-recognized, and segment the text, speech, and video modal data according to the conversation to form multiple utterances; S2. Preprocess and perform cross-modal alignment on multiple utterances to obtain text data, speech data and video data of multiple utterances; S3. The text data, speech data and video data are extracted through the single-modal feature extraction layer to obtain three single-modal features for each utterance; S4. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector of each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; S5. Utilize discourse-level fusion feature vectors through the relation-aware graph attention layer. Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance is then gated and updated with the corresponding context message to obtain the node representation of each utterance containing context dependencies. S6. The node representations containing context dependencies of each utterance are processed through the classification layer to obtain the probability distribution of the sentiment category of each utterance. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

[0025] This invention, based on heterogeneous relational directed acyclic layers, explicitly models the diverse dependencies between discourses, thereby more accurately capturing the dynamic evolution of emotions in dialogue and distinguishing different types of contextual relationships. It solves the technical problem of existing multimodal dialogue emotion recognition methods based on graph neural networks lacking the ability to distinguish dialogue dynamics and different types of contextual relationships, and significantly improves the accuracy and robustness of emotion recognition.

[0026] In one embodiment, S1, text, speech, and video modal data from the dialogue to be emotion-recognized are acquired, and the text, speech, and video modal data are segmented according to the conversation to form multiple utterances. The specific implementation process is as follows: Acquire text, audio, and video data corresponding to each utterance in each dialogue. Segment the dialogue by conversation, using "utterance" as the smallest prediction unit to form a time-ordered utterance sequence, and number each utterance.

[0027] In one embodiment, S2, multiple utterances are preprocessed and cross-modal aligned to obtain text data, speech data, and video data of the multiple utterances. The specific implementation process is as follows: The process involves cleaning and segmenting the text of each utterance into words and sub-words; performing endpoint detection, silence truncation, and uniform sampling rate for the speech; and extracting frames, detecting and aligning faces in the video. To ensure consistency of multimodal information for the same utterance, alignment relationships are established based on timestamps, ensuring precise temporal alignment and one-to-one matching of text segments, speech segments, and video frame sequences of the same utterance. This yields text data, speech data, and video data. Furthermore, in the implementation process, missing modalities in the text, speech, and video data need to be masked to avoid introducing invalid features. It should be noted that the implementation is not limited to text, speech, and video modalities; other modalities, such as physiological signal modalities and environmental modalities, can also be added.

[0028] In one embodiment, S3, feature extraction is performed on the text data, speech data, and video data through a single-modal feature extraction layer to obtain three single-modal features for each utterance. The specific implementation process is as follows: In the single-modal feature extraction layer, three encoding branches are constructed for text, speech, and vision, respectively, outputting speech-level single-modal features. Specifically: the text branch obtains text features through a pre-trained language model; the speech branch extracts acoustic features (such as prosody, energy, spectral features, etc.) and obtains speech modal features through nonlinear mapping; the vision branch extracts expression-related features from the facial region and obtains visual features. Its expression is shown in Equation 1.1 below: in, When dividing a dialogue into multiple utterances, it is necessary to number the utterances. , , They represent the first The raw input of the discourse, including text, voice, and visual data; , , These represent the corresponding text, speech, and visual encoders, respectively. , , Representing the first The textual, auditory, and visual features of the discourse.

[0029] In one embodiment, S4, the three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fusion feature vector of each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. The specific implementation process is as follows: This invention proposes a hierarchical fusion layer based on progressive attention to address the problem that traditional single-step alignment fusion strategies struggle to capture fine-grained cross-modal interactions. Instead of mixing the three modalities all at once, this mechanism stacks multiple layers of progressive fusion units, mimicking the human cognitive process of deepening information interaction between modalities layer by layer. Specifically, each progressive fusion unit comprises two stages of attention interaction and feature update. Assuming the current stage is the […]. Layers, with inputs being discourse. visual features Text features and audio characteristics This mechanism, guided by visual features, performs cross-modal information fusion in two stages: The first stage employs visual-text cross-modal attention interaction, using visual features as queries and text features as keys and values. A multi-head attention mechanism is used to calculate the supplementary information from textual semantics to visual representation, thereby clarifying ambiguous visual expressions through textual clarity. The specific calculation process is shown in the following formula: (1.2) in, They represent the first The query vector, key vector, and value vector in the layered vision-text cross-modal multi-head attention mechanism. , , The projected weight matrix is ​​a learnable matrix. Presentation layer normalization operation, This represents the multi-head attention function. Equation (1.2) completes feature projection, and Equation (1.3) aggregates text information through multi-head attention weighting, and obtains the visual representation of text enhancement through residual connections and layer normalization. .

[0030] The second stage involves visual-audio cross-modal attention interaction. Building upon the first stage, it incorporates visually enhanced features of the text. As a new query, with audio features A secondary interaction is performed to refine emotional representations using acoustic cues such as prosody and pitch. The calculation formula is as follows: in, They represent the first The layered visual-audio cross-modal multi-head attention mechanism includes query, key, and value vectors. Through this operation, the model can utilize acoustic cues such as prosody and pitch in speech to perform secondary adaptive correction and emotional feature enhancement on the visual representation that has been initially corrected by the text.

[0031] To further enhance the nonlinear characterization capability, The input is transformed by a feedforward neural network (FFN), and the final visual output of the current progressive attention layer is obtained through residual connections. : Both text features and audio features employ a two-stage cross-modal attention interaction (text features use text-visual and text-audio cross-modal attention interactions; audio features use audio-text and audio-visual cross-modal attention interactions), and are updated synchronously with visual features to obtain the output of this layer. and After stacking L layers of the progressive fusion units described above, the intermodal information interaction is refined layer by layer. The outputs of the three modal features from the Lth layer are concatenated to obtain a discourse-level fusion feature vector for subsequent context modeling. The definition is shown in Formula 1.7: in, Indicates the first The utterance-level fusion feature vector is obtained after multimodal fusion, where ⊕ represents the vector concatenation operation. By stacking multiple layers of the above progressive fusion modules and combining residual connections and layer normalization, a utterance-level fusion feature vector that can fully capture fine-grained cross-modal interactions is finally obtained. This is used for subsequent dialogue context modeling.

[0032] This progressive attention structure, through a hierarchical query mechanism of "text first, then audio," effectively avoids noise interference when processing multi-source information simultaneously, making feature fusion more orderly. It should be noted that in specific implementations, when performing cross-attention fusion on the currently enhanced feature using two additional features, the order of the two features can be interchanged, such as "audio first, then text." Simultaneously, this structure explicitly models the progressive dependencies between modalities, making it particularly suitable for handling complex emotional scenarios involving single-modal semantic conflicts, such as "irony," significantly improving the robustness and accuracy of multimodal fusion. Of course, it is understood that if this embodiment of the invention uses four single-modal features, cross-modal information fusion can be performed in three stages. Similarly, when using five single-modal features, cross-modal information fusion can be performed in four stages, and so on.

[0033] The dialogue is represented as a directed acyclic graph (G=(V,E)), where the set of nodes (V) corresponds to the utterances in the dialogue, and the set of edges (E) points from historical utterances to the current utterance to satisfy the acyclic constraint. The core innovation of this invention lies in constructing heterogeneous edges, whose relationships are automatically labeled. For any two utterance nodes in the graph that have a temporal relationship... and ( exist Previously, a directed edge was established. Using a pre-defined rule engine, the system automatically assigns a relation type to the edge based on information such as semantic similarity between utterances and whether the speakers are the same. Relationship types include, but are not limited to: temporal continuation, semantic similarity, semantic transition, causal relationship, and same / different speakers.

[0034] Thus, the dialogue is modeled as a heterogeneous graph containing multiple types of edges, providing a structural foundation for subsequent refined contextual reasoning.

[0035] In one embodiment, S5, the relationship-aware graph attention layer uses the utterance-level fusion feature vector pairs of each utterance. Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance is then gated and updated with the corresponding context message of each utterance to obtain the node representation of each utterance containing context dependencies. The specific implementation process is as follows: In the constructed heterogeneous relation directed acyclic graph Above, contextual information aggregation is performed on each node through a relationship-aware graph attention layer. For a node... (corresponding to the first) (a phrase), from all its predecessor neighboring nodes. Gathering information. Computing discourse. Regarding the current discourse Attention coefficient At that time, not only nodes are considered and Their own characteristics also embed the types of relationships between their edges. This key information is input into the attention calculation module for relation-aware attention calculation. The definition is shown in Formula 1.8 below: (1.8) in, Expression of speech Dialogue Attention coefficient; Indicates from discourse Pointing words The edge properties; Expression of speech The predecessor neighborhood. Thus, a heterogeneous relation directed acyclic graph is obtained. The attention weight of each historical node to the current node is calculated. Then, context message aggregation is performed, defined as shown in Formula 1.9: in, Expression of speech The context message vector; Represents edge attributes The associated learnable weight matrix; Expression of speech The discourse-level fusion feature vector. Then the discourse-level fusion feature vector... With context message vector Perform a gating update to obtain a node representation that includes context dependencies. The definition is shown in Formula 1.10: The above operations enable the dynamic combination of the current discourse information and the aggregated contextual messages. The gating mechanism of GRU can control the degree of forgetting and retaining historical information, thereby depicting the dynamic changes of emotions in the dialogue process.

[0036] In one embodiment, S6, the node representations containing context dependencies of each utterance are processed through a classification layer to obtain the probability distribution of the sentiment category of each utterance. The specific implementation process is as follows: Represent the final node The input is fed into a feedforward neural network classifier, and the Softmax function is used to obtain the probability distribution of the sentiment category of the utterance. The definition is shown in Formula 1.11: in, Indicates the first The probability distribution of sentiment categories for each utterance; , This represents the learnable weight matrix for the first and second layers of the feedforward network; , represents the learnable bias vectors for the first and second layer feedforward networks.

[0037] Emotion recognition results can be applied to various scenarios such as human-computer interaction, customer service, mental health monitoring, and personalized recommendations.

[0038] It should be noted that the training process of the emotion recognition model is as follows: The first step involves collecting real-world dialogue scenarios, acquiring text, audio, and video data corresponding to each utterance in each dialogue, and associating each utterance with its corresponding dialogue number, utterance sequence number, speaker identifier, and sentiment tag for training purposes. The dialogues are then segmented into sessions, with each "utterance" serving as the smallest prediction unit to form a sample set.

[0039] The second step involves processing each dialogue segment in the sample set using the same method as in step S2, to obtain the text data, voice data, and video data of each utterance corresponding to each dialogue segment in the sample set. The third step involves using a pre-trained emotion recognition model, following the processing methods in steps S3-S6, to process the text, audio, and video data of each utterance in each dialogue segment of the sample set. This yields the probability distribution of emotion categories for each utterance in each dialogue segment of the sample set. Subsequently, using the emotion tags already labeled for each utterance in the sample as the ground truth tags, and combining this with the emotion category probability distribution output by the model, a loss function is used to calculate the loss value for a single utterance. Then, batch calculations are performed to obtain the average loss value for the entire sample set. Based on this average loss value, the backpropagation algorithm is used to update the parameters of the emotion recognition model for iterative training. After each iteration, the average loss value of the sample set is recalculated until the loss value drops below a preset loss threshold (or the number of iterations reaches a preset upper limit and the loss value tends to stabilize). At this point, iteration stops, and a successfully trained emotion recognition model with satisfactory performance is obtained.

[0040] It should be noted that, in this embodiment of the invention, the pre-trained emotion recognition model employs a composite training objective and curriculum learning strategy during the training phase, specifically including: weighted cross-entropy, sample-weighted focusing loss, and a cross-modal consistency regularization term. The superimposed cross-modal consistency regularization term ensures that different modalities maintain relevance within the shared space. Through curriculum-based training scheduling, the model is guided to master dialogue emotion recognition capabilities from easy to difficult. The specific loss function is defined as shown in Formula 1.12 below: in, Hyperparameters are used to balance various losses.

[0041] The specific design of each sub-item of loss is as follows: (1) Cross-entropy loss Provides the master supervisor signal to ensure the accuracy of the final sentiment classification, defined as shown in Formula 1.13 below: Where N is the total number of discourses in the sample set, and C is the number of sentiment categories. For the first The true label of this statement To predict probabilities.

[0042] (2) Sample weighted focusing contrast loss This is used to bring similar samples closer together and push dissimilar samples further apart in the feature space. It directly affects the representation of discourse nodes and effectively alleviates the class imbalance problem caused by the long-tail distribution of minority sentiment samples. It is defined as shown in Formula 1.14 below: in, The predicted probability for positive sample pairs. These are sample category weighting coefficients introduced for long-tailed distributions.

[0043] (3) Cross-modal consistency regularization term Based on the Soft-HGR maximum correlation loss, this method forces the text, video, and audio modal branches to maintain their correlation within the shared space before aggregation, thus solving the multimodal alignment problem. The definition is shown in Equation 1.15 below. in, Represents the set of combinations of the three modes. , and To fuse the specific modal features of the output of the Lth layer of the network, This means calculating the expected value of all samples within a given training batch. , These represent two different modes in the target combination pair set. Represents the trace of a matrix. Corresponding modes and modality The covariance matrix of the eigenvectors within a given training batch.

[0044] Furthermore, a curriculum-based training schedule is employed to guide the model in mastering dialogue sentiment recognition capabilities from easy to difficult. This curriculum-based training schedule involves introducing a phased training process: initially using shorter dialogue samples with less sentiment transfer or lower noise for initial learning, and then gradually adding more complex samples to improve the model's ability to distinguish complex contexts and minority classes. During system deployment, steps S2-S6 are executed on the input dialogue, outputting utterance-by-utter sentiment labels; for missing modality samples, robust inference is performed using the masking mechanism of steps S2-S6, thus meeting the data incompleteness requirements of real-world application scenarios.

[0045] This invention also provides a multimodal dialogue emotion recognition system based on progressive attention and acyclic graphs. Emotion recognition is performed using a pre-trained emotion recognition model based on progressive attention and acyclic graphs. The emotion recognition model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation directed acyclic graph layer, a relation-aware graph attention layer, and a classification layer. The multimodal dialogue emotion recognition system includes: The data acquisition module is used to acquire text, voice, and video modal data from the dialogue to be emotion-recognized, and to segment the text, voice, and video modal data by conversation to form multiple utterances; The data processing module is used to preprocess and perform cross-modal alignment on multiple utterances to obtain text data, speech data, and video data of the multiple utterances. The model processing module is used to perform the following operations: The text data, speech data, and video data are extracted using a single-modal feature extraction layer to obtain three single-modal features for each utterance. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector for each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; By leveraging the relation-aware graph attention layer and fusion feature vectors at the discourse level, Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance and the corresponding context message are then gated and updated to obtain the node representation of each utterance containing context dependencies. By processing the context-dependent node representations of each utterance through a classification layer, the probability distribution of the sentiment category of each utterance is obtained. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

[0046] It is understood that the multimodal dialogue emotion recognition system based on progressive attention and acyclic graphs provided in this embodiment of the invention corresponds to the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, and will not be repeated here.

[0047] This invention also provides a computer-readable storage medium storing a computer program for multimodal dialogue emotion recognition based on progressive attention and acyclic graphs, wherein the computer program causes a computer to execute the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs as described above.

[0048] This invention also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs as described above.

[0049] In summary, compared with existing technologies, it has the following beneficial effects: 1. Based on heterogeneous relational directed acyclic layers, this invention explicitly models the diverse dependencies between discourses, thereby more accurately capturing the dynamic evolution of emotions in dialogue and distinguishing different types of contextual relationships. This solves the technical problem of existing multimodal dialogue emotion recognition methods based on graph neural networks lacking the ability to distinguish dialogue dynamics and different types of contextual relationships, and significantly improves the accuracy and robustness of emotion recognition.

[0050] 2. The embodiments of the present invention employ a progressive attention mechanism to perform layer-by-layer alignment and adaptive integration of text, speech and visual cues, so as to improve the utilization rate of complementary information and weaken the influence of noise modalities. The synergistic effect with the node update mechanism of heterogeneous graphs further improves the accuracy and robustness of emotion recognition.

[0051] 3. This invention significantly enhances the model's decision-making transparency and reasoning ability by introducing an interpretable relation-aware graph attention mechanism. As a core advantage, the model's internal attention weight distribution clearly reveals the crucial role of "semantic relationships" in sentiment judgment during contextual information aggregation. This makes the model's decision-making process no longer a "black box," but rather clearly traceable and analyzable, providing a solid foundation of trust for its application in critical areas with high reliability requirements, such as mental health diagnosis and financial risk control.

[0052] 4. The embodiments of the present invention effectively improve the ability to recognize minority emotions under long-tail distribution by jointly optimizing the course learning training strategy and the composite loss function.

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs, characterized in that, Emotion recognition is performed using a pre-trained emotion recognition model based on progressive attention and acyclic graphs. This model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation-oriented acyclic graph layer, a relation-aware graph attention layer, and a classification layer. The multimodal dialogue emotion recognition method includes: Acquire text, speech, and video modal data from the dialogue to be emotion-recognized, and segment the text, speech, and video modal data by conversation to form multiple utterances; Multiple utterances are preprocessed and cross-modal aligned to obtain text data, speech data, and video data of the multiple utterances; The text data, speech data, and video data are extracted using a single-modal feature extraction layer to obtain three single-modal features for each utterance. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector for each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; By leveraging the relation-aware graph attention layer and fusion feature vectors at the discourse level, Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance and the corresponding context message are then gated and updated to obtain the node representation of each utterance containing context dependencies. By processing the context-dependent node representations of each utterance through a classification layer, the probability distribution of the sentiment category of each utterance is obtained. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

2. The multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The relationship types include temporal continuation, semantic similarity, semantic transition, causal relationship, and same or different speakers.

3. The multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The process involves fusing the three single-modal features through a hierarchical fusion layer to obtain the discourse-level fusion feature vector for each utterance, including: The hierarchical fusion layer based on progressive attention includes L stacked progressive fusion units. Each progressive fusion unit contains two stages of attention interaction and feature update process. Taking visual features as an example, it includes: The first stage employs visual-text cross-modal attention interaction, using visual features as queries and text features as keys and values. A multi-head attention mechanism is used to compute the information supplementation of visual expression by textual semantics, thereby obtaining the discourse. Text-enhanced visual representation ; The second stage utilizes visual-audio cross-modal attention interaction, building upon the first stage by enhancing the visual features of the text. As a new query, with audio features To obtain a second interaction, we need to get the dialogue. The dual-enhanced visual representation ; Will The input is transformed into a feedforward neural network, and the visual output of the current progressive attention layer is obtained through residual connections. : Text features, like audio features, employ a two-stage cross-modal attention interaction, updated synchronously with visual features, to obtain the first... The words output by the layer Text output and audio output ; Output the audio from layer L. Text output and visual output By piecing them together, we can obtain the words. discourse-level fusion feature vector .

4. The multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The relationship-aware graph attention layer uses utterance-level fusion feature vectors for each utterance. Each node in the process performs context information aggregation to obtain the context message vector for each utterance, including: Calculate the directed acyclic graph of heterogeneous relationships The attention weight of each historical node to the current node is calculated above. The formula is as follows: in, Expression of speech Dialogue Attention coefficient; Indicates from discourse Pointing words The edge properties; Expression of speech The predecessor neighborhood; The context message vector is obtained by aggregating the attention weights and the utterance-level fusion feature vector, and its expression is as follows: in, Expression of speech The context message vector; Represents edge attributes The associated learnable weight matrix; Expression of speech The discourse-level fusion feature vector.

5. The multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The classification layer includes a feedforward neural network classifier, which comprises two layers of feedforward networks. The processing method of the feedforward neural network classifier includes: in, Expression of speech The probability distribution of sentiment categories; , This represents the learnable weight matrix for the first and second layers of the feedforward network; , These are the learnable bias vectors for the first and second layers of the feedforward network; Expression of speech The node representation includes context dependencies.

6. The multimodal dialogue emotion recognition method as described in any one of claims 1 to 5, characterized in that, The preprocessing and cross-modal alignment of multiple utterances yields text data, speech data, and video data for the multiple utterances, including: The text of each utterance is cleaned and segmented into words and sub-words to obtain text segments; the speech is subjected to endpoint detection, silence truncation and uniform sampling rate to obtain speech segments; the video is subjected to frame extraction, face detection and alignment to obtain video frame sequences. Alignment relationships are established based on timestamps to align text segments, audio segments, and video frame sequences of the same discourse in time, resulting in text data, audio data, and video data.

7. The multimodal dialogue emotion recognition method as described in any one of claims 1 to 5, characterized in that, The loss function of the pre-trained emotion recognition model based on progressive attention and acyclic graphs during training includes: in, Hyperparameters are used to balance various losses; Represents cross-entropy loss; This indicates the sample-weighted focus contrast loss; This represents a cross-modal consistency regularization term.

8. A multimodal dialogue emotion recognition system based on progressive attention and acyclic graphs, characterized in that, Emotion recognition is performed using a pre-trained emotion recognition model based on progressive attention and acyclic graphs. This emotion recognition model includes a unimodal feature extraction layer, a hierarchical fusion layer based on progressive attention, a heterogeneous relation-oriented acyclic graph layer, a relation-aware graph attention layer, and a classification layer. The multimodal dialogue emotion recognition system includes: The data acquisition module is used to acquire text, voice, and video modal data from the dialogue to be emotion-recognized, and to segment the text, voice, and video modal data by conversation to form multiple utterances; The data processing module is used to preprocess and perform cross-modal alignment on multiple utterances to obtain text data, speech data, and video data of the multiple utterances. The model processing module is used to perform the following operations: The text data, speech data, and video data are extracted using a single-modal feature extraction layer to obtain three single-modal features for each utterance. The three single-modal features are fused through a hierarchical fusion layer to obtain the utterance-level fused feature vector for each utterance; the dialogue to be sentiment identified is represented as a heterogeneous relation directed acyclic graph through a heterogeneous relation directed acyclic layer. ; By leveraging the relation-aware graph attention layer and fusion feature vectors at the discourse level, Each node in the process performs context information aggregation to obtain the context message vector of each utterance; the utterance-level fusion feature vector of each utterance and the corresponding context message are then gated and updated to obtain the node representation of each utterance containing context dependencies. By processing the context-dependent node representations of each utterance through a classification layer, the probability distribution of the sentiment category of each utterance is obtained. Among them, In the graph, V represents the set of nodes, corresponding to multiple utterances in the dialogue, and E represents the set of edges, representing the points from historical utterances to the current utterance to satisfy the acyclic constraint. For any two utterance nodes in the graph that have a temporal relationship... and Establish a directed edge , exist Previously; directed edges were defined using a rules engine. Label a relation type .

9. A computer-readable storage medium, characterized in that, It stores a computer program for multimodal dialogue emotion recognition based on progressive attention and acyclic graphs, wherein the computer program causes a computer to execute the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the multimodal dialogue emotion recognition method based on progressive attention and acyclic graphs as described in any one of claims 1 to 7.

Images