Multimodal Emotion Recognition Method and System for Human Dialogue

Abstract

This invention provides a multimodal emotion recognition method and system for dialogue, relating to the field of emotion recognition technology. The method includes: acquiring multimodal emotion input data and extracting features to obtain multimodal feature representations; adding the multimodal feature representations to speaker identity embeddings to obtain modal feature representations; constructing a graph structure for cross-modal interaction based on the modal feature representations; processing the graph structure through a pre-constructed Fourier graph network to obtain low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts; obtaining a low-frequency emotion consensus representation based on the global mean vector; obtaining a high-frequency discriminant representation based on the high-frequency modal mean benchmark; fusing the high-frequency discriminant representation and the low-frequency emotion consensus representation to obtain a frequency domain comprehensive representation; and obtaining emotion prediction results based on the frequency domain comprehensive representation. This invention, through the synergistic consideration of high-frequency and low-frequency features, makes the emotion recognition capability for dialogue more accurate.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a multimodal emotion recognition method and system for human dialogue. Background Technology

[0002] With the rapid development of technologies such as human-computer intelligent interaction, intelligent customer service, mental health assessment, and social media sentiment analysis, multimodal dialogue emotion recognition has become a core research direction in the fields of natural language processing, emotion computing, and multimodal intelligent analysis.

[0003] Existing graph structure modeling methods treat individual dialogue utterances as graph nodes, set dialogue contextual relationships or speaker relationships as graph edges, and utilize existing spatial domain graph neural networks and their variants to complete message passing and feature aggregation between nodes, thereby modeling the internal structured dependencies of the dialogue.

[0004] However, existing graph neural networks inherently possess low-pass filtering characteristics. During the process of neighborhood feature aggregation, they preferentially retain low-frequency global consistency information within the neighborhood, naturally suppressing high-frequency information that can reflect modal differences, local emotional mutations, and semantic conflicts. In complex multimodal dialogues, when the emotional expression of a single modality is ambiguous, such high-frequency difference information is the key to achieving cross-modal information complementarity and judging true emotions. The severe loss of this high-frequency emotional information greatly affects the accuracy of emotion recognition. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a multimodal emotion recognition method and system for dialogue, which solves the problem of insufficient emotion recognition accuracy caused by the lack of high-frequency key information in existing methods.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal emotion recognition method for dialogue, comprising: Acquire multimodal emotional input data of the dialogue of the person to be identified, and extract features from the multimodal emotional input data to obtain multimodal feature representation; The multimodal feature representations are added to the speaker identity embedding to obtain modal feature representations with context and person perception. Each modal feature representation of each utterance in the corresponding dialogue sequence is used as a graph node. Contextual and cross-modal association edges between nodes are constructed based on a preset sliding window to build a graph structure for cross-modal interaction. By performing discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure through a pre-constructed Fourier graph neural network, low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts are obtained. Calculate the global mean vector of the low-frequency feature representation, and obtain a low-frequency sentiment consensus representation based on the global mean vector; calculate the high-frequency modal mean benchmark value of the high-frequency feature representation, and obtain a high-frequency discriminative representation based on the high-frequency modal mean benchmark value; The high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation. Based on the frequency domain comprehensive representation, sentiment category classification and prediction are performed to obtain sentiment prediction results.

[0007] Preferably, the step of extracting features from the multimodal emotion input data to obtain a multimodal feature representation includes: The discourse-level semantic feature representation of the sentiment input data of the text modality is extracted through a pre-trained language model; the pre-trained language model includes RoBERTa. Speech feature representations of speech modal emotion input data are extracted using pre-trained openSMILE; Video feature representations of video modality sentiment input data are extracted using a pre-trained 3D-CNN.

[0008] Preferably, obtaining the low-frequency sentiment consensus representation based on the global mean vector includes: The matching degree between each modal feature representation and the global mean vector is calculated using an attention mechanism; The matching degree is normalized using Softmax to obtain the weights of each modality feature representation; The low-frequency feature representations are weighted and aggregated based on the weights to obtain a cross-modal low-frequency sentiment consensus representation.

[0009] Preferably, the step of performing sentiment category classification prediction based on the frequency domain comprehensive representation to obtain sentiment prediction results includes: The frequency domain synthesis representation is mapped back to the time domain space by inverse discrete Fourier transform; The comprehensive representation mapped back to the temporal space is then used to perform the final sentiment category classification prediction through a multilayer perceptron and a Softmax layer to obtain the sentiment prediction result.

[0010] Preferably, the step of calculating the high-frequency modal mean reference value of the high-frequency feature representation and obtaining the high-frequency discriminant representation based on the high-frequency modal mean reference value includes: Calculate the benchmark value of the high-frequency modal mean based on the high-frequency feature representation; Calculate the residual of each mode relative to the mean reference value of the high-frequency modes, and concatenate the residuals to obtain the mode deviation vector; By projecting the modal deviation vector back to the original dimension through linear mapping, a high-frequency discriminant representation for explicitly capturing modal dynamic conflicts is obtained.

[0011] Preferably, the step of injecting the high-frequency discriminative representation into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation includes: Generate a gated modulation vector based on the high-frequency discriminant representation; Based on the gated modulation vector, the high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation.

[0012] Preferably, the step of performing discrete Fourier transform and frequency domain filtering on the graph signal in the graph structure using a pre-constructed Fourier graph neural network to obtain a low-frequency feature representation reflecting global consistency and a high-frequency feature representation reflecting local mutations and conflicts includes: High-pass and low-pass filters are constructed based on the graph structure. The graph signal is separated based on the high-pass and low-pass filters to obtain high-frequency and low-frequency operators; a discrete Fourier transform is then performed on the graph signal. Based on the high-frequency and low-frequency operators, multi-layer depth propagation and nonlinear activation are performed on the discrete Fourier transform graph signal in the frequency domain space to obtain low-frequency feature representation and high-frequency feature representation.

[0013] Secondly, the present invention also provides a multimodal emotion recognition system for human dialogue, comprising: The acquisition module acquires multimodal emotional input data of the dialogue of the person to be identified, and performs feature extraction on the multimodal emotional input data to obtain multimodal feature representation; The identity embedding module adds the multimodal feature representation to the speaker identity embedding respectively to obtain a modal feature representation with context and person perception; The graph structure construction module uses the modal feature representations of each utterance in the corresponding dialogue sequence as graph nodes, and constructs contextual and cross-modal association edges between nodes based on a preset sliding window to build a graph structure for cross-modal interaction. The processing module performs discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure through a pre-constructed Fourier graph neural network to obtain low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts. The calculation module calculates the global mean vector of the low-frequency feature representation and obtains a low-frequency sentiment consensus representation based on the global mean vector; it also calculates the high-frequency modal mean benchmark value of the high-frequency feature representation and obtains a high-frequency discriminative representation based on the high-frequency modal mean benchmark value. The fusion module injects the high-frequency discriminative representation into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation; The prediction module performs sentiment category classification prediction based on the frequency domain comprehensive representation to obtain sentiment prediction results.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program for multimodal emotion recognition for dialogue-oriented tasks, wherein the computer program causes a computer to execute the multimodal emotion recognition method for dialogue-oriented tasks as described above.

[0015] Fourthly, the present invention also provides an electronic device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing multimodal emotion recognition for person-oriented dialogue as described above.

[0016] (III) Beneficial Effects This invention provides a multimodal emotion recognition method and system for dialogue-based communication. Compared with existing technologies, it has the following advantages: 1. This invention uses a Fourier graph network to decompose graph signals into high and low frequencies, obtaining low-frequency feature representations that reflect the global emotional consistency of the dialogue, and high-frequency feature representations that reflect local emotional abrupt changes, intermodal differences and conflicts. It breaks through the limitations of existing graph neural networks that only retain low-frequency information and suppress high-frequency information, fully retaining high-frequency detail information that is crucial for emotion discrimination. When the emotion expression of a single modality is ambiguous, it relies on high-frequency information to achieve cross-modal complementarity, which greatly improves the accuracy of recognizing real emotions in complex dialogue scenarios.

[0017] 2. This invention introduces a pre-constructed Fourier graph network to replace the traditional spatial domain graph neural network. By transforming spatial convolution into frequency domain multiplication through Fourier transform, long-distance dialogue context dependency modeling can be achieved without deliberately stacking network layers. This effectively avoids the node feature assimilation and over-smoothing problems caused by multi-layer networks, clearly preserves the classification boundaries of different emotion samples, and significantly improves the stability of emotion modeling in long-term dialogue scenarios.

[0018] 3. This invention first extracts features from multimodal emotion data and integrates speaker identity embedding, endowing features with context awareness and character identity awareness capabilities; then, using a single dialogue as a graph node and relying on a sliding window to construct node-related edges to form a cross-modal interaction graph structure, it breaks through the constraints of the strict linear sequence of traditional temporal models, and can explicitly characterize the multi-party interaction topology, speaker role association, and multimodal dependency relationships in multi-person dialogues, fundamentally solving the problems that temporal modeling methods cannot characterize complex interaction structures and have weak cross-modal interaction capabilities. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the multimodal emotion recognition method for dialogue-oriented scenarios in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This application provides a multimodal emotion recognition method and system for dialogue, which solves the problem of insufficient accuracy in emotion recognition caused by the lack of high-frequency key information in existing methods, and realizes accurate recognition of real emotions in complex dialogue scenarios.

[0023] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: Character dialogues are mostly multi-role, multi-turn interactions, integrating multimodal information such as text, voice, and vision. Emotional expression combines contextual and temporal relationships, speaker identity differences, intermodal relationships, and conflict characteristics. Accurately identifying implicit emotions in character dialogues has significant engineering application value and academic research significance for improving intelligent interaction experience, emotional risk warning, and understanding dialogue intent.

[0024] Currently, there are two main modeling approaches for multimodal emotion recognition in dialogue: one is the temporal sequence modeling method, which treats the entire dialogue as a linear time series data and uses recurrent neural networks, long short-term memory networks and their variants, or Transformer models based on self-attention mechanisms to encode and model the temporal sequence of the dialogue, relying on temporal dependencies to capture the emotional evolution patterns of the dialogue context. The other is the graph structure modeling method, which breaks through the constraints of linear sequences, treats individual dialogue utterances as graph nodes, and sets the dialogue context relationships or speaker relationships as graph edges. It uses traditional spatial domain graph neural networks and their variants to complete message passing and feature aggregation between nodes, thereby modeling the structured dependencies within the dialogue.

[0025] Two existing mainstream modeling methods still suffer from unavoidable technical shortcomings: First, temporal modeling methods always limit dialogues to a strictly linear sequence, failing to explicitly characterize the complex structures inherent in multi-party dialogue scenarios, such as the topology of multi-party interactions, speaker role relationships, and question-answer pair dependencies. Furthermore, they are weak in modeling cross-modal interaction relationships between text, speech, and vision. Second, traditional spatial domain graph neural network modeling methods face a dual bottleneck. On the one hand, capturing long-distance dialogue contextual dependencies requires increasing the model depth by stacking network layers, which easily leads to over-smoothing of graph node features, resulting in blurred classification boundaries for different emotion samples and decreased recognition accuracy. On the other hand, traditional graph neural networks inherently possess low-pass filtering characteristics, prioritizing the retention of low-frequency global consistency information within the neighborhood during neighborhood feature aggregation, naturally suppressing high-frequency information that reflects modal differences, local emotional mutations, and semantic conflicts. However, in complex multimodal dialogues, when the expression of emotion in a single modality is ambiguous, such high-frequency difference information is crucial for achieving cross-modal information complementarity and discerning true emotions. Existing technologies suffer from severe loss of high-frequency emotional information.

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] like Figure 1 As shown, this embodiment of the invention provides a multimodal emotion recognition method for dialogue, including: Step S110: Obtain multimodal emotional input data of the dialogue of the person to be identified, and perform feature extraction on the multimodal emotional input data to obtain multimodal feature representation; Step S120: Add the multimodal feature representation to the speaker identity embedding respectively to obtain a modal feature representation with context and person perception; Step S130: The modal feature representations of each utterance in the corresponding dialogue sequence are used as graph nodes. Contextual association edges and cross-modal association edges between nodes are constructed according to a preset sliding window to build a graph structure for cross-modal interaction. Step S140: The graph signal in the graph structure is subjected to discrete Fourier transform and frequency domain filtering through a pre-constructed Fourier graph neural network to obtain a low-frequency feature representation reflecting global consistency and a high-frequency feature representation reflecting local mutations and conflicts. Step S150: Calculate the global mean vector of the low-frequency feature representation, and obtain a low-frequency sentiment consensus representation based on the global mean vector; calculate the high-frequency modal mean benchmark value of the high-frequency feature representation, and obtain a high-frequency discriminant representation based on the high-frequency modal mean benchmark value. Step S160: The high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation; Step S170: Based on the frequency domain comprehensive representation, perform sentiment category classification prediction to obtain sentiment prediction results.

[0028] The multimodal emotion recognition method in this embodiment completes the high- and low-frequency decomposition of graph signals through Fourier graph networks, respectively obtaining low-frequency feature representations reflecting the global emotional consistency of the dialogue, and high-frequency feature representations reflecting local emotional abrupt changes, intermodal differences and conflicts. It breaks through the limitation of existing graph neural networks that only retain low-frequency information and suppress high-frequency information, fully retains high-frequency detail information that is crucial to emotion discrimination, and relies on high-frequency information to achieve cross-modal complementarity when the emotion expression of a single modality is ambiguous, which greatly improves the accuracy of recognizing real emotions in complex dialogue scenarios.

[0029] In one embodiment, step S110 involves acquiring multimodal emotional input data of the dialogue of the person to be identified, and performing feature extraction on the multimodal emotional input data to obtain a multimodal feature representation.

[0030] Specifically, the multimodal emotion input data comes from dialogue scenarios and includes text-based emotion input data, voice-based emotion input data, and video-based emotion input data.

[0031] After obtaining the input data, feature extraction is performed on the text, speech, and visual modalities of the input dialogue data. Specifically, discourse-level semantic feature representations of the text modal sentiment input data are extracted using a pre-trained language model, including RoBERTa. Speech feature representations of the speech modal sentiment input data are extracted using a pre-trained openSMILE. Video feature representations of the video modal sentiment input data are extracted using a pre-trained 3D-CNN.

[0032] After obtaining multimodal feature representations through feature extraction, feature encoding is further performed. For discourse-level semantic features, a bidirectional gated recurrent unit (Bi-GRU) is introduced to capture temporal context. For acoustic and visual modal speech feature representations and video feature representations, a linear mapping layer unifies the dimensions.

[0033] In one embodiment, step S120 involves adding the multimodal feature representation to the speaker identity embedding to obtain a modal feature representation with context and person perception.

[0034] Emotional expression in dialogue scenarios varies from person to person. This embodiment further introduces the third... i The speaker identity embedding vector is used to add the initial unimodal feature representation obtained above to the speaker identity embedding, resulting in a modal feature representation with context and speaker awareness. The expression for this representation is: In the formula, This represents modal feature representation that possesses contextual and character-based perception. Indicates the dialog index; Indicates modal index, These represent text modality, speech modality, and video modality, respectively. This represents multimodal sentiment input data; It indicates the speaker's identity information.

[0035] In one embodiment, step S130 involves using the modal feature representations of each utterance in the corresponding dialogue sequence as graph nodes, and constructing contextual association edges and cross-modal association edges between nodes based on a preset sliding window to build a graph structure for cross-modal interaction. This embodiment constructs a multimodal interaction graph. Specifically, after embedding and summing the speaker identities to obtain modal feature representations with context and speaker awareness, the single-modal feature representations of each utterance in the dialogue are used as nodes of the graph. Edges are constructed according to the set sliding window size (including the history window and the future window). The edges include: intramodal edges - connecting context utterance nodes of the same modality within the sliding window range, and intermodal edges - connecting nodes of different modalities at the same time. A graph structure that integrates contextual relationships and cross-modal interactions is constructed through the topological relationships of nodes and edges.

[0036] In one embodiment, step S140 involves performing Discrete Fourier Transform and frequency domain filtering on the graph signal in the graph structure using a pre-constructed Fourier graph neural network to obtain a low-frequency feature representation reflecting global consistency and a high-frequency feature representation reflecting local mutations and conflicts. The specific implementation of this embodiment includes the following steps: Step S141: Construct a high-pass filter and a low-pass filter based on the graph structure; it is understood that the high-pass filter and the low-pass filter are constructed based on the graph topology (i.e., the adjacency matrix and the degree matrix).

[0037] Step S142: Separate the graph signal based on the high-pass filter and low-pass filter to obtain high-frequency domain operators and low-frequency domain operators; perform discrete Fourier transform on the graph signal; Step S143: Based on the high-frequency and low-frequency operators, multi-layer depth propagation and nonlinear activation are performed on the discrete Fourier transform graph signal in the frequency domain space to obtain low-frequency feature representation and high-frequency feature representation.

[0038] Understandably, a graph signal is a set of feature data defined on all nodes of a graph topology. It is numerical feature information attached to graph nodes and can be processed by graph network operations. It consists of multimodal sentiment feature vectors corresponding to each modality feature representation and is the object of actual operations, transformations, and filtering in Fourier graph networks. High- and low-frequency domain operators process the distribution of feature vectors of each sentence in different modalities within the graph topology space, thereby separating complex mixed sentiment signals into low-frequency and high-frequency feature representations.

[0039] In this embodiment, to overcome the shortcomings of high computational complexity and inability to distinguish frequency semantics in traditional spatial graph convolution, a Fourier graph neural network is constructed, which uses Discrete Fourier Transform (DFT) to convolve the graph in the spatial domain. The efficient multiplication transformed into the frequency domain is expressed as follows: In the formula, Represents the Fourier operator in the frequency domain; The feature matrix (i.e., graph signal) represents a given interactive graph structure. Indicates the size of the sliding window; It is the adjacency matrix of the graph structure; In the formula, Indicates a low-pass filter; Indicates a high-pass filter; Indicates low frequency, Indicates high frequency; I is the identity matrix; It is a degree matrix; It is the adjacency matrix of the interaction graph structure; In the formula, Represents the Discrete Fourier Transform; The constructed Green kernel is represented by a learnable parameter matrix. Constructing Green's Kernel This kernel is used to characterize the coupling relationship between the graph structure and the feature transformation, and it satisfies the translation invariance condition. And defined as ; This indicates element-wise multiplication; This represents the position indices of the i-th and j-th speech nodes in the spatial interaction graph topology.

[0040] In the formula, Indicates the activation function; S For the first i Fourier plot operator for layers; m The number of layers in the Fourier transform neural network; For the set of model parameters; This is the bias parameter. Indicates by parameters Parameterized Fourier graph neural network feature mapping function (taking the graph in the spatial domain as input) , After mapping, we obtain low-frequency feature representations and high-frequency feature representations. and These correspond to the low-frequency and high-frequency processing branches, respectively.

[0041] In one embodiment, step S150 involves calculating the global mean vector of the low-frequency feature representation and obtaining a low-frequency sentiment consensus representation based on the global mean vector; calculating the high-frequency modal mean benchmark value of the high-frequency feature representation and obtaining a high-frequency discriminative representation based on the high-frequency modal mean benchmark value.

[0042] In this embodiment, a differentiated fusion strategy is designed for the different semantic functions undertaken by different frequency features, namely consensus-oriented low-frequency modal fusion and difference-aware high-frequency modal fusion.

[0043] Next, we will introduce consensus-oriented low-frequency modality fusion. In this embodiment, obtaining the low-frequency sentiment consensus representation based on the global mean vector specifically includes the following steps: Step S251: Calculate the matching degree between each modal feature representation and the global mean vector using an attention mechanism; Step S252: The matching degree is normalized using Softmax to obtain the weights of each modality feature representation; Step S253: Based on the weights, perform weighted aggregation on the low-frequency feature representations to obtain a cross-modal low-frequency sentiment consensus representation.

[0044] Specifically, the global mean vector of the multimodal low-frequency features is first calculated, as shown in the following expression: In the formula, This represents the global mean vector of multimodal low-frequency features. The matching degree between each modal feature and this global consensus vector is calculated using an attention mechanism. The weights are then obtained by Softmax normalization. .

[0045] Obtain weights Subsequently, a weighted aggregation is performed to generate a cross-modal sentiment consensus representation. Its expression is: In the formula, This indicates a low-frequency emotional consensus. This represents the low-frequency characteristics.

[0046] Next, we introduce the difference-aware high-frequency modality fusion. In this embodiment, we calculate the high-frequency modality mean benchmark value of the high-frequency feature representation, and obtain the high-frequency discriminative representation based on the high-frequency modality mean benchmark value. The specific implementation steps include: Step S351: Calculate the high-frequency modal mean reference value based on the high-frequency feature representation; Step S352: Calculate the residual of each modal high-frequency feature representation compared with the high-frequency modal mean reference value, and concatenate the residuals to obtain the modal deviation vector; Step S353: Project the modal deviation vector back to the original dimension through linear mapping to obtain a high-frequency discriminant representation for explicitly capturing modal dynamic conflicts.

[0047] Specifically, high-frequency features represent emotional shifts and conflicts. First, the baseline mean of the high-frequency modalities is calculated, and its expression is: In the formula, This represents the benchmark for the mean value of high-frequency modes; This represents the high-frequency features.

[0048] The residuals of each mode relative to the baseline are then calculated and concatenated to form a mode deviation vector. Its expression is: Finally, by projecting it back to the original dimension through a linear mapping, we obtain the high-frequency discriminant representation for explicitly capturing modal dynamic conflicts. Its expression is: In the formula, This represents the weight matrix that projects the modal difference vector onto the target dimension; This is represented as the corresponding bias vector.

[0049] To coordinate consensus (low frequency) with local difference information (high frequency), high-frequency features are used to generate gated modulation vectors. Its expression is: In the formula, Represents the gate weight matrix; This represents the gating bias vector.

[0050] In one embodiment, step S160 involves injecting the high-frequency discriminative representation into the low-frequency sentiment consensus representation using a residual modulation mechanism to obtain a frequency domain comprehensive representation. The specific implementation of this embodiment includes the following steps: Step S161: Generate a gated modulation vector based on the high-frequency discriminant representation; Step S162: Based on the gated modulation vector, the high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation.

[0051] Specifically, a residual modulation mechanism is used to adaptively inject high-frequency discrimination information into the low-frequency consensus, generating the final frequency domain synthesis representation. Its expression is: in, This is element-wise multiplication.

[0052] In one embodiment, step S170 involves performing sentiment category classification prediction based on the frequency domain comprehensive representation to obtain a sentiment prediction result. The specific implementation of this embodiment includes the following steps: Step S171: Map the frequency domain synthesis representation back to the time domain space using the inverse discrete Fourier transform; Step S172: The comprehensive representation mapped back to the temporal space is used to perform the final sentiment category classification prediction through a multilayer perceptron and a Softmax layer to obtain the sentiment prediction result.

[0053] Specifically, the features are mapped back to the time domain using the inverse discrete Fourier transform (IDFT), and its expression is as follows: In the formula, This represents the comprehensive feature representation after mapping.

[0054] The features mapped back to the temporal space are fed into a multilayer perceptron (MLP) and a softmax layer for final sentiment category classification prediction. The expression is as follows: In the formula, This indicates the final sentiment category prediction result.

[0055] It needs to be clarified that the sentiment category prediction results This represents the probability distribution of the final sentiment category prediction for the target utterance node, obtained from a sliding window based on the current context.

[0056] This invention also provides a multimodal emotion recognition system for dialogue, comprising: The acquisition module acquires multimodal emotional input data of the dialogue of the person to be identified, and performs feature extraction on the multimodal emotional input data to obtain multimodal feature representation; The identity embedding module adds the multimodal feature representation to the speaker identity embedding respectively to obtain a modal feature representation with context and person perception; The graph structure construction module uses the modal feature representations of each utterance in the corresponding dialogue sequence as graph nodes, and constructs contextual and cross-modal association edges between nodes based on a preset sliding window to build a graph structure for cross-modal interaction. The processing module performs discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure through a pre-constructed Fourier graph neural network to obtain low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts. The calculation module calculates the global mean vector of the low-frequency feature representation and obtains a low-frequency sentiment consensus representation based on the global mean vector; it also calculates the high-frequency modal mean benchmark value of the high-frequency feature representation and obtains a high-frequency discriminative representation based on the high-frequency modal mean benchmark value. The fusion module injects the high-frequency discriminative representation into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation; The prediction module performs sentiment category classification prediction based on the frequency domain comprehensive representation to obtain sentiment prediction results.

[0057] It is understood that the multimodal emotion recognition system for human-oriented dialogue provided in this embodiment of the invention corresponds to the multimodal emotion recognition method for human-oriented dialogue described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the multimodal emotion recognition method for human-oriented dialogue, and will not be repeated here.

[0058] This invention also provides a computer-readable storage medium storing a computer program for a multimodal emotion recognition method for dialogue-oriented tasks, wherein the computer program causes a computer to execute the multimodal emotion recognition method for dialogue-oriented tasks as described above.

[0059] This application also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing multimodal emotion recognition for person-oriented dialogue as described above.

[0060] In summary, compared with existing technologies, it has the following beneficial effects: 1. In this embodiment of the invention, a pre-constructed Fourier graph network is introduced to replace the traditional spatial domain graph neural network. The spatial convolution is transformed into frequency domain multiplication operation through Fourier transform. Long-distance dialogue context dependency modeling can be achieved without deliberately stacking network layers. This effectively avoids the node feature assimilation and over-smoothing problems caused by multi-layer networks, clearly preserves the classification boundaries of different emotion samples, and significantly improves the stability of emotion modeling in long-term dialogue scenarios.

[0061] 2. This invention uses a Fourier graph network to decompose graph signals into high and low frequencies, obtaining low-frequency feature representations that reflect the global emotional consistency of the dialogue, and high-frequency feature representations that reflect local emotional abrupt changes, intermodal differences and conflicts. It breaks through the limitations of existing graph neural networks that only retain low-frequency information and suppress high-frequency information, fully retaining high-frequency detail information that is crucial for emotion discrimination. When the emotion expression of a single modality is ambiguous, it relies on high-frequency information to achieve cross-modal complementarity, which greatly improves the accuracy of recognizing real emotions in complex dialogue scenarios.

[0062] 3. In this embodiment of the invention, features are first extracted from multimodal sentiment data and speaker identity is embedded to endow the features with context awareness and person identity awareness. Then, a cross-modal interaction graph structure is formed by using a single dialogue as a graph node and constructing node association edges based on a sliding window. This breaks through the constraints of the strict linear sequence of traditional time-series models and can explicitly depict the multi-party interaction topology, speaker role association, and multimodal dependency relationship of multi-person dialogues. It fundamentally solves the problem that time-series modeling methods cannot depict complex interaction structures and have weak cross-modal interaction capabilities.

[0063] 4. In this embodiment of the invention, a low-frequency sentiment consensus representation is constructed by calculating the global mean vector of low-frequency features, and a high-frequency discriminative representation is constructed by calculating the modal mean benchmark value of high-frequency features. The global sentiment consensus and local modal difference information are standardized and refined, redundant noise is filtered out, and the semantic discriminability and representation quality of multimodal sentiment features are enhanced.

[0064] 5. In this embodiment of the invention, a residual modulation mechanism is used to adaptively inject high-frequency discriminative representation into low-frequency emotional consensus representation, thereby achieving differentiated fusion of high and low frequency features. This not only preserves the overall global emotional consensus of the dialogue, but also incorporates detailed information on local emotional mutations and modal conflicts, forming a more robust frequency domain comprehensive feature representation.

[0065] 6. The embodiments of the present invention form a complete closed loop from multimodal data feature extraction, speaker identity fusion, graph structure construction, frequency domain high and low frequency decoupling, feature fusion to emotion classification prediction. It is suitable for real dialogue scenarios with multiple characters, multiple rounds, and multiple modalities, and the model has stronger generalization ability and engineering implementation.

[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal emotion recognition method for character dialogue, characterized in that, include: Acquire multimodal emotional input data of the dialogue of the person to be identified, and extract features from the multimodal emotional input data to obtain multimodal feature representation; The multimodal feature representations are added to the speaker identity embedding to obtain modal feature representations with context and person perception. Each modal feature representation of each utterance in the corresponding dialogue sequence is used as a graph node. Contextual and cross-modal association edges between nodes are constructed based on a preset sliding window to build a graph structure for cross-modal interaction. By performing discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure through a pre-constructed Fourier graph neural network, low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts are obtained. Calculate the global mean vector of the low-frequency feature representation, and obtain a low-frequency sentiment consensus representation based on the global mean vector; calculate the high-frequency modal mean benchmark value of the high-frequency feature representation, and obtain a high-frequency discriminative representation based on the high-frequency modal mean benchmark value; The high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation. Based on the frequency domain comprehensive representation, sentiment category classification and prediction are performed to obtain sentiment prediction results.

2. The multimodal emotion recognition method according to claim 1, characterized in that, The step of extracting features from the multimodal emotion input data to obtain a multimodal feature representation includes: The discourse-level semantic feature representation of the sentiment input data of the text modality is extracted through a pre-trained language model; the pre-trained language model includes RoBERTa. Speech feature representations of speech modal emotion input data are extracted using pre-trained openSMILE; Video feature representations of video modality sentiment input data are extracted using a pre-trained 3D-CNN.

3. The multimodal emotion recognition method according to claim 1, characterized in that, The process of obtaining a low-frequency sentiment consensus representation based on the global mean vector includes: The matching degree between each modal feature representation and the global mean vector is calculated using an attention mechanism; The matching degree is normalized using Softmax to obtain the weights of each modality feature representation; The low-frequency feature representations are weighted and aggregated based on the weights to obtain a cross-modal low-frequency sentiment consensus representation.

4. The multimodal emotion recognition method according to claim 1, characterized in that, The process of performing sentiment category classification prediction based on the frequency domain comprehensive representation to obtain sentiment prediction results includes: The frequency domain synthesis representation is mapped back to the time domain space by inverse discrete Fourier transform; The comprehensive representation mapped back to the temporal space is then used to perform the final sentiment category classification prediction through a multilayer perceptron and a Softmax layer to obtain the sentiment prediction result.

5. The multimodal emotion recognition method according to claim 1, characterized in that, The calculation of the high-frequency modal mean benchmark value of the high-frequency feature representation, and the acquisition of the high-frequency discriminant representation based on the high-frequency modal mean benchmark value, includes: Calculate the benchmark value of the high-frequency modal mean based on the high-frequency feature representation; Calculate the residual of each modal high-frequency feature representation relative to the high-frequency modal mean reference value, and concatenate the residuals to obtain the modal deviation vector; By projecting the modal deviation vector back to the original dimension through linear mapping, a high-frequency discriminant representation for explicitly capturing modal dynamic conflicts is obtained.

6. The multimodal emotion recognition method according to claim 1, characterized in that, The step of injecting the high-frequency discriminative representation into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation includes: Generate a gated modulation vector based on the high-frequency discriminant representation; Based on the gated modulation vector, the high-frequency discriminative representation is injected into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation.

7. The multimodal emotion recognition method according to claim 1, characterized in that, The process of performing discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure using a pre-constructed Fourier graph neural network to obtain low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts includes: High-pass and low-pass filters are constructed based on the graph structure. The graph signal is separated based on the high-pass and low-pass filters to obtain high-frequency and low-frequency operators; a discrete Fourier transform is then performed on the graph signal. Based on the high-frequency and low-frequency operators, multi-layer depth propagation and nonlinear activation are performed on the discrete Fourier transform graph signal in the frequency domain space to obtain low-frequency feature representation and high-frequency feature representation.

8. A multimodal emotion recognition system for human dialogue, characterized in that, include: The acquisition module acquires multimodal emotional input data of the dialogue of the person to be identified, and performs feature extraction on the multimodal emotional input data to obtain multimodal feature representation; The identity embedding module adds the multimodal feature representation to the speaker identity embedding respectively to obtain a modal feature representation with context and person perception; The graph structure construction module uses the modal feature representations of each utterance in the corresponding dialogue sequence as graph nodes, and constructs contextual and cross-modal association edges between nodes based on a preset sliding window to build a graph structure for cross-modal interaction. The processing module performs discrete Fourier transform and frequency domain filtering on the graph signals in the graph structure through a pre-constructed Fourier graph neural network to obtain low-frequency feature representations reflecting global consistency and high-frequency feature representations reflecting local mutations and conflicts. The calculation module calculates the global mean vector of the low-frequency feature representation and obtains a low-frequency sentiment consensus representation based on the global mean vector; it also calculates the high-frequency modal mean benchmark value of the high-frequency feature representation and obtains a high-frequency discriminative representation based on the high-frequency modal mean benchmark value. The fusion module injects the high-frequency discriminative representation into the low-frequency sentiment consensus representation through a residual modulation mechanism to obtain a frequency domain comprehensive representation; The prediction module performs sentiment category classification prediction based on the frequency domain comprehensive representation to obtain sentiment prediction results.

9. A computer-readable storage medium, characterized in that, It stores a computer program for multimodal emotion recognition for dialogue-oriented tasks, wherein the computer program causes a computer to perform the multimodal emotion recognition method for dialogue-oriented tasks as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing multimodal emotion recognition for person-oriented dialogue as described in any one of claims 1 to 7.

Images