Emotion recognition method, system, terminal and storage medium based on federated multimodal global diffusion model

Abstract

The present invention relates to the technical field of conversation information analysis and discloses an emotion recognition method, system, terminal, and storage medium based on a federated multimodal global diffusion model. The method includes capturing conversation dependencies through a conversation graph network and enforcing semantic alignment through a semantic conditional network to ensure consistency between the recovered modality and the available modality in terms of context, speaker identity, and semantics. The method also periodically freezes the recovery module and classifier module to promote federated collaborative optimization. The present invention combines federated learning with generative modal recovery to address the problem of unpredictable modality loss on the client side in multimodal conversational emotion recognition scenarios, thereby improving the accuracy of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of conversation information analysis, and in particular to an emotion recognition method, system, terminal, and computer-readable storage medium based on a federated multimodal global diffusion model. Background Art

[0002] In recent years, with the development of artificial intelligence and the advent of the big data era, emotion recognition has become an emerging and active field in affective computing. Multimodal Emotion Recognition in Conversations (MERC) accurately identifies human emotions through the synergistic integration of multimodal signals such as text, acoustic, and visual signals.

[0003] However, in real-world scenarios, the emotion recognition effect of traditional MERC methods will be greatly reduced due to modality loss caused by sensor failure, environmental noise, or privacy restrictions.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide an emotion recognition method, system, terminal and computer-readable storage medium based on a federated multimodal global diffusion model, aiming to solve the problem in the prior art that the unpredictable loss of modality will significantly reduce the model performance, thereby leading to inaccurate emotion recognition results.

[0006] To achieve the above objectives, the present invention provides an emotion recognition method based on a federated multimodal global diffusion model, the emotion recognition method based on a federated multimodal global diffusion model comprising the following steps:

[0007] Obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and use a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a splicing feature;

[0008] Determining emotional information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotional information, and concatenating the multiple semantic representations with the multiple available modalities to obtain a fused representation;

[0009] The splicing feature and the fusion representation are fused and projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, and the conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise;

[0010] Training multiple clients according to the noise loss function to obtain multiple restoration model parameters, training a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model, determining multiple missing modes according to all the available modes, inputting all the missing modes into the multimodal global diffusion model, and outputting multiple restoration features generated when all the missing modes are restored;

[0011] All the recovered features and all the available modalities are integrated to obtain complete features. After optimizing a classifier using the complete features, the target conversation is input into the optimized classifier for recognition, and an emotion recognition result is output.

[0012] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein the step of acquiring a target conversation, inputting the target conversation into a conversation diffusion model, outputting a speaker graph and a temporal dependency graph, and performing node fusion on the speaker graph and the temporal dependency graph using a graph convolutional network to obtain a spliced ​​feature, specifically includes:

[0013] Obtaining a target conversation, inputting the target conversation into a conversation graph network in a conversation diffusion model, and determining, in the conversation graph network, multiple speeches in the target conversation and all available modalities corresponding to each speech;

[0014] The conversation graph network determines multiple speakers based on the speeches corresponding to all the available modalities, constructs corresponding nodes based on the speeches corresponding to each speaker, constructs a speaker subgraph corresponding to each speaker based on all the nodes, and fuses all the speaker subgraphs to obtain and output a speaker graph;

[0015] The conversation graph network determines the temporal relationship between each of the speeches according to all the speeches corresponding to the available modalities, and adds all the temporal relationships as side information to the speaker graph to obtain and output a temporal dependency graph;

[0016] Inputting the speaker graph and the temporal dependency graph into a constructed graph convolutional network, the graph convolutional network fuses all nodes in the speaker graph and all nodes and edge information in the temporal dependency graph to obtain multiple dialogue dependency feature representations;

[0017] The graph convolutional network concatenates all the conversation dependency feature representations to obtain multiple conversation dependency vectors, and stacks all the conversation dependency vectors along the dimension of the conversation graph network to obtain and output concatenated features.

[0018] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein determining the emotion information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotion information, and concatenating the multiple semantic representations with the multiple available modalities to obtain a fused representation, specifically includes:

[0019] Inputting the target dialogue into a semantic conditional network, the semantic conditional network predicts the speeches corresponding to all the available modalities to obtain emotional information of each speech, wherein the emotional information represents the probability that the speech belongs to a certain emotional category;

[0020] Based on all the emotional information, a single available modality is extracted one by one, and the cross-attention mechanism is used to perform feature fusion and alignment processing on the available modality and other available modalities, and finally a semantic representation corresponding to each available modality is generated:

[0021] ;

[0022] ;

[0023] in, and Represent the first available mode and the second available mode respectively, express Features, express Features, Indicates Come to Perform fusion alignment processing, Indicates Come to Perform fusion alignment processing, represents the cross attention mechanism, Indicates normalization processing;

[0024] After all the semantic representations are fused by category, a fused representation sequence corresponding to each modal category is obtained, and all the fused representation sequences are spliced ​​to obtain a fused representation, and an emotion classification result is generated based on the fused representation.

[0025] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein the emotion information of all available modalities in the target conversation is determined, multiple semantic representations are generated based on all the emotion information, and multiple available modalities are fused and then spliced ​​to obtain an emotion classification result, further includes:

[0026] The splicing features are input into the self-attention layer, the semantic classification results are output, and based on the semantic classification results, the conversation graph loss function of the conversation graph network is constructed:

[0027] ;

[0028] in, represents the conversation graph loss function, represents the number of classifications after classification in the self-attention layer, Indicates the number of speeches. The index indicating the number of speeches. Indicates the It's a real label. Represents the semantic classification result, Represents the dimension of the conversation graph network;

[0029] According to the emotion classification results, the semantic condition loss function of the semantic condition network is constructed:

[0030] ;

[0031] in, represents the semantic conditional loss function, Indicates the emotion classification results;

[0032] According to the conversation graph loss function and the semantic condition loss function, the total network loss function is constructed:

[0033] ;

[0034] in, represents the total network loss function.

[0035] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein the concatenated features and the fused representation are fused and then projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, and then a conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise, specifically comprising:

[0036] The spliced ​​features and the fused representation are input into the DISC diffusion network. After the spliced ​​features and the fused representation are spliced ​​through the embedding layer of the DISC diffusion network, they are mapped to the conditional embedding space to obtain the fusion condition:

[0037] ;

[0038] in, represents the fusion condition, Represents the splicing feature, represents the fusion representation, represents linear splicing processing, Indicates structural remodeling processing, represents the norm symbol;

[0039] The fusion condition is input into the noise prediction network, and the noise prediction network fuses the channel features in multiple channels with the fusion condition respectively to map the fusion condition to the upsampling feature or the channel corresponding to the downsampling stage:

[0040] ;

[0041] in, Indicates the Channel features, Indicates the Channel features, Indicates fusion processing, represents the cross attention mechanism, Indicates linear splicing processing;

[0042] In the plurality of channels, removing the fusion condition with a fixed probability, and outputting a plurality of conditional noise predictions and a plurality of unconditional noise predictions;

[0043] All the conditional noise predictions and all the unconditional noise predictions are weightedly fused to obtain the conditional prediction noise:

[0044] ;

[0045] in, The index representing the number of diffusion steps, Indicates the The conditional prediction noise of the step diffusion, represents the enhanced weight, represents the noise prediction output, represents the conditional noise prediction, represents the unconditional noise prediction, represents the prediction noise, represents the network parameters of the noise prediction network;

[0046] All conditional fusion features are iteratively forward diffused and iteratively reverse diffused, and the conditional prediction noise is used to construct the noise loss function:

[0047] ;

[0048] in, represents the noise loss function, Indicates that the random draw is no greater than A positive integer, represents the number of diffusion steps, represents a discrete uniform distribution, represents the initial noise, represents the random variable of the initial noise, express The probability distribution function of represents the conditional prediction noise, represents the noise distribution of random draws according to the standard multivariate normal distribution, represents a normal distribution, represents the identity matrix, Express 、 and The expectation of the joint distribution of It means calculating the square of the Euclidean distance between the predicted noise and the true noise.

[0049] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein the steps of training multiple clients according to the noise loss function to obtain multiple recovery model parameters, training a recovery model according to all the recovery model parameters to obtain a multimodal global diffusion model, determining multiple missing modalities according to all the available modalities, inputting all the missing modalities into the multimodal global diffusion model, and outputting multiple recovery features generated when all the missing modalities are recovered, specifically include:

[0050] Inputting the noise loss function into multiple clients, all of the clients optimizing their respective local recovery models according to the noise loss function to obtain corresponding recovery model parameters, and receiving the recovery model parameters sent by all of the clients;

[0051] Performing weighted aggregation on all the restoration model parameters to obtain global parameters, and optimizing the constructed multimodal global diffusion model according to the global parameters;

[0052] The missing mode corresponding to each of the clients is input into the multimodal global diffusion model, and the corresponding recovery features are output.

[0053] Optionally, the emotion recognition method based on the federated multimodal global diffusion model, wherein the fusion of all the recovered features and all the available modalities to obtain a complete feature, optimizing a classifier using the complete feature, inputting the target conversation into the optimized classifier for recognition, and outputting an emotion recognition result, specifically includes:

[0054] Closing the multimodal global diffusion model, fusing all the restored features corresponding to each client and all the available modalities to obtain a complete feature of each client;

[0055] Utilizing the complete features of each client, optimizing the sub-classifier corresponding to the client, and obtaining classifier parameters corresponding to each client;

[0056] After weighted fusion of all the classifier parameters, the classifier is optimized, the target conversation is input into the optimized classifier for prediction and output of the emotion recognition result.

[0057] In addition, to achieve the above-mentioned object, the present invention further provides an emotion recognition system based on a federated multimodal global diffusion model, wherein the emotion recognition system based on a federated multimodal global diffusion model includes:

[0058] A conversation analysis module is used to obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and use a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a splicing feature;

[0059] A semantic analysis module is configured to determine the emotional information of all available modalities in the target conversation, generate multiple semantic representations based on all the emotional information, and concatenate the multiple semantic representations with the multiple available modalities to obtain a fused representation;

[0060] An information fusion module is used to fuse the splicing features and the fused representation and project them into a conditional embedding space to obtain a fusion condition, input the fusion condition into a noise prediction network for fusion processing, output conditional predicted noise, and construct a noise loss function based on the conditional predicted noise;

[0061] a model training module, configured to train multiple clients according to the noise loss function to obtain multiple restoration model parameters, train a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model, determine multiple missing modes according to all the available modes, input all the missing modes into the multimodal global diffusion model, and output multiple restoration features generated when all the missing modes are restored;

[0062] The result prediction module is used to fuse all the recovered features and all the available modalities to obtain complete features, optimize the classifier using the complete features, input the target dialogue into the optimized classifier for recognition, and output the emotion recognition result.

[0063] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an emotion recognition program based on a federated multimodal global diffusion model stored in the memory and runnable on the processor, wherein the emotion recognition program based on a federated multimodal global diffusion model implements the steps of the emotion recognition method based on a federated multimodal global diffusion model as described above when executed by the processor.

[0064] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an emotion recognition program based on a federated multimodal global diffusion model, and when the emotion recognition program based on a federated multimodal global diffusion model is executed by a processor, the steps of the emotion recognition method based on a federated multimodal global diffusion model as described above are implemented.

[0065] In the present invention, a target conversation is obtained, the target conversation is input into a conversation diffusion model, a speaker graph and a temporal dependency graph are output, and a graph convolutional network is used to perform node fusion on the speaker graph and the temporal dependency graph to obtain a spliced ​​feature; the emotional information of all available modalities in the target conversation is determined, multiple semantic representations are generated based on all the emotional information, and the multiple semantic representations are spliced ​​with the multiple available modalities to obtain a fused representation; the spliced ​​feature and the fused representation are fused and projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, and the conditional prediction noise is output. The noise loss function is constructed based on the conditional noise prediction; multiple clients are trained based on the noise loss function to obtain multiple recovery model parameters; a recovery model is trained based on all the recovery model parameters to obtain a multimodal global diffusion model; multiple missing modalities are determined based on all the available modalities, and all the missing modalities are input into the multimodal global diffusion model to output multiple recovery features generated when all the missing modalities are restored; all the recovery features and all the available modalities are fused to obtain complete features, and after optimizing the classifier using the complete features, the target conversation is input into the optimized classifier for recognition, and the emotion recognition result is output. The present invention combines federated learning with generative modal recovery to solve the problem of unpredictable modal missingness on the client side in the multimodal conversation emotion recognition scenario, thereby improving the accuracy of emotion prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a preferred embodiment of the emotion recognition method based on the federated multimodal global diffusion model of the present invention;

[0067] Figure 2 This is a diffusion model structure diagram of a preferred embodiment of the emotion recognition method based on the federated multimodal global diffusion model of the present invention;

[0068] Figure 3 1 is a schematic diagram of model training of a preferred embodiment of the emotion recognition method based on the federated multimodal global diffusion model of the present invention;

[0069] Figure 4 1 is a schematic diagram of modal response under a single available modality of a preferred embodiment of the emotion recognition method based on the federated multimodal global diffusion model of the present invention;

[0070] Figure 5 Schematic diagram of the results of a visualization ablation experiment of a preferred embodiment of the emotion recognition method based on the federated multimodal global diffusion model of the present invention;

[0071] Figure 6 1 is a structural diagram of a preferred embodiment of an emotion recognition system based on a federated multimodal global diffusion model of the present invention;

[0072] Figure 7 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] Existing recovery methods estimate and reconstruct the features of the missing modalities, so that downstream tasks can "superficially" recover to the input conditions of a complete multimodal input. This paradigm is mainly divided into two subcategories: latent space semantic recovery and explicit modal recovery. Latent space semantic recovery, including mining the deep semantic associations between multimodalities in the "latent space", thus using the observed modal side information to compensate for the missing modality. Typical representatives include: (1) GCNet (Global Context Network): constructing cross-modal temporal and speaker dependencies in the latent space through graph neural networks, using the temporal sequence and speaker information of the modalities available in the conversation to infer and compensate for the semantic features of the missing modality; (2) SDR-GNN (Spectral Domain Reconstruction Graph Neural Network) extends the GCNet idea and uses graph neural networks to align multi-frequency signals to achieve more comprehensive cross-modal feature alignment and compensation. It performs consistent alignment of multi-frequency features in the latent space, so that the semantic information of the missing modality can be mapped and restored more accurately.

[0075] However, these two approaches suffer from the drawbacks of feature confusion and semantic distortion, as well as failure to address extreme distributions. Explicit modality recovery methods also suffer from excessive reliance on complete data and generation failures due to missing distribution priors. Therefore, this paper proposes a multimodal emotion recognition method based on Federated Dialogue-guided and Semantic-Consistent Diffusion (FedDISC), which for the first time introduces federated learning to recover missing modalities. By federating and aggregating modality-specific diffusion models trained separately on clients and broadcasting the aggregated model to clients with missing modalities, FedDISC overcomes the dependency of individual clients on modality completeness. Furthermore, the DISC-Diffusion (Dialogue-guided and Semantic-Consistent Diffusion) module captures conversational dependencies through a dialogue graph network, and enforces semantic alignment through a semantic conditional network, ensuring that the recovered modality is consistent with the available modalities in terms of context, speaker identity, and semantics. The present invention further introduces an innovative alternating freezing aggregation strategy, which periodically freezes the recovery module and the classifier module to promote federated collaborative optimization.

[0076] The emotion recognition method based on the federated multimodal global diffusion model described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the emotion recognition method based on the federated multimodal global diffusion model includes the following steps:

[0077] Step S10: Obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and use a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a splicing feature.

[0078] Among them, DISC-Diffusion is the core module of the present invention for restoring missing modalities. Its main idea is to explicitly generate feature representations of missing modalities in the feature space based on the conditional diffusion model. This module provides more accurate context and semantic information through the Dialogue Graph Network (DGN) and the Semantic Conditioning Network (SCN) to guide the diffusion process.

[0079] Specifically, a target conversation is obtained, the target conversation is input into a conversation graph network in a conversation diffusion model, and multiple speeches in the target conversation and all available modalities corresponding to each speech are determined in the conversation graph network; the conversation graph network determines multiple speakers based on the speeches corresponding to all the available modalities, constructs corresponding nodes based on the speeches corresponding to each speaker, and constructs a speaker subgraph corresponding to each speaker based on all the nodes, fuses all the speaker subgraphs, and obtains and outputs a speaker graph; the conversation graph network determines each speaker based on the speeches corresponding to all the available modalities. The method comprises the following steps: first, determining the temporal relationship between speeches, and adding all the temporal relationships as side information to the speaker graph to obtain and output a temporal dependency graph; inputting the speaker graph and the temporal dependency graph into the constructed graph convolutional network, and fusing all nodes in the speaker graph and all nodes and side information in the temporal dependency graph to obtain multiple dialogue dependency feature representations; and finally, concatenating all the dialogue dependency feature representations to obtain multiple dialogue dependency vectors, and stacking all the dialogue dependency vectors along the dimension of the dialogue graph network to obtain and output a concatenated feature.

[0080] Among them, Figure 2 As shown in Figure 2, DGN extracts speaker association and temporal dependency information in the conversation context by constructing speaker graph and context graph, respectively. Figure 2 There are multiple speaker graphs (i.e., speaker graphs) and context graphs in the context graph. The splicing process involves splicing each pair of speaker graphs and context graphs. Because different utterances in the same conversation are related not only in terms of textual content but also in terms of speaker identity and temporal order, DGN models these dependencies through graph convolutional structures, helping the diffusion model to take into account the overall conversation context when generating missing modalities.

[0081] For a target dialogue, including multiple speeches (such as Figure 2 in 、 and ), each speech corresponds to multiple modalities, including visual features, acoustic features, text features, and other features (such as Figure 2 in 、 、 、 and ), and in constructing the network topology graph (i.e., the speaker graph and the timing dependency graph, where the timing dependency graph is Figure 2When constructing a context graph in speech, all available modalities in each speech are selected, and the available visual and acoustic features in each speech are defined as nodes to construct the graph. The degree of association between two nodes is then defined as the weight of the edge to quantify the graph structure. The type of edge reflects the communication mode between nodes. Through the graph structure, the dependencies between the previous and next speeches of the same speaker and the adjacent sentences of the conversation are explicitly retained, avoiding the deficiency of traditional temporal encoders in losing cross-information between speakers.

[0082] Furthermore, the speaker graph uses the relationship between speakers to capture speaker dependencies in a conversation. For example, if a conversation is cut into multiple segments, each segment will belong to a specific speaker, and the emotion category of each segment has a certain dependency relationship with the speaker to which the segment belongs. The present invention uses the speaker graph to capture this relationship; the temporal dependency graph aims to capture the contextual temporal relationship between nodes. The temporal relationship includes three types of relationships: forward, reverse, and present. The forward direction represents the timing of the first speaker speaking to the second speaker, the reverse direction represents the timing of the second speaker speaking to the first speaker, and the present direction represents the timing of the current speaker speaking to the current speaker; the captured conversation dependency information helps the diffusion model maintain consistency with the text modality when restoring missing visual or audio, thereby reducing semantic drift.

[0083] Furthermore, we apply graph convolutional networks to the two graphs, perform message passing and aggregation on node features, and finally obtain multiple dialogue dependency feature representations at each node position (such as Figure 2 in 、 、 、 and ), after concatenating them, we get the final dialogue dependency vector (such as Figure 2 in 、 、 and ), and then all dialogue dependency vectors are stacked along the dimension of the sample to obtain the concatenated features. Figure 2 in 、 and Represent three different modes respectively. 、 、 and They represent different edges.

[0084] Step S20: determining the emotional information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotional information, and concatenating the multiple semantic representations with the multiple available modalities to obtain a fused representation.

[0085] Among them, SCN is used to extract high-level semantic information from each available modality, and obtain semantic representations aligned with other modalities through multi-head cross-attention fusion. The purpose is to provide precise guidance at the semantic level for diffusion generation, so that the recovered modality is consistent with the contextual semantics.

[0086] Specifically, the target dialogue is input into a semantic conditional network, which predicts the speeches corresponding to all the available modalities and obtains the emotional information of each speech, wherein the emotional information represents the probability that the speech belongs to a certain emotional category; based on all the emotional information, a single available modality is extracted one by one, and the cross-attention mechanism is used to perform feature fusion and alignment processing on the available modality and other available modalities, and finally a semantic representation corresponding to each available modality is generated:

[0087] ;

[0088] ;

[0089] in, and Represent the first available mode and the second available mode respectively, express Features, express Features, Indicates Come to Perform fusion alignment processing, Indicates Come to Perform fusion alignment processing, represents the cross attention mechanism, Representation normalization processing; after fusing all the semantic representations by category, a fusion representation sequence corresponding to each modal category is obtained, and all the fusion representation sequences are spliced ​​to obtain a fusion representation, and an emotion classification result is generated based on the fusion representation.

[0090] Among them, high-level semantic information represents the probability that each speech belongs to a certain emotion category, that is, the above-mentioned emotion information. After obtaining the emotion information, the available modalities are extracted one by one, and then the semantic representations aligned with other available modalities are obtained, and finally a semantic representation corresponding to each available modality is obtained. In order to capture the semantic consistency across modalities, the present invention uses a cross-attention layer to realize modal fusion, and then for each fused fusion representation sequence, the head of the sequence is spliced ​​and used as the input of the full-link layer to finally obtain the emotion classification result; wherein, the cross-attention mechanism enables information of different modalities to refer to each other, and the generated semantic summary is more focused on the common semantic areas between the modalities, reducing the semantic deviation between modalities.

[0091] Furthermore, the splicing features are input into the self-attention layer, a semantic classification result is output, and based on the semantic classification result, a conversation graph loss function of the conversation graph network is constructed:

[0092] ;

[0093] in, represents the conversation graph loss function, represents the number of classifications after classification in the self-attention layer, Indicates the number of speeches. The index indicating the number of speeches. Indicates the It's a real label. Represents the semantic classification result, Represents the dimension of the conversation graph network; based on the emotion classification results, the semantic condition loss function of the semantic condition network is constructed:

[0094] ;

[0095] in, represents the semantic conditional loss function, Represents the emotion classification result; based on the conversation graph loss function and the semantic condition loss function, the total network loss function is constructed:

[0096] ;

[0097] in, represents the total network loss function.

[0098] Among them, the splicing features are obtained by stacking, and the classification results are obtained after passing through the self-attention layer and the classification layer. At the same time, the loss function of DGN is constructed together according to the real labels of the input features. Similarly, after SCN obtains the emotion classification results, the real labels of the input features are used to construct the loss function of SCN. Based on these two loss functions, a total loss function is constructed. The total loss function can be used as an optimized variance to pre-train the FedDISC module. Figure 3 As shown in the figure, the training process of FedDISC is shown, where Sever represents the local server, and E, R and C in each client represent different modes. Then represents the total loss function; the conversation dependencies are captured by the conversation graph network, and the semantic condition network enforces semantic alignment to ensure that the recovered modality is consistent with the available modality in context, speaker identity and semantics.

[0099] Step S30: The splicing feature and the fusion representation are fused and projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, the conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise.

[0100] Among them, the noise prediction network, as the basic backbone of the diffusion network, usually contains symmetrical downsampling and upsampling branches, and retains multi-scale features through jump connections.

[0101] Specifically, the spliced ​​features and the fused representation are input into the DISC diffusion network, and after the spliced ​​features and the fused representation are spliced ​​through the embedding layer of the DISC diffusion network, they are mapped to the conditional embedding space to obtain the fusion condition:

[0102] ;

[0103] in, represents the fusion condition, Represents the splicing feature, represents the fusion representation, represents linear splicing processing, Indicates structural remodeling processing, Represents the norm symbol; the fusion condition is input into the noise prediction network, and the noise prediction network fuses the channel features in multiple channels with the fusion condition respectively to map the fusion condition to the upsampling feature or the channel corresponding to the downsampling stage:

[0104] ;

[0105] in, Indicates the Channel features, Indicates the Channel features, Indicates fusion processing, represents the cross attention mechanism, represents linear splicing processing; in the multiple channels, the fusion condition is removed with a fixed probability, and multiple conditional noise predictions and multiple unconditional noise predictions are output; all the conditional noise predictions and all the unconditional noise predictions are weightedly fused to obtain conditional prediction noise:

[0106] ;

[0107] in, The index representing the number of diffusion steps, Indicates the The conditional prediction noise of the step diffusion, represents the enhanced weight, represents the noise prediction output, represents the conditional noise prediction, represents the unconditional noise prediction, represents the prediction noise, Represents the network parameters of the noise prediction network; all conditional fusion features are iteratively forward diffused and iteratively reverse diffused, and the noise loss function is constructed using the conditional prediction noise:

[0108] ;

[0109] in, represents the noise loss function, Indicates that the random draw is no greater than A positive integer, represents the number of diffusion steps, represents a discrete uniform distribution, represents the initial noise, represents the random variable of the initial noise, express The probability distribution function of represents the conditional prediction noise, represents the noise distribution of random draws according to the standard multivariate normal distribution, represents a normal distribution, represents the identity matrix, Express 、 and The expectation of the joint distribution of It means calculating the square of the Euclidean distance between the predicted noise and the true noise.

[0110] Among them, in each upsampling or downsampling stage of the noise prediction network, the current features of the current channel input are cross-attention fused with the fusion conditions in the conditional embedding space. During this training process, the fusion conditions are randomly "discarded" with a fixed probability, so that the network is allowed to learn unconditional noise prediction at this time. In the subsequent reasoning process, the conditional noise prediction and the unconditional noise prediction are weightedly fused to obtain the conditional prediction noise.

[0111] Furthermore, for each diffusion step in the diffusion process, a noise loss function can be constructed by defining the training objective to minimize the mean square error between the predicted noise and the actual noise, and combining the conditional prediction noise; guided by the fusion condition, the missing mode and the available mode of the response obtained after sampling have consistent emotional information, which is subsequently used to train the local modal diffusion models of different clients, so that each local modal diffusion model can perform targeted optimization training based on its own missing mode, and can accurately generate the missing mode, thereby optimizing the output results of each local modal diffusion model.

[0112] Step S40: training multiple clients according to the noise loss function to obtain multiple recovery model parameters, training the recovery model according to all the recovery model parameters to obtain a multimodal global diffusion model, determining multiple missing modes according to all the available modes, inputting all the missing modes into the multimodal global diffusion model, and outputting multiple recovery features generated when all the missing modes are restored.

[0113] Among them, in order to solve the problems of gradient conflict and inconsistency of optimization objectives in the joint training process of the generative recovery module and the classifier module, the present invention proposes an "alternating freezing aggregation strategy". The core idea is to activate the recovery module or the classifier module in stages, and freeze the corresponding modules alternately on the client and server.

[0114] Specifically, the noise loss function is input into multiple clients, all the clients optimize their respective local response models according to the noise loss function to obtain corresponding recovery model parameters, and receive the recovery model parameters sent by all the clients; all the recovery model parameters are weightedly aggregated to obtain global parameters, and the constructed multimodal global diffusion model is optimized according to the global parameters; the missing mode corresponding to each of the clients is input into the multimodal global diffusion model, and the corresponding recovery features are output.

[0115] For each client, the encoder module maps the input modality (text, visual, or acoustic) into a unified high-level feature representation, typically using a pretrained network. This module is a pretrained model and is frozen at all stages, not participating in any local updates, to ensure consistent and reproducible feature extraction. DISC-Diffusion, on the other hand, is specifically designed for generative recovery of missing modalities. It is activated for parameter updates during the recovery module training phase and frozen during the classifier optimization phase.

[0116] Specifically, the classifier is first frozen, and only the recovery module (DISC-Diffusion) is updated locally. Multiple local training runs are performed for each available modality to minimize the diffusion model loss. After training, the local recovery model parameters (recovery model parameters) are uploaded to the server. The server then performs a weighted aggregation of the recovery model parameters for each client to obtain global parameters, which are used to train the multimodal global diffusion model. Simultaneously, the updated global parameters are broadcast to all clients for the next stage of training.

[0117] Step S50: Fusing all the restored features and all the available modalities to obtain complete features, optimizing a classifier using the complete features, inputting the target conversation into the optimized classifier for recognition, and outputting an emotion recognition result.

[0118] In the classifier stage, the complete, multimodal concatenated features are received and the sentiment classification result is output through a series of attention layers and fully connected layers. In the "classifier optimization stage", it is activated to update the parameters, while the multimodal global diffusion model is frozen.

[0119] Specifically, the multimodal global diffusion model is turned off, and all the recovered features and all the available modalities corresponding to each client are fused to obtain the complete features of each client; the complete features of each client are used to optimize the sub-classifier corresponding to the client to obtain the classifier parameters corresponding to each client; all the classifier parameters are weightedly fused and then the classifier is optimized, and the target conversation is input into the optimized classifier for prediction and output of the emotion recognition result.

[0120] During this process, only the classifier module is updated locally. The missing modalities are restored using the multimodal global diffusion model broadcast by the server to obtain the restored features. The restored features are concatenated with the available modal features to form a complete feature set. The local classifier is trained using the complete feature set, minimizing the cross-entropy loss. The local classifier parameters are then uploaded to the server. The classifier parameters uploaded by the participating clients are aggregated using a weighted average to obtain a global classifier, which is then broadcast to all participating clients.

[0121] In these two separate processes, the recovery phase does not update the classifier parameters, and the classifier phase does not update the recovery model parameters; the updates in the parameter space between the two do not interfere with each other. The recovery module focuses on learning how to better reconstruct the missing modality in its own phase, while the classifier fully utilizes the recovery results for sentiment discrimination in the next phase. This creates a virtuous cycle, avoids adjusting two large-scale parameter sets simultaneously in the same round, reduces unnecessary optimization jitter, and accelerates global model convergence.

[0122] The present invention combines federated learning with generative modality recovery to solve the problem of unpredictable modality loss on the client side in multimodal dialogue emotion recognition scenarios, thereby improving the accuracy of emotion prediction.

[0123] Furthermore, to validate the effectiveness of the proposed model, we conducted extensive experiments on three benchmark datasets: IEMOCAP (The Interactive Emotional Dyadic Motion Capture), CMUMOSI (Multimodal Opinion Sentiment Intensity, a dataset that integrates text, audio, and video modalities and annotates sentiment intensity values ​​focused on opinion-based content), and CMUMOSEI (Multimodal Opinion Sentiment and Emotion Intensity, an extension of CMUMOSI that adds annotations for emotion categories). The proposed model, FedDISC, was comprehensively compared with other advanced MERC methods (a technique for emotion recognition in multimodal conversations), including the modality recovery model (GCNet (Global Context Network), MMIN (Missing Modality Imagination Network), SDR-GNN (Software Defined Radio-based Graph Neural Network), IMDer (a model for addressing the problem of incomplete multimodal emotion recognition), DiCMoR (Discrete Choice Modeling with Robustness), and DCCAE (a non-restoration model). To verify the generalization ability of the proposed model, the present invention uses DDPM (Denoising Diffusion Probabilistic Model) and DDIM (Denoising Diffusion Implicit Models) to implement the conditional diffusion model part.

[0124] Furthermore, for the fixed missing protocol, that is, all data of one or more modalities are fixed missing, it is used to simulate the more extreme modal missing conditions in real scenarios. The performance of each model under the fixed missing protocol is listed in Table 1:

[0125] Table 1: Comparison of IEMOCAP4 and IEMOCAP6 data results

[0126]

[0127] The best results are marked in dark text. On the IEMOCAP4 dataset (a dataset representing the four-category sentiment subset of IEMOCAP), FedDISC(P) (FedDISC(P) using the DDPM sampling method, and FedDISC(I) using the DDIM sampling method) delivered state-of-the-art classification results in all scenarios, improving accuracy by 1.3% to 7.5% over the best state-of-the-art methods. On the IEMOCAP6 dataset (a dataset representing the six-category sentiment subset of IEMOCAP), FedDISC(P) lagged slightly behind SDR-GNN by approximately 0.9% to 3.2% in some incomplete modality settings ({l}, {l, v}, {l, a}). However, considering that federated clients are only assigned a few samples, while other state-of-the-art methods have access to complete training data, this result highlights the ability of our method to generate high-quality missing modality features in a few-shot setting.

[0128] Among them, Figure 4 As shown in Figure 2, the distribution of missing modal features generated by FedDISC and other recovery-based methods is compared with the distribution of real features under the condition of fixed missing protocol and only single modality available, where Indicates modality The original characteristics of Indicates modality The original characteristics of Indicates modality The original characteristics of Indicates modality The original characteristics of Indicates modality The original characteristics of Indicates modality Experiments were conducted on the IEMOCAP test set (from a single client) and these features were mapped to a two-dimensional space using t-SNE (t-Distributed Stochastic Neighbor Embedding). Results show that FedDISC(P) has the highest denoising ability, with its predicted features highly consistent with the original distribution. FedDISC(I) is slightly inferior to FedDISC(P) in recovering the l and v modalities, but still outperforms other recovery methods. This demonstrates that FedDISC is significantly effective in ensuring distributional consistency between recovered and original data.

[0129] Furthermore, for random missing protocols, which are used to simulate relatively random modal missingness, the present invention uses the missingness rate to control the degree of modal missingness. A higher missingness rate indicates more severe modal missingness. Table 2 below shows the comparative results for missingness rates ranging from 0.0 to 0.7. On the CMUMOSI and CMUMOSEI datasets, when the missingness rate is below 0.4, FedDISC is slightly lower than the state-of-the-art model in terms of accuracy and weighted ratio. However, when the missingness rate is greater than 0.4, FedDISC achieves the highest accuracy and weighted ratio.

[0130] Table 2: Comparison results under different missing rates

[0131]

[0132] Furthermore, in another embodiment of the present invention, the importance of the conversation and semantic dependency extracted by DGN and SCN in the DISC-Diffusion model proposed by the present invention is verified through visual ablation experiments. Figure 5 As shown in the figure, the restored modal features and original modal features under two conditions on the IEMOCAP6 dataset: (1) unconditional restoration; (2) restoration based on DGN and SCN conditions. It can be observed that under the unconditional restoration setting, due to the lack of semantic guidance, the restored features show obvious semantic ambiguity. However, after conditional restoration under the guidance of DGN and SCN, the restored modalities and the original modalities of the same category are closely clustered in the two-dimensional space, while maintaining a clear separation between the categories. This verifies the effectiveness of DGN and SCN in capturing comprehensive conversation and semantic information.

[0133] Further, if Figure 6 As shown, based on the above-mentioned emotion recognition method based on the federated multimodal global diffusion model, the present invention also provides an emotion recognition system based on the federated multimodal global diffusion model, wherein the emotion recognition system based on the federated multimodal global diffusion model includes:

[0134] The conversation analysis module 51 is used to obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and perform node fusion on the speaker graph and the temporal dependency graph using a graph convolutional network to obtain a splicing feature;

[0135] A semantic analysis module 52 is configured to determine the emotional information of all available modalities in the target conversation, generate multiple semantic representations based on all the emotional information, and concatenate the multiple semantic representations with the multiple available modalities to obtain a fused representation;

[0136] An information fusion module 53 is configured to fuse the splicing features and the fused representation and project them into a conditional embedding space to obtain a fusion condition, input the fusion condition into a noise prediction network for fusion processing, output a conditional predicted noise, and construct a noise loss function based on the conditional predicted noise;

[0137] a model training module 54 configured to train multiple clients according to the noise loss function to obtain multiple restoration model parameters, train a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model, determine multiple missing modes according to all the available modes, input all the missing modes into the multimodal global diffusion model, and output multiple restoration features generated when all the missing modes are restored;

[0138] The result prediction module 55 is used to fuse all the restored features and all the available modalities to obtain complete features, optimize the classifier using the complete features, input the target dialogue into the optimized classifier for recognition, and output the emotion recognition result.

[0139] Further, if Figure 7 As shown, based on the above-mentioned emotion recognition method and system based on the federated multimodal global diffusion model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0140] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program codes of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an emotion recognition program 40 based on a federated multimodal global diffusion model is stored on the memory 20. The emotion recognition program 40 based on a federated multimodal global diffusion model can be executed by the processor 10, thereby realizing the emotion recognition method based on a federated multimodal global diffusion model in the present application.

[0141] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the emotion recognition method based on the federated multimodal global diffusion model.

[0142] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0143] In one embodiment, when the processor 10 executes the emotion recognition program 40 based on the federated multimodal global diffusion model in the memory 20 , the steps of the emotion recognition method based on the federated multimodal global diffusion model as described above are implemented.

[0144] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an emotion recognition program based on a federated multimodal global diffusion model, and when the emotion recognition program based on a federated multimodal global diffusion model is executed by a processor, the steps of the emotion recognition method based on a federated multimodal global diffusion model as described above are implemented.

[0145] In summary, the present invention provides an emotion recognition method and related equipment based on a federated multimodal global diffusion model, the method comprising: obtaining a target conversation, inputting the target conversation into a conversation diffusion model, outputting a speaker graph and a temporal dependency graph, and using a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a spliced ​​feature; determining the emotion information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotion information, splicing the multiple semantic representations with the multiple available modalities to obtain a fused representation, and generating an emotion classification result based on the fused representation; fusing the spliced ​​feature and the fused representation and then projecting them into a conditional embedding space to obtain a fusion condition, and then After the fusion conditional input noise prediction network is fused and processed, the conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise; multiple clients are trained based on the noise loss function to obtain multiple recovery model parameters, and a recovery model is trained based on all the recovery model parameters to obtain a multimodal global diffusion model; multiple missing modalities are determined based on all the available modalities, and all the missing modalities are input into the multimodal global diffusion model, outputting multiple recovery features generated when all the missing modalities are restored; all the recovery features and all the available modalities are fused to obtain complete features, and after optimizing the classifier using the complete features, the target dialogue is input into the optimized classifier for recognition, and the emotion recognition result is output. The present invention combines federated learning with generative modal recovery to solve the problem of unpredictable modality loss on the client side in the multimodal dialogue emotion recognition scenario, thereby improving the accuracy of emotion prediction.

[0146] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0147] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0148] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An emotion recognition method based on a federated multimodal global diffusion model, characterized in that: The emotion recognition method based on the federated multimodal global diffusion model includes: Obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and use a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a splicing feature; Determining emotional information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotional information, and concatenating the multiple semantic representations with the multiple available modalities to obtain a fused representation; The splicing feature and the fusion representation are fused and projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, and the conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise; Training multiple clients according to the noise loss function to obtain multiple restoration model parameters, training a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model, determining multiple missing modes according to all the available modes, inputting all the missing modes into the multimodal global diffusion model, and outputting multiple restoration features generated when all the missing modes are restored; All the recovered features and all the available modalities are integrated to obtain complete features. After optimizing a classifier using the complete features, the target conversation is input into the optimized classifier for recognition, and an emotion recognition result is output.

2. The emotion recognition method based on the federated multimodal global diffusion model according to claim 1, characterized in that: The step of obtaining a target conversation, inputting the target conversation into a conversation diffusion model, outputting a speaker graph and a temporal dependency graph, and performing node fusion on the speaker graph and the temporal dependency graph using a graph convolutional network to obtain a splicing feature specifically includes: Obtaining a target conversation, inputting the target conversation into a conversation graph network in a conversation diffusion model, and determining, in the conversation graph network, multiple speeches in the target conversation and all available modalities corresponding to each speech; The conversation graph network determines multiple speakers based on the speeches corresponding to all the available modalities, constructs corresponding nodes based on the speeches corresponding to each speaker, constructs a speaker subgraph corresponding to each speaker based on all the nodes, and fuses all the speaker subgraphs to obtain and output a speaker graph; The conversation graph network determines the temporal relationship between each of the speeches according to all the speeches corresponding to the available modalities, and adds all the temporal relationships as side information to the speaker graph to obtain and output a temporal dependency graph; Inputting the speaker graph and the temporal dependency graph into a constructed graph convolutional network, the graph convolutional network fuses all nodes in the speaker graph and all nodes and edge information in the temporal dependency graph to obtain multiple dialogue dependency feature representations; The graph convolutional network concatenates all the conversation dependency feature representations to obtain multiple conversation dependency vectors, and stacks all the conversation dependency vectors along the dimension of the conversation graph network to obtain and output concatenated features.

3. The emotion recognition method based on the federated multimodal global diffusion model according to claim 1, characterized in that: Determining the emotional information of all available modalities in the target conversation, generating multiple semantic representations based on all the emotional information, and concatenating the multiple semantic representations with the multiple available modalities to obtain a fused representation specifically includes: Inputting the target dialogue into a semantic conditional network, the semantic conditional network predicts the speeches corresponding to all the available modalities to obtain emotional information of each speech, wherein the emotional information represents the probability that the speech belongs to a certain emotional category; Based on all the emotional information, a single available modality is extracted one by one, and the cross-attention mechanism is used to perform feature fusion and alignment processing on the available modality and other available modalities, and finally a semantic representation corresponding to each available modality is generated: ； ； in, and Represent the first available mode and the second available mode respectively, express Features, express Features, Indicates Come to Perform fusion alignment processing, Indicates Come to Perform fusion alignment processing, represents the cross attention mechanism, Indicates normalization processing; After all the semantic representations are fused by category, a fused representation sequence corresponding to each modal category is obtained, and all the fused representation sequences are spliced ​​to obtain a fused representation, and an emotion classification result is generated based on the fused representation.

4. The emotion recognition method based on the federated multimodal global diffusion model according to claim 3 is characterized in that: The step of determining emotion information of all available modalities in the target conversation and generating a plurality of semantic representations based on all the emotion information further includes: The splicing features are input into the self-attention layer, the semantic classification results are output, and based on the semantic classification results, the conversation graph loss function of the conversation graph network is constructed: ； in, represents the conversation graph loss function, represents the number of classifications after classification in the self-attention layer, Indicates the number of speeches. The index indicating the number of speeches. Indicates the True labels, Represents the semantic classification result, Represents the dimension of the conversation graph network; According to the emotion classification results, the semantic condition loss function of the semantic condition network is constructed: ； in, represents the semantic conditional loss function, Indicates the emotion classification results; According to the conversation graph loss function and the semantic condition loss function, the total network loss function is constructed: ； in, represents the total network loss function.

5. The emotion recognition method based on the federated multimodal global diffusion model according to claim 1, characterized in that: The splicing feature and the fusion representation are fused and then projected into a conditional embedding space to obtain a fusion condition, the fusion condition is input into a noise prediction network for fusion processing, and a conditional predicted noise is output, and a noise loss function is constructed based on the conditional predicted noise, specifically including: The spliced ​​features and the fused representation are input into the DISC diffusion network. After the spliced ​​features and the fused representation are spliced ​​through the embedding layer of the DISC diffusion network, they are mapped to the conditional embedding space to obtain the fusion condition: ； in, represents the fusion condition, Represents the splicing feature, represents the fusion representation, represents linear splicing processing, Indicates structural remodeling processing, represents the norm symbol; The fusion condition is input into the noise prediction network, and the noise prediction network fuses the channel features in multiple channels with the fusion condition respectively to map the fusion condition to the upsampling feature or the channel corresponding to the downsampling stage: ； in, Indicates the Channel features, Indicates the Channel features, Indicates fusion processing, represents the cross attention mechanism, Indicates linear splicing processing; In the plurality of channels, removing the fusion condition with a fixed probability, and outputting a plurality of conditional noise predictions and a plurality of unconditional noise predictions; All the conditional noise predictions and all the unconditional noise predictions are weightedly fused to obtain the conditional prediction noise: ； in, The index representing the number of diffusion steps, Indicates the The conditional prediction noise of the step diffusion, represents the enhanced weight, represents the noise prediction output, represents the conditional noise prediction, represents the unconditional noise prediction, represents the prediction noise, represents the network parameters of the noise prediction network; All conditional fusion features are iteratively forward diffused and iteratively reverse diffused, and the conditional prediction noise is used to construct the noise loss function: ； in, represents the noise loss function, Indicates that the random draw is no greater than A positive integer, represents the number of diffusion steps, represents a discrete uniform distribution, represents the initial noise, represents the random variable of the initial noise, express The probability distribution function of represents the conditional prediction noise, represents the noise distribution of random draws according to the standard multivariate normal distribution, represents a normal distribution, represents the identity matrix, Express 、 and The expectation of the joint distribution of It means calculating the square of the Euclidean distance between the predicted noise and the true noise.

6. The emotion recognition method based on the federated multimodal global diffusion model according to claim 1, characterized in that: The method further comprises: training multiple clients according to the noise loss function to obtain multiple restoration model parameters; training a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model; determining multiple missing modes according to all the available modes; inputting all the missing modes into the multimodal global diffusion model; and outputting multiple restoration features generated when all the missing modes are restored. Inputting the noise loss function into multiple clients to control all the clients to optimize their respective local recovery models according to the noise loss function to obtain corresponding recovery model parameters, and receiving the recovery model parameters sent by all the clients; Performing weighted aggregation on all the restoration model parameters to obtain global parameters, and optimizing the constructed multimodal global diffusion model according to the global parameters; The missing modality corresponding to each of the clients is determined according to all the available modalities, and all the missing modalities are input into the multimodal global diffusion model to output corresponding restoration features.

7. The emotion recognition method based on the federated multimodal global diffusion model according to claim 6, characterized in that: The method of fusing all the recovered features and all the available modalities to obtain a complete feature, optimizing a classifier using the complete feature, inputting the target conversation into the optimized classifier for recognition, and outputting an emotion recognition result specifically includes: Closing the multimodal global diffusion model, fusing all the restored features corresponding to each client and all the available modalities to obtain a complete feature of each client; Utilizing the complete features of each client, optimizing the sub-classifier corresponding to the client, and obtaining classifier parameters corresponding to each client; After weighted fusion of all the classifier parameters, the classifier is optimized, the target conversation is input into the optimized classifier for prediction and output of the emotion recognition result.

8. An emotion recognition system based on a federated multimodal global diffusion model, characterized in that: The emotion recognition system based on the federated multimodal global diffusion model includes: A conversation analysis module is used to obtain a target conversation, input the target conversation into a conversation diffusion model, output a speaker graph and a temporal dependency graph, and use a graph convolutional network to perform node fusion on the speaker graph and the temporal dependency graph to obtain a splicing feature; A semantic analysis module is configured to determine the emotional information of all available modalities in the target conversation, generate multiple semantic representations based on all the emotional information, and concatenate the multiple semantic representations with the multiple available modalities to obtain a fused representation; An information fusion module is used to fuse the splicing features and the fused representation and project them into a conditional embedding space to obtain a fusion condition, input the fusion condition into a noise prediction network for fusion processing, output conditional predicted noise, and construct a noise loss function based on the conditional predicted noise; a model training module, configured to train multiple clients according to the noise loss function to obtain multiple restoration model parameters, train a restoration model according to all the restoration model parameters to obtain a multimodal global diffusion model, determine multiple missing modes according to all the available modes, input all the missing modes into the multimodal global diffusion model, and output multiple restoration features generated when all the missing modes are restored; The result prediction module is used to fuse all the recovered features and all the available modalities to obtain complete features, optimize the classifier using the complete features, input the target dialogue into the optimized classifier for recognition, and output the emotion recognition result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an emotion recognition program based on a federated multimodal global diffusion model stored in the memory and executable on the processor. When the emotion recognition program based on a federated multimodal global diffusion model is executed by the processor, the steps of the emotion recognition method based on a federated multimodal global diffusion model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an emotion recognition program based on a federated multimodal global diffusion model. When the emotion recognition program based on a federated multimodal global diffusion model is executed by a processor, the steps of the emotion recognition method based on a federated multimodal global diffusion model as described in any one of claims 1 to 7 are implemented.

Images