Incomplete multi-mode emotion recognition method based on diffusion model and federal model

Abstract

The embodiment of the invention provides an incomplete multi-mode emotion recognition method based on a diffusion model and a federal model, and belongs to the technical field of data recognition. The method specifically comprises the steps that a client downloads a global model to the local; the client extracts incomplete multi-modal features through a shallow feature extractor, the missing modal features are input as random noise, the extracted original features are input into a global model for reverse diffusion denoising, and finally the missing modal features are complemented; sending the complemented full-modal features into an emotion classifier to perform an emotion classification task; the central node receives the model parameters uploaded by the clients, evaluates the contribution degree of each client, calculates the reputation of the clients, preferentially determines the final weight of each client according to the reputation of the clients, and carries out aggregation according to the final weight; and the central node updates the global model according to the local model set and the aggregated weight. Through the scheme disclosed by the invention, the recognition accuracy and safety are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of data recognition technology, and in particular to an incomplete multimodal emotion recognition method based on a diffusion model and a federated model. Background Art

[0002] Multimodal emotion recognition refers to the use of multiple perception methods and information sources (such as voice, facial expressions, body language, text, ECG waveforms, etc.) to identify and understand an individual's emotional state. Traditionally, emotion recognition mainly relies on a single source of information, such as voice tone analysis or facial expression recognition. Multimodal emotion recognition attempts to combine multiple sources of information to improve the accuracy and comprehensiveness of emotion recognition. Although multimodality is superior to single modality, in real-world scenarios, due to data collection, privacy issues, and other reasons, there is usually a problem of modality missing. Modality missing refers to the incomplete or missing information of certain perceptual modalities during the emotion recognition process, which will affect the accuracy of the multimodal emotion recognition system in inferring the individual's emotional state.

[0003] Existing solutions still have defects. For example, many methods for restoring missing modalities ignore the correlation between existing modalities, resulting in incoordination between modalities after restoration, affecting the accuracy and completeness of the overall recovery. In addition, the emotion recognition task itself involves private data such as face and voice, and most existing solutions do not consider the privacy issues involved. In addition, the quality of data provided by each emotion recognition client varies, and existing solutions do not screen them.

[0004] It can be seen that there is an urgent need for an incomplete multimodal emotion recognition method based on diffusion model and federated model that can improve recognition accuracy and security. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide an incomplete multimodal emotion recognition based on a diffusion model and a federated model, which at least partially solves the problems of poor recognition accuracy and security in the prior art.

[0006] The present disclosure provides an incomplete multimodal emotion recognition method based on a diffusion model and a federated model, including:

[0007] Step 1: The central node initializes the global model, and the client downloads the global model to the local computer.

[0008] Step 2: The client extracts incomplete multimodal features through a shallow feature extractor, inputs the missing modal features as random noise, inputs the extracted original features into the global model for reverse diffusion denoising, and finally completes the missing modal features;

[0009] Step 3: Send the completed full-modal features to the emotion classifier to perform emotion classification tasks and obtain emotion recognition results;

[0010] Step 4: The central node receives the model parameters uploaded by the client, evaluates the contribution of each client, calculates the client reputation f, and determines the final weight θ of each client based on the client reputation f, and aggregates according to the final weight θ;

[0011] Step 5: The central node updates the global model based on the local model set and the aggregated weights, after which the client downloads the new global model.

[0012] According to a specific implementation of the embodiment of the present disclosure, after step 1, the method further includes:

[0013] Each client trains the initialized global model based on local data.

[0014] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0015] Step 2.1, input the incomplete multimodal data into the shallow feature extractor, extract the text features, image features and audio features in the incomplete multimodal data through three independent encoders, input the missing modal features as random noise, and then map all features to the same distribution space through a layer of convolution;

[0016] Step 2.2, input the mapped text features, image features and audio features into the diffusion network of the global model for reverse diffusion, and construct the missing mode recovery differential equation, wherein the expression of the missing mode recovery differential equation is:

[0017] dx miss =-σ 2 tf miss (x miss ,x present ,t;θ miss )dt+σtdW;

[0018] Among them, x miss represents the missing modal features, x present represents the available modal feature set, f miss (x miss ,x present ,t;θ miss ) function simulates x miss 、x present The effect of time t on the diffusion network, parameter θ miss , σ represents the standard deviation of the variation with time t, describing the amplitude of random fluctuations, and dW represents the small increment of Brownian motion in the opposite direction;

[0019] Step 2.3, iterate the missing modal recovery differential equation to complete the missing modal features, wherein the expression for completing the missing modal features is:

[0020]

[0021] where Δt represents a small increment of time; η t represents the standard Brownian motion increment, η t ~N(0,1);

[0022] Step 2.4, calculating the reconstruction loss and diffusion loss according to the missing modal features and the completed missing modal features;

[0023] In step 2.5, the reconstruction loss and diffusion loss are combined to obtain the total loss and the parameters of the global model are updated according to the total loss.

[0024] According to a specific implementation of the embodiment of the present disclosure, the expression of the diffusion loss is:

[0025]

[0026] in, represents the expected value of missing data, z~N(0,1) represents the noise term that conforms to the standard normal distribution, t~U(0,T) represents the time variable uniformly distributed in the interval (0,T), and f miss The function is used to simulate the relationship between missing features and available features and capture the changing trend of missing data;

[0027] The expression of the reconstruction loss is

[0028]

[0029] Among them, x k is the value in the original data, is the value obtained by reconstruction, |Z miss | indicates missing modal size.

[0030] According to a specific implementation of an embodiment of the present disclosure, the emotion classifier includes a text modality sub-classifier, an image modality sub-classifier, and an audio modality sub-classifier, and step 3 specifically includes:

[0031] Step 3.1, input different modal features into the corresponding text modal sub-classifier, image modal sub-classifier and audio modal sub-classifier respectively to obtain the classification results of each modal feature;

[0032] Step 3.2, at the decision layer, the classification results of each modal classifier are summarized to form a voting set, and the final emotion recognition result is obtained by combining the voting mechanism, where the expression of the voting mechanism is:

[0033]

[0034] Among them, C final is the final emotion recognition result, arg max c It means to select the category c that maximizes the following expression. is an indicator function, if If yes, it is 1, otherwise it is 0. is the weight of the jth classifier of the ith modality, i = 1, 2, ..., M, j = 1, 2, ..., K i .

[0035] According to a specific implementation of the embodiment of the present disclosure, the model parameters include the total loss and the classification results of the three sub-classifiers, and the step 4 specifically includes:

[0036] Step 4.1, assume that the total loss of client k is The classification results of the three sub-classifiers of client k processing the mth incomplete multimodal data in different modalities are The final emotion classification result of client k is Calculate the average classification accuracy

[0037]

[0038] Among them, α l , α e , α a are the weights of the three modes, which can be set according to actual conditions;

[0039] Step 4.2, calculate the reputation of client k

[0040] Step 4.3, calculate the final weight by combining the average classification accuracy and reputation of client k

[0041]

[0042] Among them, α, β, γ are hyperparameters, and ∈ is a constant.

[0043] The incomplete multimodal emotion recognition scheme based on the diffusion model and the federated model in the disclosed embodiment includes: step 1, the central node initializes the global model, and the client downloads the global model locally; step 2, the client extracts incomplete multimodal features through a shallow feature extractor, inputs the missing modal features as random noise, inputs the extracted original features into the global model for reverse diffusion denoising, and finally completes the missing modal features; step 3, the completed full modal features are sent to the emotion classifier to perform the emotion classification task, and the emotion recognition result is obtained; step 4, the central node receives the model parameters uploaded by the client, evaluates the contribution of each client, calculates the client reputation f, and determines the final weight θ of each client based on the client reputation f, and aggregates according to the final weight θ; step 5, the central node updates the global model according to the local model set and the aggregated weight, and then the client downloads the new global model.

[0044] The beneficial effects of the disclosed embodiments are as follows: through the disclosed scheme, a diffusion network is designed to process multimodal data with missing modalities. This method effectively utilizes the features of existing modalities to restore the features of missing modalities, and finally classifies emotions through restored speech, picture, and audio features, effectively improving the accuracy of emotion classification in missing modal scenarios; adopts a federated learning approach, implements the whole process of downloading the model to the local client, and the client runs the model locally, and finally only uploads part of the parameters to the central node to update the global model, so that the privacy data such as the face in the client does not go out of the local, and the global data is available but not visible; adopts federated learning weight aggregation to evaluate the contribution of client tasks, and solves the impact of low-quality or malicious clients on the global model. In the process of uploading some parameters from the client to the central node, a weight aggregation mechanism is designed, and the final weight is obtained by weight aggregation based on the reputation of each client and the classification accuracy and loss of the client model, thereby improving recognition accuracy and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of an incomplete multimodal emotion recognition method based on a diffusion model and a federated model provided in an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram of a framework of an incomplete multimodal emotion recognition method based on a diffusion model and a federated model provided in an embodiment of the present disclosure;

[0048] Figure 3 A workflow diagram of an emotion sub-classifier provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0050] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0051] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0052] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0053] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0054] Multimodal emotion recognition refers to the use of multiple perception methods and information sources (such as voice, facial expressions, body language, text, ECG waveforms, etc.) to identify and understand an individual's emotional state. Traditionally, emotion recognition mainly relies on a single source of information, such as voice tone analysis or facial expression recognition. Multimodal emotion recognition attempts to combine multiple sources of information to improve the accuracy and comprehensiveness of emotion recognition. Although multimodality is superior to single modality, in real-world scenarios, due to data collection, privacy issues, and other reasons, there is usually a problem of modality missing. Modality missing refers to the incomplete or missing information of certain perceptual modalities during the emotion recognition process, which affects the accuracy of the multimodal emotion recognition system in inferring the individual's emotional state. Therefore, solving this modality missing problem is conducive to improving the robustness of multimodal emotion recognition.

[0055] Research shows that there are two main solutions to the above-mentioned modality missing problem: one is the data augmentation method, which randomly deletes the information of a certain modality during the training process to simulate the situation of modality missing in reality, thereby improving the model's emotion recognition performance under the condition of missing modality; the other is the data recovery method, which uses some relevant information to restore the missing modality data. Using available modalities to restore missing modalities is a common and effective method. The core idea of ​​this method is to use existing information (available modalities) to infer or generate missing information (missing modalities), and finally integrate the restored modal data with the original modal data into a multimodal emotion recognition system. At present, the existing solutions include zero-value-based recovery, average-based recovery, and deep learning-based methods. Among these methods, zero-value-based and average-based methods are only suitable for handling some simple emotion data filling problems because they do not use any supervised information. Deep learning-based methods can better estimate the missing modality by leveraging its powerful feature representation capabilities. For example, Tran et al. used cascaded residual autoencoders to restore missing modalities. Zhao et al. solved the missing modality problem by creating a missing modality imagination network. Lian et al. used graph neural networks to reconstruct missing modalities to improve accuracy.

[0056] Although these solutions have made some progress in solving the problem of missing modalities in multimodal emotion recognition, there are still some limitations. For example, many methods for restoring missing modalities ignore the correlation between existing modalities, resulting in inconsistency between modalities after restoration, affecting the accuracy and completeness of overall restoration; and the emotion recognition task itself involves private data such as face and voice, and most existing solutions do not consider the privacy issues involved; and the quality of data provided by each emotion recognition client varies, and existing solutions do not screen them.

[0057] The disclosed embodiment provides an incomplete multimodal emotion recognition method based on a diffusion model and a federated model, and the method can be applied to the emotion recognition process in Internet scenarios.

[0058] See also Figure 1 , is a flow chart of an incomplete multimodal emotion recognition method based on a diffusion model and a federated model provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:

[0059] Step 1: The central node initializes the global model, and the client downloads the global model to the local computer.

[0060] Furthermore, after step 1, the method further comprises:

[0061] Each client trains the initialized global model based on local data.

[0062] In the specific implementation, this implementation case provides 1 central node and K clients. The central node has an initialized global model ω t＝0 , each client has a local private incomplete multimodal dataset D k , D k contains i incomplete multimodal data {x 1 ,x 2 ,…,x i}, assuming that each data contains at most 3 modalities (speech l, ECG waveform e, audio a).

[0063] The client downloads the latest global model from the central node t Finally, the model is trained locally on the client to ensure that the data does not leave the local computer.

[0064] Step 2: The client extracts incomplete multimodal features through a shallow feature extractor, inputs the missing modal features as random noise, inputs the extracted original features into the global model for reverse diffusion denoising, and finally completes the missing modal features;

[0065] Based on the above embodiment, step 2 specifically includes:

[0066] Step 2.1, input the incomplete multimodal data into the shallow feature extractor, extract the text features, image features and audio features in the incomplete multimodal data through three independent encoders, input the missing modal features as random noise, and then map all features to the same distribution space through a layer of convolution;

[0067] Step 2.2, input the mapped text features, image features and audio features into the diffusion network of the global model for reverse diffusion, and construct the missing mode recovery differential equation, wherein the expression of the missing mode recovery differential equation is:

[0068] dx miss =-σ 2 tf miss (x miss ,x present ,t;θ miss )dt+σtdW;

[0069] Among them, x miss represents the missing modal features, x present represents the available modal feature set, f miss (x miss ,x present ,t;θ miss ) function simulates x miss 、x present The effect of time t on the diffusion network, parameter θ miss , σ represents the standard deviation of the variation with time t, describing the amplitude of random fluctuations, and dW represents the small increment of Brownian motion in the opposite direction;

[0070] Step 2.3, iterate the missing modal recovery differential equation to complete the missing modal features, wherein the expression for completing the missing modal features is:

[0071]

[0072] where Δt represents a small increment of time; η t represents the standard Brownian motion increment, η t ~N(0,1);

[0073] Step 2.4, calculating the reconstruction loss and diffusion loss according to the missing modal features and the completed missing modal features;

[0074] In step 2.5, the reconstruction loss and diffusion loss are combined to obtain the total loss and the parameters of the global model are updated according to the total loss.

[0075] Furthermore, the expression of the diffusion loss is:

[0076]

[0077] in, represents the expected value of missing data, z~N(0,1) represents the noise term that conforms to the standard normal distribution, t~U(0,T) represents the time variable uniformly distributed in the interval (0,T), and f missThe function is used to simulate the relationship between missing features and available features and capture the changing trend of missing data;

[0078] The expression of the reconstruction loss is

[0079]

[0080] Among them, x k is the value in the original data, is the value obtained by reconstruction, |Z miss | indicates missing modal size.

[0081] In a specific implementation, the described method of restoring the missing mode using the global model specifically includes the following steps:

[0082] 1) Incomplete multimodal data (may be missing one or two modal data), features are extracted through a shallow feature extractor, which contains three independent encoders E l 、E e 、E a They are used to extract the features of text, ECG and audio in multimodal data respectively. Then these features are mapped to the same distribution space h through a layer of convolution. mod =Conv1D(x mod ).

[0083] 2) Input the processed data in the previous step into the diffusion network. The process of recovering the missing mode is a reverse time diffusion process, that is, the missing mode features are gradually recovered from a random distribution based on the known mode. Through the stochastic differential equation, the diffusion process is formally established (t starts from time 0) as follows:

[0084] dx=a(x,t)dt+b(t)dw

[0085] a(x, t) represents the drift coefficient of the differential equation; b(t) represents the standard deviation of the fluctuation term; dw represents the small increment of Brownian motion. Since the reverse diffusion process is also a diffusion process, the differential equation of the reverse diffusion process can be expressed as:

[0086]

[0087] represents a small increment of Brownian motion in the opposite direction; Indicates that at time t, the probability density function p t (x) is the gradient of x. The differential equation of the missing modal feature recovery process in the emotion recognition scenario of the embodiment of the present disclosure is expressed as:

[0088]

[0089] x miss Indicates a missing modal feature; x present represents the available modal feature group; f miss (x miss ,x present ,t;θ miss ) function simulates x miss 、x present The effect of time t on the diffusion network, parameter θ miss ; σ represents the standard deviation that changes over time t, describing the amplitude of random fluctuations.

[0090] 3) According to the above differential equation, the emotion mode recovery rule can be iterated:

[0091]

[0092] Δt represents a small increment of time; η t represents the standard Brownian motion increment, η t ~N(0,1).

[0093] The loss function of this diffusion network contains two parts, one of which is the diffusion loss, which can be expressed as:

[0094]

[0095] in, represents the expected value of missing data, z~N(0,1) represents the noise term that conforms to the standard normal distribution, t~U(0,T) represents the time variable uniformly distributed in the interval (0,T), and f miss The function is used to simulate the relationship between missing features and available features and capture the changing trend of missing data.

[0096] The other part is the reconstruction loss, which can be expressed as:

[0097]

[0098] The total loss of the mode recovery diffusion network in the embodiment of the present disclosure is expressed as:

[0099]

[0100] Step 3: Send the completed full-modal features to the emotion classifier to perform emotion classification tasks and obtain emotion recognition results;

[0101] Based on the above embodiment, the emotion classifier includes a text modality sub-classifier, an image modality sub-classifier and an audio modality sub-classifier, and step 3 specifically includes:

[0102] Step 3.1, input different modal features into the corresponding text modal sub-classifier, image modal sub-classifier and audio modal sub-classifier respectively to obtain the classification results of each modal feature;

[0103] Step 3.2, at the decision layer, the classification results of each modal classifier are summarized to form a voting set, and the final emotion recognition result is obtained by combining the voting mechanism, where the expression of the voting mechanism is:

[0104]

[0105] Among them, C final is the final emotion recognition result, arg max c It means to select the category c that maximizes the following expression. is an indicator function, if If yes, it is 1, otherwise it is 0. is the weight of the jth classifier of the ith modality, i = 1, 2, ..., M, j = 1, 2, ..., K i .

[0106] In specific implementation, the emotion classifier contains three sub-classifiers with high accuracy that have been trained. Each sub-classifier makes decisions based on different modal features (speech, ECG waveform, audio), and finally the classification results are fused at the decision layer. The process is as follows: Figure 3 As shown. The specific method of fusion at the decision layer is to use a "voting" method. Specifically, the classification results of each modality classifier are summarized to form a voting set. Assume that there are M modalities, such as speech, electrocardiogram, and audio in the embodiment of the present disclosure, and each modality i (where i = 1, 2, ..., M) has K i possible emotion categories. represents the classification result of the jth classifier of mode i, where j = 1, 2, ..., K i The voting mechanism formula can be expressed as:

[0107]

[0108] C final is the final emotion classification result; arg max c Indicates the selection of category c that maximizes the following expression; is an indicator function, if If yes, it is 1, otherwise it is 0; is the weight of the j-th classifier of the ith mode, which can be determined based on the performance of the classifier or other factors.

[0109] Step 4: The central node receives the model parameters uploaded by the client, evaluates the contribution of each client, calculates the client reputation f, and determines the final weight θ of each client based on the client reputation f, and aggregates according to the final weight θ;

[0110] Based on the above embodiment, the model parameters include the total loss and the classification results of the three sub-classifiers, and the step 4 specifically includes:

[0111] Step 4.1, assume that the total loss of client k is The classification results of the three sub-classifiers of client k processing the mth incomplete multimodal data in different modalities are The final emotion classification result of client k is Calculate the average classification accuracy

[0112]

[0113] Among them, α l , α e , α a are the weights of the three modes, which can be set according to actual conditions;

[0114] Step 4.2, calculate the reputation of client k

[0115] Step 4.3, calculate the final weight by combining the average classification accuracy and reputation of client k

[0116]

[0117] Among them, α, β, γ are hyperparameters, and ∈ is a constant.

[0118] In specific implementation, the parameters uploaded by the client model include the total loss of the diffusion network and the individual classification results of the three sub-classifiers. If a client can always correctly classify most of the incomplete multimodal data, it can be considered that the credibility of this client is high. Prevent the parameters uploaded by clients with low credibility from misleading the update of the global model. Assume that the diffusion model loss of client k is The classification results of the three sub-classifiers of client k processing the mth incomplete multimodal data in different modalities are The final emotion classification result obtained by client k according to step 3 is The average classification accuracy can be expressed as {α l , α e , α a} are the weights of the three modes, which can be set according to the actual situation; reputation can be expressed as Directly reflects the performance of the model during training. Such a formula ensures that even if the loss is small, the reputation is still a positive value, while giving higher penalties to clients with large losses. The final weight θ k Can be written as {α, β, γ} are hyperparameters and ∈ is a small constant used to avoid division by zero errors.

[0119] Step 5: The central node updates the global model based on the local model set and the aggregated weights, after which the client downloads the new global model.

[0120] In specific implementation, considering the differences in data processing speed and data volume of different clients, the following strategy can be adopted in the model update process: the central node coordinates the clients to ensure that they each process the same amount of incomplete multimodal emotion data as one iteration. Once the client completes its data processing round, the average classification accuracy of that round and the credibility of the client are recorded. After all clients have completed, the central node summarizes the information to perform the model update operation. Doing so can ensure the efficiency and fairness of the model, is conducive to the optimal allocation of resources and the incentive for the contribution of high-quality data, and overall improves the quality of this federated learning.

[0121] The incomplete multimodal emotion recognition method based on the diffusion model and the federated model provided in this embodiment processes the multimodal data of the missing modality by designing a diffusion network. The method effectively uses the features of the existing modality to restore the features of the missing modality, and finally classifies the emotions through the restored speech, picture, and audio features, effectively improving the accuracy of emotion classification in the missing modality scenario; adopts the method of federated learning, implements the whole process of downloading the model to the local client, and the client runs the model locally, and finally uploads only part of the parameters to the central node to update the global model, so that the privacy data such as the face in the client does not go out of the local, and the global data is available but not visible; adopts federated learning weight aggregation to evaluate the contribution of the client task, and solves the impact of low-quality or malicious clients on the global model. In the process of uploading some parameters from the client to the central node, a weight aggregation mechanism is designed, and the weight aggregation is performed according to the reputation of each client and the classification accuracy and loss of the client model to obtain the final weight, thereby improving the recognition accuracy and security.

[0122] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0123] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An incomplete multimodal emotion recognition method based on diffusion model and federated model, characterized in that: include: Step 1: The central node initializes the global model, and the client downloads the global model to the local computer. Step 2: The client extracts incomplete multimodal features through a shallow feature extractor, inputs the missing modal features as random noise, inputs the extracted original features into the global model for reverse diffusion denoising, and finally completes the missing modal features; Step 3: Send the completed full-modal features to the emotion classifier to perform emotion classification tasks and obtain emotion recognition results; Step 4: The central node receives the model parameters uploaded by the client, evaluates the contribution of each client, calculates the client reputation f, and determines the final weight θ of each client based on the client reputation f, and aggregates according to the final weight θ; Step 5: The central node updates the global model based on the local model set and the aggregated weights, after which the client downloads the new global model.

2. The method according to claim 1, characterized in that After step 1, the method further comprises: Each client trains the initialized global model based on local data.

3. The method according to claim 1, characterized in that , the step 2 specifically includes: Step 2.1, input the incomplete multimodal data into the shallow feature extractor, extract the text features, image features and audio features in the incomplete multimodal data through three independent encoders, input the missing modal features as random noise, and then map all features to the same distribution space through a layer of convolution; Step 2.2, input the mapped text features, image features and audio features into the diffusion network of the global model for reverse diffusion, and construct the missing mode recovery differential equation, wherein the expression of the missing mode recovery differential equation is: dx miss =-s 2 tf miss (x miss ,x present ,t;θ miss )dt+σtdW; Among them, x miss represents the missing modal features, x present represents the available modal feature set, f miss (x miss ,x present ,t;θ miss ) function simulates x miss 、x present The effect of time t on the diffusion network, parameter θ miss , σ represents the standard deviation of the variation with time t, describing the amplitude of random fluctuations, and dW represents the small increment of Brownian motion in the opposite direction; Step 2.3, iterate the missing modal recovery differential equation to complete the missing modal features, wherein the expression for completing the missing modal features is: where Δt represents a small increment of time; η t represents the standard Brownian motion increment, η t ~N(0,1); Step 2.4, calculating the reconstruction loss and diffusion loss according to the missing modal features and the completed missing modal features; In step 2.5, the reconstruction loss and diffusion loss are combined to obtain the total loss and the parameters of the global model are updated according to the total loss.

4. The method according to claim 3, characterized in that , the expression of diffusion loss is in, represents the expected value of missing data, z~N(0,1) represents the noise term that conforms to the standard normal distribution, t~U(0,T) represents the time variable uniformly distributed in the interval (0,T), and f miss The function is used to simulate the relationship between missing features and available features and capture the changing trend of missing data; The expression of the reconstruction loss is Among them, x k is the value in the original data, is the value obtained by reconstruction, |Z miss | indicates missing modal size.

5. The method according to claim 4, characterized in that , the emotion classifier includes a text modality sub-classifier, an image modality sub-classifier and an audio modality sub-classifier, and the step 3 specifically includes: Step 3.1, input different modal features into the corresponding text modal sub-classifier, image modal sub-classifier and audio modal sub-classifier respectively to obtain the classification results of each modal feature; Step 3.2, at the decision layer, the classification results of each modal classifier are summarized to form a voting set, and the final emotion recognition result is obtained by combining the voting mechanism, where the expression of the voting mechanism is: Among them, C final is the final emotion recognition result, arg max c It means to select the category c that maximizes the following expression. is an indicator function, if If yes, it is 1, otherwise it is 0. is the weight of the jth classifier of the ith modality, i = 1, 2, ..., M, j = 1, 2, ..., K i .

6. The method according to claim 5, characterized in that , the model parameters include the total loss and the classification results of the three sub-classifiers, and the step 4 specifically includes: Step 4.1, assume that the total loss of client k is The classification results of the three sub-classifiers of client k processing the mth incomplete multimodal data in different modalities are The final emotion classification result of client k is Calculate the average classification accuracy Among them, α l , α e , α a are the weights of the three modes, which can be set according to actual conditions; Step 4.2, calculate the reputation of client k Step 4.3, calculate the final weight by combining the average classification accuracy and reputation of client k Among them, α, β, γ are hyperparameters, and ∈ is a constant.

Images