A multi-modal sentiment recognition method and system based on mutual information

Abstract

The application provides a multimodal emotion recognition method and system based on mutual information, which comprises the following steps: acquiring video data and audio data; obtaining an emotion classification result according to the video data and the audio data and a trained multimodal emotion recognition model; wherein in the multimodal emotion recognition model, a total loss function is defined, and the multimodal emotion recognition model is trained based on the obtained total loss; the total loss function is composed of a single-modal unique information learning loss function, an inter-modal common information learning loss function, a multimodal fusion feature minimum sufficient representation loss function and an emotion recognition classification loss function. The application uses mutual information to eliminate redundant information, extract relevant information and mine the complementarity between different modalities, thereby improving the multimodal emotion recognition precision.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, and relates to a multimodal emotion recognition method and system based on mutual information. Background Technology

[0002] Emotion recognition technology identifies and understands human emotions by analyzing nonverbal information such as facial expressions, voice tone, and body language, as well as through text analysis. With the continuous advancement of information technology, the amount of data on the internet is exploding, existing in multimodal forms including text, audio, and images. Multimodal emotion recognition technology, by fully leveraging the complementary advantages of multimodal data and effectively integrating this data, can significantly improve the accuracy of emotion recognition, thereby providing a more natural and intelligent human-computer interaction experience.

[0003] Research in multimodal emotion recognition focuses on extracting comprehensive and accurate multimodal feature representations from different modalities. Currently, most multimodal feature representation learning methods concentrate on learning common representations by embedding different modalities into the same feature space. However, different modalities not only share common information but also possess unique information specific to each modality. Learning only common features during multimodal feature representation learning weakens or removes the independent and unique information of each modality, resulting in significant information loss. Furthermore, the noise and redundant information in the high-dimensional multimodal representation vectors generated during multimodal fusion can lead to overfitting. Therefore, how to fully exploit the complementary and shared information between modalities and filter out noise information contained in the modalities to obtain a sufficiently concise multimodal feature representation has become a pressing technical problem.

[0004] While existing multimodal emotion recognition technologies can improve recognition accuracy, they overemphasize the expression of common features and neglect modality-specific information, leading to information loss and decreased accuracy. Furthermore, noise and redundant information in high-dimensional fusion vectors increase complexity, potentially causing overfitting and reducing recognition stability. Summary of the Invention

[0005] The technical problem to be solved by this invention is: how to use mutual information to remove redundant information, extract common information, and mine unique information of different modalities in order to improve the accuracy of multimodal emotion recognition.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0007] In a first aspect, the present invention provides a multimodal emotion recognition method based on mutual information, comprising:

[0008] Acquire video and audio data;

[0009] Based on the video and audio data and the pre-trained multimodal emotion recognition model, the emotion classification result is obtained;

[0010] In the multimodal emotion recognition model,

[0011] Define an overall loss function, and train a multimodal emotion recognition model based on the obtained overall loss;

[0012] The overall loss function consists of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function for multimodal fusion features, and an emotion recognition and classification loss function;

[0013] The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality.

[0014] The common information learning loss function between modalities is used to learn the common information between modalities;

[0015] The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation.

[0016] The emotion recognition classification loss function is used to obtain the final emotion classification result;

[0017] The overall loss function is expressed as:

[0018] ;

[0019] In the formula, This represents the loss function for learning information specific to a single audio modality. This represents the loss function for learning information specific to a single modality of the video. This represents the loss function for learning information specific to a single modality. The loss function represents the shared information learning between modalities. This represents the loss function that minimizes the fully sufficient representation of multimodal fusion features. This represents the loss function for emotion recognition and classification.

[0020] The audio single-modal unique information learning loss function is expressed as:

[0021] ;

[0022] In the formula, The minimum sufficient representation of audio modal features. The true label representing the audio modality. This represents the initial audio unimodal features. Represents the Lagrange multipliers. This represents the mutual information between two feature vectors;

[0023] The video single-modal unique information learning loss function is expressed as:

[0024] ;

[0025] In the formula, The minimum sufficient representation of video modal features. The true label representing the video modality, This represents the initial single-modal features of the video;

[0026] The loss function for learning common information among the modalities is expressed as:

[0027] ;

[0028] The minimum sufficient representation loss function of the multimodal fusion features is expressed as:

[0029] ;

[0030] In the formula, This represents the minimum sufficient representation of multimodal fusion features. This represents the initial multimodal fusion features. Indicates the true label of the sample;

[0031] The emotion recognition classification loss function is expressed as follows:

[0032] ;

[0033] In the formula, Represents the cross-entropy loss function Indicates the predicted value. Indicates the true label;

[0034] Initial video unimodal features Initial audio unimodal features The data are then fed into a multilayer perceptron to obtain single-modal features. ;

[0035] Based on the information bottleneck theory, a loss function for learning the unique information of the single modality is defined, and mutual information is used to learn the single modality features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features .

[0036] Furthermore, based on the single-modal-specific information learning loss function and the information bottleneck theory, mutual information is used to learn the learned single-modal features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features ,include:

[0037] Using the upper bound of mutual information to estimate CLUB minimization of the initial single-modal feature representation and single-modal features The mutual information between them is used to compress redundant information; the formula for calculating the upper bound of mutual information is expressed as:

[0038] ;

[0039] In the formula, Indicates the sample batch size. This represents the neural network parameters sampled from real samples. This represents the variational distribution of the positive sample. This represents the variational distribution of negative samples. Indicates the first The final single-modal features of each sample Indicates the first Initial single-modal features of each sample Indicates the first Initial single-modal features of each sample The logarithmic function representing the variational distribution;

[0040] Using mutual information lower bounds to estimate MINE maximizes single-modal features and modal true label value Mutual information between them is used to obtain a minimum sufficient representation of video modal features. Minimal sufficient representation of audio modal features The formula for calculating the lower bound of mutual information is as follows:

[0041] ;

[0042] In the formula, The network parameters sampled from real samples are: neural networks, Denotes the expected value of the joint distribution. This represents the expected value of the product of the marginal distributions. Indicates the supremacy. Represents the set of network parameters. Represents the natural base. Represents a logarithmic function.

[0043] Furthermore, the minimum sufficient representation of video modal features is... Minimal sufficient representation of audio modal features The features are then concatenated to obtain the initial multimodal fusion features. .

[0044] Furthermore, based on the information bottleneck theory, a minimum sufficient representation loss function for the multimodal fusion features is defined to constrain the minimum sufficient representation of the multimodal fusion features. Learning to obtain the minimum sufficient representation of multimodal fusion features .

[0045] Furthermore, based on the information bottleneck theory, a minimum sufficient representation loss function for the multimodal fusion features is defined to constrain the minimum sufficient representation of the multimodal fusion features. Learning to obtain the minimum sufficient representation of multimodal fusion features ,include:

[0046] Estimating CLUB minimization of initial multimodal fusion features using the upper bound of mutual information Minimal Sufficient Representation of Multimodal Fusion Features The mutual information between them is used to compress redundant information and filter out noise carried during the multimodal fusion process;

[0047] Using mutual information lower bound estimation, MINE maximizes the initial multimodal fusion features. and sample true labels The mutual information between them yields the minimum sufficient representation of the multimodal fusion features. .

[0048] Furthermore, based on the emotion recognition classification loss function, the multimodal fusion features are minimized and fully represented. The data is fed into a linear classifier to obtain the predicted value. This serves as the result of sentiment classification.

[0049] Secondly, the present invention provides a multimodal emotion recognition system based on mutual information, comprising the following modules:

[0050] The dataset acquisition module is used to acquire video and audio data;

[0051] The emotion result analysis module is used to obtain emotion classification results based on the video data, audio data, and the pre-trained multimodal emotion recognition model;

[0052] In the multimodal emotion recognition model,

[0053] Define an overall loss function, and train a multimodal emotion recognition model based on the obtained overall loss;

[0054] The overall loss function consists of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function for multimodal fusion features, and an emotion recognition and classification loss function;

[0055] The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality.

[0056] The common information learning loss function between modalities is used to learn the common information between modalities;

[0057] The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation.

[0058] The emotion recognition classification loss function is used to obtain the final emotion classification result;

[0059] The overall loss function is expressed as:

[0060] ;

[0061] In the formula, This represents the loss function for learning information specific to a single audio modality. This represents the loss function for learning information specific to a single modality of the video. This represents the loss function for learning information specific to a single modality. The loss function represents the shared information learning between modalities. This represents the loss function that minimizes the fully sufficient representation of multimodal fusion features. This represents the loss function for emotion recognition and classification.

[0062] The audio single-modal unique information learning loss function is expressed as:

[0063] ;

[0064] In the formula, The minimum sufficient representation of audio modal features. The true label representing the audio modality. This represents the initial audio unimodal features. Represents the Lagrange multipliers. This represents the mutual information between two feature vectors;

[0065] The video single-modal unique information learning loss function is expressed as:

[0066] ;

[0067] In the formula, The minimum sufficient representation of video modal features. The true label representing the video modality, This represents the initial single-modal features of the video;

[0068] The loss function for learning common information among the modalities is expressed as:

[0069] ;

[0070] The minimum sufficient representation loss function of the multimodal fusion features is expressed as:

[0071] ;

[0072] In the formula, This represents the minimum sufficient representation of multimodal fusion features. This represents the initial multimodal fusion features. Indicates the true label of the sample;

[0073] The emotion recognition classification loss function is expressed as follows:

[0074] ;

[0075] In the formula, Represents the cross-entropy loss function Indicates the predicted value. Indicates the true label;

[0076] Initial video unimodal features Initial audio unimodal features The data are then fed into a multilayer perceptron to obtain single-modal features. ;

[0077] Based on the information bottleneck theory, a loss function for learning the unique information of the single modality is defined, and mutual information is used to learn the single modality features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features .

[0078] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0079] This invention proposes a multimodal emotion recognition method based on mutual information. It can use mutual information to remove redundant information from single-modal features, extract task-related information, fully explore the unique information of single modalities, improve the complementarity between modalities in the single-modal representation learning process, and also use mutual information to extract common information between modalities, reduce the differences between modalities, maximize the consistency between modalities, and extract the minimum sufficient representation of multimodal fusion features using mutual information.

[0080] The present invention proposes a multimodal emotion recognition method based on mutual information, which can filter out noise information carried by the minimum sufficient representation of multimodal fusion features after passing through the fusion network while being compatible with common and unique information. This achieves the purpose of learning a concise multimodal representation and reducing the risk of overfitting. The multimodal emotion recognition method based on mutual information proposed in this invention can also be flexibly embedded in various multimodal fusion models and has strong applicability. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the model structure of a multimodal emotion recognition method based on mutual information provided in an embodiment of the present invention. Detailed Implementation

[0082] Example 1

[0083] like Figure 1 As shown in the figure, this embodiment introduces a multimodal emotion recognition method based on mutual information, including:

[0084] (1) Obtain the dataset:

[0085] Obtain the original dataset for multimodal emotion recognition, and divide the dataset into training and test sets proportionally:

[0086] The obtained raw dataset for multimodal emotion recognition is represented as follows: , Indicates serial number This represents the i-th training sample. Indicates the number of training samples. This represents the true label of the i-th training sample. , This indicates the number of emotion categories.

[0087] Each training sample It consists of two modalities: video and audio data, represented as video data. and audio data .

[0088] (2) Preprocess the training set in the dataset:

[0089] For the video data in the training set respectively and audio data Preprocessing:

[0090] From each video data Three frames of images were sampled at uniform intervals.

[0091] Using Fast Fourier Transform to convert audio data Convert it into a two-dimensional spectrum.

[0092] (3) Extract the initial single-modal feature representations from the preprocessed training set. Obtain initial video single-modal features and initial audio unimodal features ;

[0093] Preprocessed video data and audio data Initial single-modal feature representations are extracted using ResNet and LSTM networks. The initial video single-modal features were obtained. and initial audio unimodal features ,in, , , This represents the feature dimension of each single modality.

[0094] (4) Define the overall loss function:

[0095] The overall loss function is defined as consisting of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function of multimodal fusion features, and an emotion recognition and classification loss function;

[0096] The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality.

[0097] The common information learning loss function between modalities is used to learn the common information between modalities;

[0098] The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation.

[0099] The emotion recognition classification loss function is used to obtain the final emotion classification result.

[0100] The overall loss function is expressed as:

[0101] ;

[0102] In the formula, This represents the loss function for learning information specific to a single audio modality. This represents the loss function for learning information specific to a single modality of the video. This represents the loss function for learning information specific to a single modality. The loss function represents the shared information learning between modalities. This represents the loss function that minimizes the fully sufficient representation of multimodal fusion features. This represents the loss function for emotion recognition and classification.

[0103] The audio single-modal unique information learning loss function is expressed as:

[0104] ;

[0105] In the formula, The minimum sufficient representation of audio modal features. The true label representing the audio modality. This represents the initial audio unimodal features. Represents the Lagrange multipliers. This represents the mutual information between two feature vectors;

[0106] The video single-modal unique information learning loss function is expressed as:

[0107] ;

[0108] In the formula, The minimum sufficient representation of video modal features. The true label representing the video modality, This represents the initial single-modal features of the video;

[0109] The loss function for learning common information among the modalities is expressed as:

[0110] ;

[0111] The minimum sufficient representation loss function of the multimodal fusion features is expressed as:

[0112] ;

[0113] In the formula, This represents the minimum sufficient representation of multimodal fusion features. This represents the initial multimodal fusion features. Indicates the true label of the sample;

[0114] The emotion recognition classification loss function is expressed as follows:

[0115] ;

[0116] In the formula, Represents the cross-entropy loss function Indicates the predicted value. This indicates the actual label.

[0117] (4) Initial video single-modal features Initial audio unimodal features The data are then fed into a multilayer perceptron to obtain single-modal features. ;

[0118] (5) Based on the information bottleneck theory, define the single-modal unique information learning loss function, and use mutual information to learn the single-modal features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features .

[0119] Specifically, based on the information bottleneck theory, the minimum sufficient feature representation of the single-modal features is learned using mutual information, thereby obtaining the minimum sufficient representation of the video modal features. Minimal sufficient representation of audio modal features ;include:

[0120] Using the upper bound of mutual information to estimate CLUB minimization of the initial single-modal feature representation and single-modal features The mutual information between them is used to compress redundant information, where The formula for calculating the upper bound of mutual information is expressed as:

[0121] ;

[0122] In the formula, Indicates the sample batch size. This represents the neural network parameters sampled from real samples. This represents the variational distribution of the positive sample. This represents the variational distribution of negative samples. Indicates the first The final single-modal features of each sample Indicates the first Initial single-modal features of each sample Indicates the first Initial single-modal features of each sample The logarithmic function representing the variational distribution;

[0123] Using mutual information lower bounds to estimate MINE maximizes single-modal features and modal true label value Mutual information between them is used to obtain a minimum sufficient representation of video modal features. Minimal sufficient representation of audio modal features ;in The formula for calculating the lower bound of mutual information is expressed as:

[0124] ;

[0125] In the formula, The network parameters sampled from real samples are: neural networks, Denotes the expected value of the joint distribution. This represents the expected value of the product of the marginal distributions. Indicates the supremacy. Represents the set of network parameters. Represents the natural base. Represents a logarithmic function.

[0126] Based on the initial video single-modal features Initial audio unimodal features Learning single-modal features Each has its own unique information and filters out noise and redundancy in the audio and video modalities.

[0127] (5) Minimal sufficient representation based on video modal features Minimal sufficient representation of audio modal features The loss function is learned based on the shared information between modalities, and the mutual information is used to constrain the features of a single modality. Learn shared information between modalities;

[0128] (6) Minimal sufficient representation of video modal features Minimal sufficient representation of audio modal features The features are then concatenated to obtain the initial multimodal fusion features. ,in, , Indicates the feature dimension;

[0129] (7) Based on the information bottleneck theory, define the minimum sufficient representation loss function of the multimodal fusion feature to constrain the minimum sufficient representation of the multimodal fusion feature. Learning to obtain the minimum sufficient representation of multimodal fusion features :

[0130] The specific steps are as follows:

[0131] The initial multimodal fusion features are further extracted using a multilayer perceptron network to extract the minimum sufficient representation of the multimodal fusion features. , represented as:

[0132] ;

[0133] in, This represents a multilayer perceptron. d represents the parameters of a multilayer perceptron network. f This represents the feature dimension of the final multimodal fusion feature minimum sufficient representation. Indicates feature dimension as A real vector;

[0134] Estimating CLUB minimization of initial multimodal fusion features using the upper bound of mutual information Minimal Sufficient Representation of Multimodal Fusion Features The mutual information between them is used to compress redundant information and filter out noise carried during the multimodal fusion process;

[0135] Using mutual information lower bound estimation, MINE maximizes the initial multimodal fusion features. and sample true labels The mutual information between them yields the minimum sufficient representation of the multimodal fusion features. .

[0136] (8) Based on the emotion recognition classification loss function, minimize the full representation of the multimodal fusion features. The data is fed into a linear classifier to obtain the predicted value. This serves as the result of sentiment classification.

[0137] The predicted value , represented as:

[0138] ;

[0139] In the formula, Y pred This represents the predicted value obtained by a multimodal fusion classification network based on mutual information. Indicates a fully connected layer. Indicates the parameters of the fully connected layer. The feature dimension of the predicted value vector. Indicates feature dimension as A real vector.

[0140] (9) Combining the loss function for learning unique information of a single modality, the loss function for learning common information between modalities, the loss function for minimum sufficient representation of multimodal fusion features, and the loss function for emotion recognition classification, a multimodal emotion recognition model is trained in an end-to-end manner based on the overall loss.

[0141] (10) Input the test set into the trained multimodal emotion recognition model to obtain the emotion classification results.

[0142] Example 2

[0143] Based on the same concept as Embodiment 1, this embodiment provides a multimodal emotion recognition system based on mutual information, including the following modules:

[0144] The dataset acquisition module is used to acquire video and audio data;

[0145] The emotion result analysis module is used to obtain emotion classification results based on the video data, audio data, and the pre-trained multimodal emotion recognition model;

[0146] In the multimodal emotion recognition model,

[0147] Define an overall loss function, and train a multimodal emotion recognition model based on the obtained overall loss;

[0148] The overall loss function consists of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function for multimodal fusion features, and an emotion recognition and classification loss function;

[0149] The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality.

[0150] The common information learning loss function between modalities is used to learn the common information between modalities;

[0151] The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation.

[0152] The emotion recognition classification loss function is used to obtain the final emotion classification result.

[0153] In summary, the present invention can utilize mutual information to remove redundant information in single-modal data, extract task-related information, fully exploit the unique information of single modality, improve the complementarity between modalities during single-modal representation learning, and also utilize mutual information to extract common information between modalities, reduce differences between modalities, maximize consistency between modalities, and utilize mutual information to extract minimum sufficient representation of multimodal fusion features.

[0154] This invention can also filter out noise information carried by the minimum sufficient representation of multimodal fusion features after passing through the fusion network while being compatible with common and unique information, so as to learn a concise multimodal representation and reduce the risk of overfitting. The multimodal emotion recognition method based on mutual information proposed in this invention can also be flexibly embedded in various multimodal fusion models and has strong applicability.

[0155] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method based on mutual information, characterized in that, include: Acquire video and audio data; Based on the video and audio data and the pre-trained multimodal emotion recognition model, the emotion classification result is obtained; In the multimodal emotion recognition model, Define an overall loss function, and train a multimodal emotion recognition model based on the obtained overall loss; The overall loss function consists of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function for multimodal fusion features, and an emotion recognition and classification loss function; The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality. The common information learning loss function between modalities is used to learn the common information between modalities; The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation. The emotion recognition classification loss function is used to obtain the final emotion classification result; The overall loss function is expressed as: ； In the formula, This represents the loss function for learning information specific to a single audio modality. This represents the loss function for learning information specific to a single modality of the video. This represents the loss function for learning information specific to a single modality. The loss function represents the shared information learning between modalities. This represents the loss function that minimizes the fully sufficient representation of multimodal fusion features. This represents the loss function for emotion recognition and classification. The audio single-modal unique information learning loss function is expressed as: ； In the formula, The minimum sufficient representation of audio modal features. The true label representing the audio modality. This represents the initial audio unimodal features. Represents the Lagrange multipliers. This represents the mutual information between two feature vectors; The video single-modal unique information learning loss function is expressed as: ； In the formula, The minimum sufficient representation of video modal features. The true label representing the video modality, This represents the initial single-modal features of the video; The loss function for learning common information among the modalities is expressed as: ； The minimum sufficient representation loss function of the multimodal fusion features is expressed as: ； In the formula, This represents the minimum sufficient representation of multimodal fusion features. This represents the initial multimodal fusion features. Indicates the true label of the sample; The emotion recognition classification loss function is expressed as follows: ； In the formula, Represents the cross-entropy loss function Indicates the predicted value. Indicates the true label; Initial video unimodal features Initial audio unimodal features The data are then fed into a multilayer perceptron to obtain single-modal features. ; Based on the information bottleneck theory, a loss function for learning the unique information of the single modality is defined, and mutual information is used to learn the single modality features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features .

2. The multimodal emotion recognition method based on mutual information according to claim 1, characterized in that, Based on the loss function learned from the unique information of the single modality, and using mutual information to learn the single modality features based on the information bottleneck theory, the single modality features are learned. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features ,include: Using the upper bound of mutual information to estimate CLUB minimization of the initial single-modal feature representation and single-modal features The mutual information between them is used to compress redundant information; the formula for calculating the upper bound of mutual information is expressed as: ； In the formula, Indicates the sample batch size. This represents the neural network parameters sampled from real samples. This represents the variational distribution of the positive sample. This represents the variational distribution of negative samples. Indicates the first The final single-modal features of each sample Indicates the first Initial single-modal features of each sample Indicates the first Initial single-modal features of each sample The logarithmic function representing the variational distribution; Using mutual information lower bounds to estimate MINE maximizes single-modal features and modal true label value Mutual information between them is used to obtain a minimum sufficient representation of video modal features. Minimal sufficient representation of audio modal features The formula for calculating the lower bound of mutual information is as follows: ； In the formula, The network parameters sampled from real samples are: neural networks, Denotes the expected value of the joint distribution. This represents the expected value of the product of the marginal distributions. Indicates the supremacy. Represents the set of network parameters. Represents the natural base. Represents a logarithmic function.

3. The multimodal emotion recognition method based on mutual information according to claim 2, characterized in that, Minimal sufficient representation of video modal features Minimal sufficient representation of audio modal features The features are then concatenated to obtain the initial multimodal fusion features. .

4. The multimodal emotion recognition method based on mutual information according to claim 3, characterized in that, Based on the information bottleneck theory, a minimum sufficient representation loss function for the multimodal fusion features is defined to constrain the minimum sufficient representation of the multimodal fusion features. Learning to obtain the minimum sufficient representation of multimodal fusion features .

5. The multimodal emotion recognition method based on mutual information according to claim 4, characterized in that, Based on the information bottleneck theory, a minimum sufficient representation loss function for the multimodal fusion features is defined to constrain the minimum sufficient representation of the multimodal fusion features. Learning to obtain the minimum sufficient representation of multimodal fusion features ,include: Estimating CLUB minimization of initial multimodal fusion features using the upper bound of mutual information Minimal Sufficient Representation of Multimodal Fusion Features The mutual information between them is used to compress redundant information and filter out noise carried during the multimodal fusion process; Using mutual information lower bound estimation, MINE maximizes the initial multimodal fusion features. and sample true labels The mutual information between them yields the minimum sufficient representation of the multimodal fusion features. .

6. The multimodal emotion recognition method based on mutual information according to claim 5, characterized in that, Based on the emotion recognition classification loss function, the multimodal fusion features are minimized and fully represented. The data is fed into a linear classifier to obtain the predicted value. This serves as the result of sentiment classification.

7. A multimodal emotion recognition system based on mutual information, characterized in that, Includes the following modules: The dataset acquisition module is used to acquire video and audio data; The emotion result analysis module is used to obtain emotion classification results based on the video data, audio data, and the pre-trained multimodal emotion recognition model; In the multimodal emotion recognition model, Define an overall loss function, and train a multimodal emotion recognition model based on the obtained overall loss; The overall loss function consists of a single-modal specific information learning loss function, a common information learning loss function between modalities, a minimum sufficient representation loss function for multimodal fusion features, and an emotion recognition and classification loss function; The single-modal unique information learning loss function includes an audio single-modal unique information learning loss function and a video single-modal unique information learning loss function, which are used to learn the unique information of different modalities and filter out noise and redundant information in each modality. The common information learning loss function between modalities is used to learn the common information between modalities; The multimodal fusion feature minimum sufficient representation loss function is used to extract the information most relevant to the emotion recognition task, filter out noise and redundant information in the fusion features, and obtain the multimodal fusion feature minimum sufficient representation. The emotion recognition classification loss function is used to obtain the final emotion classification result; The overall loss function is expressed as: ； In the formula, This represents the loss function for learning information specific to a single audio modality. This represents the loss function for learning information specific to a single modality of the video. This represents the loss function for learning information specific to a single modality. The loss function represents the shared information learning between modalities. This represents the loss function that minimizes the fully sufficient representation of multimodal fusion features. This represents the loss function for emotion recognition and classification. The audio single-modal unique information learning loss function is expressed as: ； In the formula, The minimum sufficient representation of audio modal features. The true label representing the audio modality. This represents the initial audio unimodal features. Represents the Lagrange multipliers. This represents the mutual information between two feature vectors; The video single-modal unique information learning loss function is expressed as: ； In the formula, The minimum sufficient representation of video modal features. The true label representing the video modality, This represents the initial single-modal features of the video; The loss function for learning common information among the modalities is expressed as: ； The minimum sufficient representation loss function of the multimodal fusion features is expressed as: ； In the formula, This represents the minimum sufficient representation of multimodal fusion features. This represents the initial multimodal fusion features. Indicates the true label of the sample; The emotion recognition classification loss function is expressed as follows: ； In the formula, Represents the cross-entropy loss function Indicates the predicted value. Indicates the true label; Initial video unimodal features Initial audio unimodal features The data are then fed into a multilayer perceptron to obtain single-modal features. ; Based on the information bottleneck theory, a loss function for learning the unique information of the single modality is defined, and mutual information is used to learn the single modality features. By extracting their unique information and filtering out noise and redundancy from the audio and video modalities, the minimum sufficient representation of the video modal features is obtained. Minimal sufficient representation of audio modal features .

Images