Multi-modal emotion recognition method based on facial image and text feature induction

Abstract

The invention discloses a multi-modal emotion recognition method based on facial image and text feature induction. The method comprises the following steps: acquiring a plurality of facial images to form an image set and carrying out manual marking; obtaining a plurality of texts to form a text set and performing manual marking; performing cross combination on the image set and the text set according to a marking result to form a stimulation combination set containing a plurality of image-text stimulation materials; collecting EEG signals and EOG signals of a subject under different combinations of stimulation materials; the EEG signal and the EOG signal are preprocessed; respectively extracting EEG features and EOG features of the EEG signals and the EOG signals after preprocessing; and training a prediction model through the extracted EEG features and EOG features, and dynamically performing emotion prediction on the image-text published by the user through the trained prediction model. According to the multi-modal emotion recognition method based on facial image and text feature induction, emotion recognition can be carried out on image-text data issued by a user.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion signal processing, and in particular relates to a multimodal emotion recognition method based on induction of facial images and text features. Background Art

[0002] With the rapid development of artificial intelligence, emotional computing has gradually become a hot research topic. Emotional data is the research object in the field of emotional computing, and it is a necessary guarantee for subsequent research. High-quality emotional data can improve the level of the entire research and ensure the reliability of the research topic. Therefore, it is necessary to establish an emotional database for research related to emotional computing.

[0003] Human emotions are closely related to sensory experiences, and traditional data sets can only stimulate two senses at most. Most of them are induced by single stimulation such as pictures (visual), audio (auditory), and video (visual and auditory). The number of stimulated senses is relatively monotonous, and the induced emotions are relatively weak.

[0004] Social media platforms, especially Xiaohongshu, a social platform where users generate content and share their lives and experiences, often contain a large amount of user face image information, as well as text information expressing personal feelings. If the text content can be harmonious with the image content, they will complement each other and create a stronger emotional atmosphere. However, if the combination is not good, it will be mutually exclusive, causing users to be separated from the overall created scene and their emotions to change uncontrollably.

[0005] In addition, the feature expression of multimodal emotions and multimodal fusion methods are two key issues in multimodal emotion recognition. Good feature representation can capture the rich emotional clues that can summarize various contents, and a good fusion mechanism can effectively integrate the information of each modality. In theory, emotion recognition based on multimodal fusion can combine the complementary information between different modalities, but choosing a suitable fusion method is a major problem in the application of multimodal emotion recognition.

[0006] Therefore, there is an urgent need to provide a new multimodal emotion recognition method based on joint induction of facial image and text to solve the above problems. Summary of the invention

[0007] The present invention provides a multimodal emotion recognition method based on facial image and text feature induction to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0008] A multimodal emotion recognition method based on facial image and text feature induction, comprising:

[0009] Obtain a number of facial images to form an image set and manually mark them;

[0010] Obtain a number of texts to form a text set and manually mark them;

[0011] The image set and the text set are cross-combined according to the marking results to form a stimulus combination set containing a number of image-text stimulus materials;

[0012] The EEG and EOG signals of the subjects under different combinations of stimulus materials were collected, and the subjects gave feedback on their emotional experience after completing the experiment. All the collected EEG and EOG signals were intercepted for useful data from the event occurrence calibration point to the event end calibration point to establish a multi-stimulus-induced multimodal emotion database;

[0013] Preprocessing EEG signals and EOG signals;

[0014] Extracting respective EEG features and EOG features from the preprocessed EEG signals and EOG signals;

[0015] The prediction model is trained by extracting EEG features and EOG features, and the trained prediction model is used to predict the emotions of the pictures and texts posted by users.

[0016] Furthermore, the specific method of obtaining a plurality of facial images to form an image set and manually marking them is:

[0017] The AffectNet large-scale facial expression dataset is selected as the image set. The AffectNet large-scale facial expression dataset contains eight facial expressions as well as the intensity of valence and arousal. The emotional polarity and emotional intensity of each facial image are marked. The emotional polarity includes positive and negative, and the emotional intensity includes high and low.

[0018] Furthermore, the specific method of obtaining a plurality of texts to form a text set and manually marking them is:

[0019] Crawl the texts posted by users whose number of fans is greater than the preset value on the Xiaohongshu social platform, and mark the sentiment polarity and sentiment intensity of each text. The sentiment polarity includes positive and negative, and the sentiment intensity includes high and low.

[0020] Furthermore, the specific method of cross-combining the image set and the text set according to the marking results to form a stimulus combination set containing a plurality of image-text stimulus materials is:

[0021] According to the different emotional polarities and emotional intensities of the pictures and the different emotional polarities and emotional intensities of the texts, they were combined in pairs to form stimulus materials with a total of 16 combinations, including high emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-positive emotion, high emotion-positive emotion-low emotion-positive emotion, high emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-negative emotion, high emotion-positive emotion-low emotion-negative emotion, high emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-negative emotion, low emotion-negative emotion-low emotion-negative emotion and low emotion-positive emotion-high emotion-negative emotion.

[0022] Furthermore, the specific method of preprocessing the EEG signal and the EOG signal includes at least three of removing useless channel signals, filtering, bad conductor detection and interpolation, re-referencing, dividing the test unit data and removing artifacts.

[0023] Furthermore, the specific method of training the prediction model by extracting EEG features and EOG features is:

[0024] The features of the EEG signal and the EOG signal are fused in a feature layer according to the series rule to obtain a fused feature vector.

[0025] The EEG features, EOG features and fused feature vectors are respectively sent to the classification model for joint training. The loss of each classification model is weighted based on the contribution of each feature to the emotion recognition effect to obtain the total loss value. The parameters of the next round are updated by back propagation to optimize the model training and improve the prediction effect, and finally the prediction accuracy of each model is obtained.

[0026] According to the analysis, the maximum value rule is used to perform the decision-making layer fusion strategy on the three prediction results obtained in the previous step to determine the final predicted emotion category.

[0027] Furthermore, the EEG features, EOG features and the fused feature vectors are each sent to a classification model for learning and training. During the training, the loss weight of each input feature is determined according to its contribution to emotion recognition. The weight optimization method is to weigh multiple loss functions by considering the mean square error uncertainty of each task.

[0028] Furthermore, the classification model adopts a Transformer classifier.

[0029] Furthermore, a supervised approach is adopted to train the user emotion four-classification model, using the acquired EEG features and EOG features, taking the multi-class cross entropy loss as the optimization objective function, and using the back-propagation algorithm to return the gradient to minimize the objective function;

[0030] The objective function is calculated as:

[0031]

[0032] Where M represents the total number of samples in the training data set. Represents the true label of the Zth text-image combination; minimize the objective function Loss to obtain the trained user emotion four-classification model.

[0033] Further, the image set is divided into a male image set and a female image set according to gender, and the text set is divided into a male text set and a female text set according to gender;

[0034] The male image set and the male text set are cross-combined to form a male stimulus combination set. Based on the above process, a male prediction model is obtained, and the emotional prediction of the image and text dynamics posted by male users is performed through the male prediction model;

[0035] The female image set and the female text set are cross-combined to form a female stimulus combination set. Based on the above process, a female prediction model is obtained, and the emotional prediction of the image and text dynamics posted by female users is performed through the female prediction model.

[0036] The benefit of the present invention lies in that the multimodal emotion recognition method based on facial image and text feature induction is capable of performing emotion recognition on graphic and text data posted by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 It is a schematic diagram of a multimodal emotion recognition method based on facial image and text feature induction according to the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] like Figure 1 The present invention shows a multimodal emotion recognition method based on facial image and text feature induction, comprising:

[0041] S1: Obtain a number of facial images to form an image set and manually label them.

[0042] In the implementation of the present application, the specific method of obtaining a plurality of facial images to form an image set and manually marking them is:

[0043] The AffectNet large-scale facial expression dataset is selected as the image set. The AffectNet large-scale facial expression dataset contains about 400,000 facial images, which contain eight facial expressions (neutral, happy, angry, sad, fearful, surprised, disgusted, and contemptuous) as well as the intensity of valence and arousal. The emotional polarity and emotional intensity of each facial image are marked. The emotional polarity includes positive and negative, and the emotional intensity includes high and low.

[0044] S2: Obtain several texts to form a text set and manually label them.

[0045] In the implementation manner of the present application, the specific method of obtaining a plurality of texts to form a text set and manually marking them is:

[0046] The texts published by users whose number of fans on the Xiaohongshu social platform is greater than a preset value are captured, and the emotional polarity and emotional intensity of each text are marked. The emotional polarity includes positive and negative, and the emotional intensity includes high and low. Specifically, the number of fans can be set as needed, and the number of users can also be set as needed.

[0047] In this application, it is necessary to manually calibrate the sentiment polarity and sentiment intensity. Specifically, in this application, 6 students with a master's or doctoral degree in marketing were invited to use a professional scale for annotation.

[0048] S3: The image set and the text set are cross-combined according to the labeling results to form a stimulus combination set containing several image-text stimulus materials.

[0049] In the implementation manner of the present application, the specific method of cross-combining the image set and the text set according to the marking results to form a stimulus combination set containing a plurality of image-text stimulus materials is:

[0050] According to the different emotional polarities and emotional intensities of the pictures and the different emotional polarities and emotional intensities of the texts, they were combined in pairs to imitate the content of social platforms, such as Xiaohongshu, to form stimulus materials with 16 combinations of high emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-positive emotion, high emotion-positive emotion-low emotion-positive emotion, high emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-negative emotion, high emotion-positive emotion-low emotion-negative emotion, high emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-negative emotion, low emotion-negative emotion-low emotion-negative emotion and low emotion-positive emotion-high emotion-negative emotion, forming a stimulus combination set.

[0051] S4: Collect the EEG and EOG signals of the subjects under different combinations of stimulus materials, and ask the subjects to feedback their emotional experience after completing the experiment. The purpose of feedback on emotional experience is to verify and correct the degree of match between the emotions marked by the combination of images and texts and the actual emotional response of the subjects. This feedback can help researchers determine whether the preset emotional labels are consistent with the actual induced emotions, thereby improving the accuracy of model predictions. Useful data from all collected EEG and EOG signals are intercepted from the event occurrence calibration point to the event end calibration point to establish a multi-stimulus induced multimodal emotion database.

[0052] S5: Preprocess the EEG signal and the EOG signal.

[0053] In the embodiments of the present application, the specific method of preprocessing the EEG signal and the EOG signal includes but is not limited to: removing useless channel signals, filtering, bad conduction detection and interpolation, re-referencing, dividing the test unit data, and removing artifacts. Specifically, at least three of the methods of removing useless channel signals, filtering, bad conduction detection and interpolation, re-referencing, dividing the test unit data, and removing artifacts are selected.

[0054] S6: Extracting respective EEG features and EOG features from the pre-processed EEG signal and EOG signal.

[0055] In the present application, the extracted features include but are not limited to power spectral density PSD, differential entropy DE, differential asymmetry DASM and rational asymmetry RASM.

[0056] S7: The prediction model is trained by extracting EEG features and EOG features, and the emotion prediction of the pictures and texts posted by users is performed through the trained prediction model.

[0057] In the embodiment of the present application, the specific method of training the prediction model by extracting EEG features and EOG features is:

[0058] S71: The features of the two modalities of EEG signal and EOG signal are fused according to the feature layer fusion strategy according to the series rule to obtain a fused feature vector.

[0059] S72: EEG features, EOG features and fused feature vectors are respectively sent to the classification model for joint training, and the loss of each classification model is weighted based on the contribution of each feature to the emotion recognition effect to obtain the total loss value, and the parameters of the next round are updated by back propagation to optimize the model training and improve the prediction effect, and finally obtain the respective prediction accuracy. In this application, the classification model adopts the Transformer classifier.

[0060] Specifically, the EEG features, EOG features and the fused feature vectors are each sent to a classification model for learning and training. During the training, the loss weight of each input feature is determined according to its contribution to emotion recognition. The weight optimization method is to weigh multiple loss functions by considering the mean square error uncertainty of each task.

[0061] This method improves the accuracy, robustness and efficiency of emotion recognition through multimodal fusion and dynamic weight adjustment combined with the mean square error uncertainty trade-off loss function.

[0062] S73: Based on the analysis, the maximum value rule is used to perform a decision-making layer fusion strategy on the three prediction results obtained in the previous step to determine the final predicted emotion category.

[0063] Here, the prediction model predicts the results containing sentiment polarity and sentiment intensity (high and low) and sentiment polarity (positive and negative).

[0064] In the embodiment of the present invention, a supervised method is adopted to train the user emotion four-classification model, using the feature set extracted in step S6, and taking the multi-class cross entropy loss (Loss) as the optimization objective function. The gradient is returned by the back propagation algorithm to gradually minimize the objective function, thereby optimizing the training process of the model and improving the accuracy of emotion recognition;

[0065] The objective function is calculated as:

[0066]

[0067] Minimize the objective function Loss to obtain the trained user emotion four-classification model.

[0068] As an optional implementation, the image set is divided into a male image set and a female image set according to gender, and the text set is divided into a male text set and a female text set according to gender. The male image set and the male text set are cross-combined to form a male stimulus combination set, and a male prediction model is obtained based on the above process. The male prediction model is used to perform emotional prediction on the image and text dynamics posted by male users. The female image set and the female text set are cross-combined to form a female stimulus combination set, and a female prediction model is obtained based on the above process. The female prediction model is used to perform emotional prediction on the image and text dynamics posted by female users.

[0069] It is understandable that there are certain differences in the details of text expression between men and women. In order to improve the classification accuracy of the classification model, it is preferred to train a prediction model for men and women respectively. When the publisher is a woman, the emotion type is predicted by the female prediction model. When the publisher is a man, the emotion type is predicted by the male prediction model.

[0070] In some cases, the system captured selfies on the Xiaohongshu platform, which contained a large number of images without obvious expressions. Although the expressions in the images were not obvious, the system still performed emotional annotation by analyzing subtle facial features, such as slight changes in the eyes and mouth corners, combined with the text descriptions posted by users. For example, a user posted an expressionless selfie with a strong emotional caption. The system identified the user's emotional state as positive or negative by analyzing the selfie and text.

[0071] The method of the present application can also perform emotion recognition on facial images without obvious expressions, specifically comprising the following steps:

[0072] Obtain a set of selfie images with no expression or slight facial expression, and extract potential emotional features by analyzing subtle facial features, including but not limited to small muscle changes in the corners of the eyes, mouth corners, eyebrows, etc.;

[0073] The features of expressionless images are integrated with the text content posted by users, and the emotional features of images and texts are jointly processed using a multimodal fusion strategy. The accuracy of emotion recognition is improved by analyzing the emotional consistency or complementarity between images and texts.

[0074] Use machine learning methods (such as deep neural networks, Transformer models, etc.) in emotion recognition models to enhance the model's ability to recognize emotion in expressionless images through joint training of expressionless images and text features;

[0075] The above method is applicable to different types of images including selfies, static images posted on social platforms, advertising images, etc., and is especially widely used in selfies. Through this method, the user's emotional state can be accurately predicted in expressionless images, and the prediction results of emotional polarity (positive or negative) and emotional intensity (high or low) can be generated.

[0076] The present invention has the ability to process images without obvious facial expressions, and can accurately predict the user's emotional state by analyzing subtle facial features and image background, combined with text information. This technological breakthrough overcomes the limitations of traditional emotion recognition methods that only rely on obvious facial expressions. The present invention is particularly suitable for processing self-portraits on social media. Even if the user's facial expressions in these photos are not obvious, the system can still perform emotion recognition through subtle changes in facial features and in combination with text. This technological breakthrough makes emotion recognition not only limited to traditional expression changes, but also able to provide accurate emotion prediction in the absence of expression.

[0077] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method based on facial image and text feature induction, characterized in that: Include: Obtain a number of facial images to form an image set and manually mark them; Obtain a number of texts to form a text set and manually mark them; The image set and the text set are cross-combined according to the marking results to form a stimulus combination set containing a number of image-text stimulus materials; The EEG and EOG signals of the subjects under different combinations of stimulus materials were collected, and the subjects gave feedback on their emotional experience after completing the experiment. All the collected EEG and EOG signals were intercepted for useful data from the event occurrence calibration point to the event end calibration point to establish a multi-stimulus-induced multimodal emotion database; Preprocessing EEG signals and EOG signals; Extracting respective EEG features and EOG features from the preprocessed EEG signals and EOG signals; The prediction model is trained by extracting EEG features and EOG features, and the trained prediction model is used to predict the emotions of the pictures and texts posted by users.

2. The multimodal emotion recognition method based on facial image and text feature induction according to claim 1, characterized in that: The specific method of obtaining a plurality of facial images to form an image set and manually marking them is: The AffectNet large-scale facial image expression dataset is selected as the image set. The AffectNet large-scale facial image expression dataset contains eight facial expressions and the intensity of valence and arousal. The emotional polarity and emotional intensity of each facial image are marked. The emotional polarity includes positive and negative, and the emotional intensity includes high and low.

3. The multimodal emotion recognition method based on facial image and text feature induction according to claim 2, characterized in that: The specific method of obtaining a plurality of texts to form a text set and manually marking them is: Crawl the texts posted by users whose number of fans is greater than the preset value on the Xiaohongshu social platform, and mark the sentiment polarity and sentiment intensity of each text. The sentiment polarity includes positive and negative, and the sentiment intensity includes high and low.

4. The multimodal emotion recognition method based on facial image and text feature induction according to claim 3 is characterized in that: The specific method of cross-combining the image set and the text set according to the marking results to form a stimulus combination set containing a plurality of image-text stimulus materials is: According to the different emotional polarities and emotional intensities of the pictures and the different emotional polarities and emotional intensities of the texts, they were combined in pairs to form stimulus materials with a total of 16 combinations, including high emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-positive emotion, high emotion-positive emotion-low emotion-positive emotion, high emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, high emotion-negative emotion-high emotion-negative emotion, high emotion-positive emotion-low emotion-negative emotion, high emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-low emotion-negative emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-positive emotion-low emotion-positive emotion, low emotion-negative emotion-low emotion-positive emotion, low emotion-positive emotion-high emotion-positive emotion, low emotion-negative emotion-high emotion-negative emotion, low emotion-negative emotion-low emotion-negative emotion and low emotion-positive emotion-high emotion-negative emotion.

5. The multimodal emotion recognition method based on facial image and text feature induction according to claim 1, characterized in that: The specific method of preprocessing the EEG signal and the EOG signal includes at least three of removing useless channel signals, filtering, bad conductor detection and interpolation, re-referencing, dividing the experimental unit data and removing artifacts.

6. The multimodal emotion recognition method based on facial image and text feature induction according to claim 1, characterized in that: The specific method of training the prediction model by extracting EEG features and EOG features is: The features of the EEG signal and the EOG signal are fused in a feature layer according to the series rule to obtain a fused feature vector. The EEG features, EOG features and fused feature vectors are respectively sent to the classification model for joint training. The loss of each classification model is weighted based on the contribution of each feature to the emotion recognition effect to obtain the total loss value. The parameters of the next round are updated by back propagation to optimize the model training and improve the prediction effect, and finally the prediction accuracy of each model is obtained. According to the analysis, the maximum value rule is used to perform the decision-making layer fusion strategy on the three prediction results obtained in the previous step to determine the final predicted emotion category.

7. The multimodal emotion recognition method based on facial image and text feature induction according to claim 6, characterized in that: The EEG features, EOG features and the fused feature vectors are each sent to a classification model for learning and training. During the training, the loss weight of each input feature is determined according to its contribution to emotion recognition. The weight optimization method is to weigh multiple loss functions by considering the mean square error uncertainty of each task.

8. The multimodal emotion recognition method based on facial image and text feature induction according to claim 7, characterized in that: The classification model adopts Transformer classifier.

9. The multimodal emotion recognition method based on facial image and text feature induction according to claim 8, characterized in that: A supervised approach is adopted to train the user emotion four-classification model, using the acquired EEG features and EOG features, taking the multi-class cross entropy loss as the optimization objective function, and using the back-propagation algorithm to return the gradient to minimize the objective function; The objective function is calculated as: Among them, i represents 4 classification categories, represents the true label of the Zth sample, It indicates the probability that the model predicts that the sample belongs to the i-th category.

10. The multimodal emotion recognition method based on facial image and text feature induction according to claim 1, characterized in that: The image set is divided into a male image set and a female image set according to gender, and the text set is divided into a male text set and a female text set according to gender; The male image set and the male text set are cross-combined to form a male stimulus combination set. Based on the above process, a male prediction model is obtained, and the emotional prediction of the image and text dynamics posted by male users is performed through the male prediction model; The female image set and the female text set are cross-combined to form a female stimulus combination set. Based on the above process, a female prediction model is obtained, and the emotional prediction of the image and text dynamics posted by female users is performed through the female prediction model.

Images