Audio and video bimodal emotion recognition method based on decision level fusion

Through the synchronous slice and dynamic weight alignment of CNN-LSTM and YOLO models, combined with the Valence-Arousal emotion model, the problem of inconsistent and resultless emotions in audio and video emotion recognition is solved, and the dual-mode emotion recognition of audio and video is realized Synchronization and alignment enhance the accuracy and reliability of emotion recognition.

CN119028377AActive Publication Date: 2024-11-26SUZHOU INST OF ARTIFICIAL INTELLIGENCE SHANGHAI JIAOTONG UNIV

Patent Information

Application Number
CN202410939678.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-26
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing audio and video emotion recognition method based on decision-level fusion has problems such as inconsistent emotions, no results, and lack of continuous dimensional data, which leads to the inability to effectively integrate emotional recognition results.

Method used

The CNN-LSTM model is used to perform audio single-mode emotion recognition and the YOLO model is used to perform video single-mode emotion recognition. Through synchronous slices and dynamic weight alignment, combined with the Valence-Arousal emotion model, the enumeration weight method is used to perform audio and video dual-mode emotion Identify the fusion of the results and formulate correction rules to process none values.

Benefits of technology

The synchronization and alignment of dual-mode emotions recognition of audio and video is achieved, solving the problem of fusion of emotions recognition results as none values, and obtaining more accurate and continuous emotional expressions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028377A_ABST
    Figure CN119028377A_ABST
Patent Text Reader

Abstract

The invention discloses an audio and video dual-mode emotion recognition method based on decision-level fusion, which comprises the following steps: carrying out audio single-mode emotion recognition by adopting a CNN-LSTM model, carrying out video single-mode emotion recognition by adopting a YOLO model, and slicing a video according to the length of Ls by taking the length of an audio clip as a reference to realize synchronization of audio and video dual-mode emotion recognition results; jointly forming a time sequence with the step length of Ls by the emotion confidence coefficients and the PAD dimension values of the audio slices and the video slices, and aligning the 7-classification emotion of the video mode with the 5-classification emotion of the audio mode; and fusing the audio and video bimodal emotion recognition results by adopting an enumeration weight method to obtain a time sequence of emotion confidence and a final emotion after audio and video bimodal fusion. Synchronization of audio and video dual-mode emotion recognition is achieved, for the same time step length, the audio and video modes can synchronously output a confidence result, and a better audio and video dual-mode emotion fusion result is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of emotion recognition and relates to an audio and video dual-modal emotion recognition method based on decision-level fusion. Background Art

[0002] In the field of emotion recognition research, audio and video single-modal emotion recognition have achieved good recognition results. In order to make full use of the emotional information of audio signals and facial expressions, audio and video dual-modal emotion recognition has been widely studied. The fusion strategies of audio and video dual-modal emotion recognition are divided into data-level fusion, feature-level fusion and decision-level fusion. Among them, decision-level fusion is widely used because of its simple training and the fact that it does not require absolute synchronization of the timing of audio and video signals.

[0003] Decision-level fusion is to find out the credibility of each modality and then coordinate and make joint decisions. It is easier to perform than feature-level fusion, but the key is to explore the importance of each modality for emotion recognition. Figure 1 As shown in the figure, the fusion strategies adopted at the decision level include those based on statistical rules (sum rule, product rule, maximum / minimum / median rule, etc.), enumeration weights, adaptive enhancement, Bayesian inference and its generalization theory (Dempster-Shafer theory, dynamic Bayesian network), fuzzy integral, etc., among which the enumeration weight method is the most widely used method.

[0004] However, there are many problems in practical applications. Compared with voice audio, facial expression videos often have richer information and features, which leads to the fact that the types of emotions that can be recognized by the two modalities are not uniform. For example, 7 emotions can be recognized in the video modality, but only 5 emotions can be recognized in the audio modality. Existing audio and video emotion recognition methods based on decision-level fusion basically "castrate" the types of emotions in the video and audio modalities, such as only realizing 3 emotion classifications. This does solve the problem of inconsistent types of emotion recognition in dual modalities, but the method is relatively simple and crude and cannot obtain accurate emotional expression.

[0005] At the same time, whether it is emotion recognition in the facial expression video mode or discrete emotion recognition in the voice audio mode, there are cases where no results are obtained. For example, if no face appears in the video screen or no voice is detected, the result of emotion recognition will be empty, which will be reflected as none in the output result of the model or program, resulting in the problem of inability to integrate due to the none result.

[0006] At the same time, existing audio and video emotion recognition methods based on decision-level fusion are all based on the fusion of discrete speech emotion recognition in the audio modality and facial expression recognition in the video modality at the decision level, and the output emotion results lack continuous dimensional data.

[0007] Therefore, it is necessary to improve the prior art to overcome the defects in the prior art. Summary of the invention

[0008] The purpose of the present invention is to provide an audio-video dual-modal emotion recognition method based on decision-level fusion, which reprocesses the data collected in the video mode and the data collected in the audio mode to obtain better audio-video dual-modal emotion fusion results.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A method for audio and video bimodal emotion recognition based on decision-level fusion, comprising the following steps:

[0011] S1. Audio single-modal emotion recognition: The CNN-LSTM model processes the input audio sequence and slices the real-time audio according to the length of Ls. The CNN-LSTM model predicts each sliced ​​audio to achieve discrete 5-category emotion recognition and continuous speech dimension value PAD prediction, and obtains a time series of emotion confidence and a PAD time series;

[0012] S2. Emotion recognition in single-mode video: The YOLO model performs facial emotion recognition on the input video and obtains a sequence of emotion confidence:

[0013]

[0014] Among them, f n represents the nth frame image of the video stream; p is the confidence level, Represents the confidence vector of the nth frame image of the video stream;

[0015] S3, synchronization of audio and video dual-modal emotion recognition results: based on the length of the audio clip, the real-time video is sliced ​​according to the length of Ls, and a confidence vector can be obtained for each video slice, thus achieving synchronization of audio and video dual-modal emotion recognition results;

[0016] S4. Alignment of audio and video bimodal emotion recognition results: The emotion confidence and PAD dimension values ​​of the audio and video slices are combined into a time series with a step length of Ls. The Valence-Arousal emotion model is combined with dynamic weights to classify the confidence of the Disgusted and Fear emotions of the face into the confidence of the Angry, Sad and Surprised emotions respectively, and the 7 categories of the video modality are aligned with the 5 categories of the audio modality.

[0017] S5. Fusion of audio and video dual-modal emotion recognition results: After aligning the emotion results of the audio modality and the video modality, the enumeration weight method is used to fuse the audio and video dual-modal emotion recognition results to obtain a time series of emotion confidence and the final emotion after the fusion of the audio and video dual modalities.

[0018] As a further improvement of an embodiment of the present invention, in step S1, the CNN-LSTM model performs VAD detection before predicting the audio sequence to distinguish between static audio clips without voice and non-static audio clips with voice; when the CNN-LSTM model inputs an audio clip without voice, the CNN-LSTM model will output a result without emotion classification, and the PAD value is assigned to 0.

[0019] As a further improvement of an embodiment of the present invention, in step S2, the YOLO model does not detect the facial parts in the video frame and outputs a result without emotion classification. The YOLO model detects the facial parts in the video frame and obtains features that effectively represent different emotions, and obtains a sequence of emotion confidence based on the above-obtained features.

[0020] As a further improvement of an embodiment of the present invention, in step S3, the frame rate of the input video is f, then each s corresponds to a frame of the video, and each video slice contains L*f frame images; the YOLO model detects the L*f frame images and obtains a set of no more than L*f confidence vectors, that is, The confidence sequence is further processed as follows:

[0021] (3) The none value is removed to form a new sequence Since 0≤none number≤L*f, 0≤m≤L*f, when none number=L*f, the emotion result of the video slice is also none;

[0022] (4) Statistics The number of corresponding emotion categories, the emotion with the most occurrences is taken as the sequence The baseline emotion is then removed from the sequence Extract it from the

[0023] will sequence The median of the seven emotional confidences is taken as the confidence of the video slice.

[0024] As a further improvement of an embodiment of the present invention, in step S4, the confidence of the Disgusted and Fear emotions of the face is respectively included in the confidence of the Angry, Sad and Surprised emotions by the following calculation method:

[0025]

[0026] Where n is a positive integer, t nL represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, Represents the confidence of Angry at the n*Lth second; R represents the maximum value range of the PAD dimension value; Arousal is the predicted value of the continuous speech emotion dimension under the audio mode.

[0027] As a further improvement of an embodiment of the present invention, in step S5, the calculation method for fusion of the emotion recognition results of the audio and video dual modalities is:

[0028]

[0029] Where n is a positive integer, t 2n represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, represents the confidence of emotion recognition in video mode at the n*Lth second, represents the confidence of the audio modality emotion recognition at the n*Lth second, It represents the confidence of emotion recognition of audio and video dual-modal fusion at the n*Lth second, w1 represents the fusion weight coefficient of the video modality, and w2 represents the fusion weight coefficient of the audio modality.

[0030] As a further improvement of an embodiment of the present invention, it also includes S6, correction of the audio and video dual-modal emotion fusion result: correction is performed for the situation where there is no voice in the continuous segment or no face is detected in the continuous segment, thereby solving the problem that the audio and video modal emotion recognition result cannot be fused when the value is none.

[0031] As a further improvement of an embodiment of the present invention, in step S6, the fusion rule is as follows:

[0032] Rule 1: When the recognition results of audio and video modalities are both emotion confidence, the output result is the linearly weighted confidence with PAD dimension value;

[0033] Rule 2: When the recognition result of the video modality is the value of none and the recognition result of the audio modality is the emotion confidence, the output result is the unfused audio modality emotion confidence with a PAD dimension value;

[0034] Rule 3: When the recognition result of the video modality is the emotion confidence and the recognition result of the audio modality is the value of none, the output result is the unfused video modality emotion confidence without the PAD dimension value;

[0035] Rule 4: When the recognition results of the video modality are all none values, the output result is none value, and there is no PAD dimension value.

[0036] The above technical solution has the following beneficial effects:

[0037] (1) The synchronization of audio and video dual-modal emotion recognition is achieved. For the same time step, the audio and video modalities can synchronously output the confidence results, thus obtaining better audio and video dual-modal emotion fusion results;

[0038] (2) Achieved emotional alignment of 7 categories in video modality and 5 categories in audio modality;

[0039] (3) It effectively solves the fusion problem of audio and video modal emotion recognition results being none. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0042] Figure 1 A schematic diagram of decision-level fusion provided by the present invention.

[0043] Figure 2 The Valence-Arousal emotion model provided by the present invention.

[0044] Figure 3 The present invention provides a fusion rule for the video mode and the audio mode.

[0045] Figure 4 A schematic diagram of the process provided by the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meanings as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0048] In the present invention, unless otherwise specified, the directional words used, such as "up, down, top, bottom", usually refer to the directions shown in the drawings, or to the components themselves in the vertical, perpendicular or gravity directions; similarly, for ease of understanding and description, "inside and outside" refer to the inside and outside relative to the outline of each component itself, but the above-mentioned directional words are not used to limit the present invention.

[0049] Example

[0050] See also Figure 4 As shown, a method for audio and video dual-modal emotion recognition based on decision-level fusion includes the following steps:

[0051] S1, Emotion recognition in audio unimodality.

[0052] Based on the CNN-LSTM model, the audio sequence is predicted to realize discrete 5-category emotion recognition (Angry, Happy, Neutral, Sad, Surprised) and continuous speech dimension value prediction (PAD, pleasure-displeasure, arousal-nonarousal, dominance-submissiveness).

[0053] Among them, CNN-LSTM (Convolutional Neural Network-Long Short-Term Memory Network) is a hybrid model that combines Convolutional Neural Network (CNN) and Long Short-Term Memory Network (LSTM). In natural language processing, CNN is first used to extract features from the input text, and then the extracted features are input into LSTM for sequence modeling and prediction.

[0054] The present invention adopts the PAD three-dimensional emotion model, which is different from the discrete emotion categories that only classify basic emotions and ignore the relationship between different emotions. The PAD three-dimensional emotion space can continuously and smoothly express different categories of emotions, and then can express the relationship between emotions, where P represents pleasure (pleasure-displeasure), A represents arousal (arousal-nonarousal), and D represents dominance (dominance-submissiveness).

[0055] When used, the CNN-LSTM model extracts Mel-spectrogram features based on the input audio, and then inputs the extracted features into models such as convolutional neural networks (CNNs) and CNN-LSTMs. The models give discrete emotion category predictions and continuous emotion intensity predictions based on the input speech features. In the process of audio unimodal emotion recognition, the real-time audio is sliced ​​according to the length of Ls, and the sliced ​​audio is input into models such as CNN-LSTM to obtain a time series of emotion confidence and a PAD time series.

[0056] In actual applications, there may be continuous clips without voice. If they are directly input to the CNN-LSTM model for recognition, a large number of false detections will occur. Therefore, it is very necessary to first perform VAD detection (Voice Activity Detection) before the CNN-LSTM model predicts emotions. VAD detection can distinguish between static audio clips without voice and non-static audio clips with voice. When the CNN-LSTM model is input with an audio clip without voice, the CNN-LSTM model will output a result of no emotion classification, that is, none, and the PAD value is assigned to 0.

[0057] The following table is an example of discrete emotion recognition results and continuous emotion recognition results:

[0058]

[0059]

[0060] Table 1 Examples of discrete and continuous emotion recognition results

[0061] S2, emotion recognition in video unimodality.

[0062] The video unimodal facial emotion recognition based on the YOLO model can accurately recognize seven types of facial expressions, namely Angry, Happy, Neutral, Sad, Surprised, Disgusted, and Fear.

[0063] The above-mentioned single-mode facial emotion recognition in videos mainly covers key links such as face preprocessing, feature extraction and expression recognition. Among them, the face preprocessing link plays a fundamental role in the whole process. Especially when facing the task of facial expression recognition in complex environments, it is particularly necessary to detect the face first.

[0064] Face detection is the first step in face preprocessing. Its core task is to accurately locate the face parts in the input video frame. However, in some cases, there may be no face in the input video frame. In this case, the YOLO model will output a result without emotion classification, which is marked as "none". Only when a face is successfully detected in the input video frame will the YOLO model further carry out subsequent work and conduct in-depth classification of facial expressions.

[0065] In the feature extraction stage, the YOLO model extracts key features that can effectively represent different emotions from the detected face images. These features may include the movement patterns of facial muscles, changes in the morphology of facial features, subtle changes in skin color, etc. By accurately capturing and analyzing these features, the YOLO model can provide rich and valuable information for subsequent expression recognition.

[0066] In the expression recognition phase, the YOLO model will classify and judge facial expressions based on the previously extracted features, combined with deep learning algorithms and a large amount of training data. Finally, it outputs a sequence of emotion confidence, as follows:

[0067]

[0068] Among them, f n represents the nth frame image of the video stream; p is the confidence level, Represents the confidence vector of the nth frame image in the video stream.

[0069] This sequence reflects the model's degree of certainty for each possible emotion classification. For example, if the YOLO model has a high confidence that a certain facial expression is "Happy", but a low confidence for other emotions, then it can be relatively certain that the emotion expressed by this face is happy.

[0070] S3, synchronization of audio and video dual-modal emotion recognition results.

[0071] In the process of video unimodal emotion recognition, in order to ensure the synchronization of audio and video recognition results, this patent uses the length of the audio segment as a benchmark and slices the real-time video according to the length of Ls. This method cleverly establishes the association between audio and video, laying the foundation for achieving accurate synchronous recognition.

[0072] Assuming that the frame rate of the video output by the surveillance camera is f, then each Corresponding to a frame of the video, each video slice contains L*f frame images. The YOLO model detects the L*f frame images and obtains a set of no more than L*f confidence vectors, that is, (In this embodiment, f is set to 25fps and L is set to 2 seconds)

[0073] The confidence sequence is further processed as follows:

[0074] (5) The none value is removed to form a new sequence Since 0≤none number≤L*f, 0≤m≤L*f, when none number=L*f, the emotion result of the video slice is also none;

[0075] (6) Statistics The number of corresponding emotion categories, the emotion with the most occurrences is taken as the sequence The baseline emotion is then removed from the sequence Extract it from the

[0076] (7) Change the sequence The median of the seven emotional confidences is taken as the confidence of the video slice.

[0077] Through the above method, a confidence vector can be obtained for each video slice, realizing the synchronization of audio and video dual-modal emotion recognition results.

[0078] This synchronous slicing and detection mechanism has many advantages. First, it ensures the temporal consistency of audio and video, allowing the information carried by the two to be accurately compared and analyzed on the same timeline. It avoids information deviation and misinterpretation caused by time asynchrony. Second, by accurately slicing the video, key image information can be effectively extracted, reducing the complexity and computational complexity of data processing and improving the operating efficiency of the entire system.

[0079] The following table is an example of facial expression recognition results:

[0080]

[0081]

[0082] Table 2 Examples of facial expression recognition results

[0083] S4, Alignment of audio and video bimodal emotion recognition results.

[0084] Through steps S1, S2, and S3, the emotion confidence and PAD dimension value of the audio and video slices can be obtained, which together form a time series with a step length of Ls. However, the emotion recognition of audio and video modalities is a 5-classification result and a 7-classification result respectively. The emotion recognition results need to be aligned to output a unified 5-classification result.

[0085] This patent uses the Arousal-Nonarousal dimension value in the PAD three-dimensional emotion predicted by the continuous speech emotion recognition model, combined with the Valence-Arousal emotion model (such as Figure 2 As shown in the figure, the confidences of the two additional emotions in the video mode, Disgusted and Fear, are weighted and added to the confidences of the three emotions, Angry, Sad, and Surprised.

[0086] This patent adopts dynamic weights to classify the confidence of Disgusted and Fear emotions into the confidence of Angry, Sad and Surprised emotions respectively, as follows:

[0087]

[0088]

[0089] Where n is a positive integer, t nL represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, represents the confidence of Angry at the n*Lth second; R represents the maximum value range of the PAD dimension value (the value is 3 in this embodiment); Arousal is the continuous speech emotion dimension prediction value under the audio mode.

[0090] Through the above method, the 7 categories of video modality can be aligned with the 5 categories of audio modality, which helps to improve the accuracy and reliability of emotion recognition. By aligning the classification results of two different modalities (video and audio), the unique information they carry can be comprehensively utilized. The video modality may focus on capturing visual clues such as facial expressions and body movements, while the audio modality is better at reflecting acoustic features such as tone, speaking speed, and volume. The combination of the two can make up for the limitations that may exist in a single modality, thereby identifying emotions more comprehensively and accurately.

[0091] S5, fusion of audio and video dual-modal emotion recognition results.

[0092] After aligning the emotion results of the audio and video modalities, this patent uses the enumeration weight method to fuse the emotion recognition results of the audio and video dual modalities. The specific method is as follows:

[0093]

[0094] Where n is a positive integer, t 2n represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, represents the confidence of emotion recognition in video mode at the n*Lth second, represents the confidence of the audio modality emotion recognition at the n*Lth second, Indicates the confidence of emotion recognition of audio and video dual-modal fusion at the n*Lth second. In this embodiment, the fusion weight coefficient w1 of the video modality is 0.7, and the fusion weight coefficient w2 of the audio modality is 0.3.

[0095] After the above processing, the final is a time series of sentiment confidence. The emotion category corresponding to the maximum value is taken as the final emotion after the audio and video bimodal fusion.

[0096] By fusing the emotion results of the aligned audio and video modalities, the comprehensiveness and accuracy of emotion recognition can be enhanced. The video modality can provide rich visual information, such as facial expressions, body postures, etc., while the audio modality focuses on sound features, such as intonation, speaking speed, timbre, etc. Fusion of the classification results of these two modalities can make full use of their respective advantages, avoid the information loss or misjudgment that may exist in a single modality, and thus capture and understand emotions more comprehensively and accurately.

[0097] S6, Correction of audio and video bimodal emotion fusion results.

[0098] In practical applications, there may be situations where there is no voice or no face detected in a continuous segment, resulting in the emotion recognition result of the audio or video modality being none. Therefore, this patent formulates the following rules to further solve the problem of none value fusion in practical applications, such as Figure 3 shown.

[0099] The fusion rules are explained as follows:

[0100] Rule 1: When the recognition results of audio and video modalities are both emotion confidence, the output result is the linearly weighted confidence with PAD dimension value;

[0101] Rule 2: When the recognition result of the video modality is the value of none and the recognition result of the audio modality is the emotion confidence, the output result is the unfused audio modality emotion confidence, that is, the emotion confidence of the five speech categories, and has the PAD dimension value;

[0102] Rule 3: When the recognition result of the video modality is the emotion confidence and the recognition result of the audio modality is the value of none, the output result is the emotion confidence of the unfused video modality, that is, the emotion confidence of the 7 categories of facial expressions, without the PAD dimension value. Specifically, the PAD dimension values ​​are all assigned 0;

[0103] Rule 4: When the recognition results of the video modality are all none values, the output result is none value, and there is no PAD dimension value. Specifically, all PAD dimension values ​​are assigned 0.

[0104] Through the above effective fusion rules, the problem of being unable to fuse when the audio and video modal emotion recognition results are none is solved.

[0105] The following table is an example of the emotion recognition result of audio and video dual-modal fusion:

[0106] Time / s Emotional fusion results Confidence Pleasure value Arousal Value Dominance value 1L Happy 54.39% 1.53 2.98 -0.52 2L Happy 41.65% 58.30% -2.76 -1.05 3L Sad 53.39% -2.32 -0.05 -2.65 4L Happy 51.05% 1.03 0.88 2.96 5L Happy 51.38% 2.55 -2.00 -2.04 6L Happy 39.52% -2.03 0.16 -0.60 7L Sad 47.09% 1.60 -2.43 -2.75 8L Happy 50.22% 0.11 1.67 -2.85 9L None None 0.00 0.00 0.00 10L Happy 57.31% -1.88 0.50 -2.42 11L Happy 36.62% -1.69 -1.47 2.77 12L Happy 31.18% 38.37% -1.58 -0.35 13L Neutral 53.28% 38.13% 1.06 0.60 14L Neutral 26.09% 46.20% 1.50 -0.49 15L Neutral 58.45% 43.69% 0.76 1.17 16L Neutral 61.85% 41.85% 2.95 -0.60 17L Neutral 76.26% 44.47% 2.07 1.36 18L None None 0.00 0.00 0.00 19L Neutral 32.06% -2.28 1.81 -0.61 20L Neutral 39.96% 1.45 -2.48 2.81

[0107] Table 3 Example of emotion recognition results of audio and video dual-modal fusion The parameters involved in this embodiment are as follows:

[0108]

[0109]

[0110] By correcting the audio and video dual-modal emotion fusion results, the problem of being unable to be fused when the audio and video modal emotion recognition results are none is solved, minimizing the recognition errors caused by special circumstances and improving the accuracy and reliability of emotion recognition.

[0111] In summary, the embodiments of the present invention achieve the following technical effects: the tone, speech speed, volume, rhythm and other elements of the sound obtained through the audio single mode contain rich emotion recognition features, and the video single mode obtains emotion recognition features such as facial expressions, body postures, and eye changes. Through the emotion recognition processing of both, the limitations of each are overcome, such as audio may be disturbed by background noise, and video may cause inaccurate information due to problems such as light and angle. Comprehensively considering the continuity and correlation of the audio and video dual modes on the timeline, the synchronization, alignment, fusion and correction of the recognition results of the two are achieved, and finally a more comprehensive and accurate emotional change is obtained, which brings more convenience and value to human life and work.

[0112] The key technical points and protection points of this patent are as follows:

[0113] (1) Based on the Valence-Arousal emotion model, dynamic weights are used to classify the confidence of the Disgusted and Fear emotions of the face into the confidence of the Angry, Sad, and Surprised emotions respectively, achieving the alignment of the seven categories of the video modality and the five categories of the audio modality;

[0114] (2) A set of effective fusion rules was invented to solve the problem that the audio and video modal emotion recognition results cannot be fused when the value is none;

[0115] (3) An effective method for synchronizing audio and video emotion recognition results was developed, which consists of removing the none value to construct a sequence, extracting the emotion category with the largest number of highest confidence statistics to construct a new sequence, and calculating the median of the sequence as the confidence of the video slice.

[0116] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0117] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0118] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for audio and video dual-modal emotion recognition based on decision-level fusion, characterized in that: The steps include: S1. Audio single-modal emotion recognition: The CNN-LSTM model processes the input audio sequence and slices the real-time audio according to the length of Ls. The CNN-LSTM model predicts each sliced ​​audio to achieve discrete 5-category emotion recognition and continuous speech dimension value PAD prediction, and obtains a time series of emotion confidence and a PAD time series; S2. Emotion recognition in single-mode video: The YOLO model performs facial emotion recognition on the input video and obtains a sequence of emotion confidence: Among them, f n represents the nth frame image of the video stream; p is the confidence level, p fn Represents the confidence vector of the nth frame image of the video stream; S3, synchronization of audio and video dual-modal emotion recognition results: based on the length of the audio clip, the real-time video is sliced ​​according to the length of Ls, and a confidence vector can be obtained for each video slice, thus achieving synchronization of audio and video dual-modal emotion recognition results; S4. Alignment of audio and video bimodal emotion recognition results: The emotion confidence and PAD dimension values ​​of the audio and video slices are combined into a time series with a step length of Ls. The Valence-Arousal emotion model is combined with dynamic weights to classify the confidence of the Disgusted and Fear emotions of the face into the confidence of the Angry, Sad and Surprised emotions respectively, and the 7 categories of the video modality are aligned with the 5 categories of the audio modality. S5. Fusion of audio and video dual-modal emotion recognition results: After aligning the emotion results of the audio modality and the video modality, the enumeration weight method is used to fuse the audio and video dual-modal emotion recognition results to obtain a time series of emotion confidence and the final emotion after the fusion of the audio and video dual modalities.

2. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: In step S1, the CNN-LSTM model performs VAD detection before predicting the audio sequence to distinguish between static audio clips without voice and non-static audio clips with voice. When the CNN-LSTM model inputs an audio clip without voice, the CNN-LSTM model will output a result without emotion classification, and the PAD value is assigned to 0.

3. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: In step S2, the YOLO model does not detect the facial parts in the video frame and outputs a result of no emotion classification. The YOLO model detects the facial parts in the video frame and obtains features that effectively represent different emotions, and obtains a sequence of emotion confidence based on the above-obtained features.

4. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: In step S3, the frame rate of the input video is f, then each Corresponding to a frame of the video, each video slice contains L*f frame images; the YOLO model detects the L*f frame images and obtains a set of no more than L*f confidence vectors, that is, The confidence sequence is further processed as follows: (1) The none value is removed to form a new sequence Since 0≤none number≤L*f, 0≤m≤L*f, when none number=L*f, the emotion result of the video slice is also none; (2) Statistics The number of corresponding emotion categories, the emotion with the most occurrences is taken as the sequence The baseline emotion is then removed from the sequence Extract it from the will sequence The median of the seven emotional confidences is taken as the confidence of the video slice.

5. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: In step S4, the confidence of the Disgusted and Fear emotions of the face is respectively classified into the confidence of the Angry, Sad and Surprised emotions by the following calculation method: Where n is a positive integer, t nL represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, Represents the confidence of Angry at the n*Lth second; R represents the maximum value range of the PAD dimension value; Arousal is the predicted value of the continuous speech emotion dimension under the audio mode.

6. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: In step S5, the calculation method for fusion of audio and video dual-modal emotion recognition results is: Where n is a positive integer, t 2n represents the Lth, 2Lth, 3Lth, ..., n*Lth seconds, represents the confidence of emotion recognition in video mode at the n*Lth second, represents the confidence of the audio modality emotion recognition at the n*Lth second, It represents the confidence of emotion recognition of audio and video dual-modal fusion at the n*Lth second, w1 represents the fusion weight coefficient of the video modality, and w2 represents the fusion weight coefficient of the audio modality.

7. The audio-video dual-modal emotion recognition method according to claim 1, characterized in that: It also includes S6, correction of audio and video dual-modal emotion fusion results: correction is made for the situation where there is no voice in the continuous segment or no face is detected in the continuous segment, which solves the problem that the audio and video modal emotion recognition results cannot be fused when the value is none.

8. The audio-video dual-modal emotion recognition method according to claim 7, characterized in that: In step S6, the fusion rules are as follows: Rule 1: When the recognition results of audio and video modalities are both emotion confidence, the output result is the linearly weighted confidence with PAD dimension value; Rule 2: When the recognition result of the video modality is the value of none and the recognition result of the audio modality is the emotion confidence, the output result is the unfused audio modality emotion confidence with a PAD dimension value; Rule 3: When the recognition result of the video modality is the emotion confidence and the recognition result of the audio modality is the value of none, the output result is the unfused video modality emotion confidence without the PAD dimension value; Rule 4: When the recognition results of the video modality are all none values, the output result is none value, and there is no PAD dimension value.

Citation Information

Patent Citations

  • Multi-modal emotion classification method based on text, voice and video fusion

    CN110287389A

  • Audience emotion recognition method, device and system

    CN111401198A

  • Video face emotion recognition method based on frame attention mechanism

    CN115393933A

  • Three-dimensional facial expression animation generation method, system and device and medium

    CN116778044A

  • Multi-modal situational emotion recognition method and system based on wide time category

    CN117149944A

Cited By

  • Multi-modal emotion recognition method and system for service-oriented robot

    CN120995416A

  • A multi-modal emotion recognition method and system for a service-oriented robot

    CN120995416B