A decision-level fusion-based audio-video dual-modal emotion recognition method

Abstract

The application discloses an audio-video dual-mode emotion recognition method based on decision-level fusion, adopts a CNN-LSTM model to perform emotion recognition of an audio single mode, adopts a YOLO model to perform emotion recognition of a video single mode, cuts video slices according to the length of Ls based on the length of an audio segment, and synchronizes emotion recognition results of the audio-video dual mode; emotion confidence and PAD dimension values of the audio slices and the video slices are jointly composed into a time sequence with a step of Ls, 7 classifications of the video mode are aligned with 5 classifications of the audio mode; an enumeration weight method is adopted to fuse emotion recognition results of the audio-video dual mode, and a time sequence of emotion confidence and final emotion after audio-video dual-mode fusion are obtained. The audio-video dual-mode emotion recognition is synchronized, confidence results can be synchronously output for the audio-video mode for the same time step, and better audio-video dual-mode emotion fusion results are obtained.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology and relates to a dual-modal emotion recognition method for audio and video based on decision-level fusion. Background Technology

[0002] In the field of emotion recognition research, both audio and video unimodal emotion recognition have achieved good results. To fully utilize the emotional information from audio signals and facial expressions, audio-video bimodal emotion recognition has been extensively studied. The fusion strategies for audio-video bimodal emotion recognition are divided into three types: data-level fusion, feature-level fusion, and decision-level fusion. Among them, decision-level fusion is widely used due to its simple training and the fact that it does not require absolute synchronization of the audio and video signals.

[0003] Decision-level fusion involves determining the credibility of each modality before coordinating and jointly making decisions. This is easier to perform than feature-level fusion, but the key is to explore the importance of each modality for emotion recognition. For example... Figure 1 As shown, the fusion strategies adopted at the decision level include those based on statistical rules (sum rule, product rule, maximum / minimum / median rule, etc.), enumeration weights, adaptive enhancement, Bayesian inference and its generalization theory (Dempster-Shafer theory, dynamic Bayesian network), fuzzy integrals, etc., among which the enumeration weight method is the most widely used method.

[0004] However, numerous problems exist in practical applications. Facial expression videos often contain richer information and features than audio / speech, leading to inconsistencies in the types of emotions that can be recognized between the two modalities. For example, the video modality can recognize seven emotions, while the audio modality can only recognize five. Existing decision-level fusion-based audio-visual emotion recognition methods essentially "cripple" the types of emotions in both video and audio modalities, such as only achieving three-level emotion classification. While this does solve the problem of inconsistent emotion recognition across the two modalities, the method is relatively simplistic and crude, failing to obtain accurate emotion representations.

[0005] Meanwhile, both facial expression video-based emotion recognition and discrete emotion recognition in speech and audio modes can result in no results. For example, if no face appears in the video or no voice is detected, the emotion recognition result will be empty, which will be reflected as "none" in the output of the model or program. This will cause the inability to fuse the results due to the "none" result.

[0006] Meanwhile, existing audio and video emotion recognition methods based on decision-level fusion are all based on the fusion of discrete speech emotion recognition in audio modality and facial expression recognition in video modality at the decision level, and their output emotion results lack continuous dimensional data.

[0007] Therefore, it is necessary to improve existing technologies to overcome their shortcomings. Summary of the Invention

[0008] The purpose of this invention is to provide a decision-level fusion-based audio-video bimodal emotion recognition method, which reprocesses the data collected in the video modality and the data collected in the audio modality to obtain better audio-video bimodal emotion fusion results.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] A decision-level fusion-based audio-video bimodal emotion recognition method includes the following steps:

[0011] S1. Audio monomodal emotion recognition: The CNN-LSTM model processes the input audio sequence and slices the real-time audio according to the length of Ls. The CNN-LSTM model predicts each slice of audio, realizing discrete 5-class emotion recognition and continuous speech dimension value PAD prediction, and obtaining a time series of emotion confidence and PAD time series.

[0012] S2. Video-based unimodal emotion recognition: The YOLO model performs facial emotion recognition on the input video, obtaining a sequence of emotion confidence scores.

[0013]

[0014] Among them, f n This represents the nth frame of the video stream; p is the confidence level. This represents the confidence vector of the nth frame of the video stream;

[0015] S3. Synchronization of audio and video bimodal emotion recognition results: Based on the length of the audio segment, the real-time video is sliced ​​according to the length of Ls. Each video slice can yield a confidence vector, thus realizing the synchronization of audio and video bimodal emotion recognition results.

[0016] S4. Alignment of audio and video bimodal emotion recognition results: The emotion confidence and PAD dimension values ​​of audio and video slices are combined to form a time series with a step size of Ls. Combined with the Valence-Arousal emotion model and using dynamic weights, the confidence of face Disgusted and Fear emotions are assigned to the confidence of Angry, Sad and Surprised emotions, respectively. The 7 categories of video modality are aligned with the 5 categories of audio modality emotion.

[0017] S5. Fusion of audio and video bimodal emotion recognition results: After aligning the emotion results of the audio modality and the video modality, the enumeration weight method is used to fuse the emotion recognition results of the audio and video bimodality to obtain a time series of emotion confidence and the final emotion after audio and video bimodal fusion.

[0018] As a further improvement of one embodiment of the present invention, in step S1, the CNN-LSTM model performs VAD detection before predicting the audio sequence to distinguish between static audio segments without voice and non-static audio segments with voice; when the CNN-LSTM model is input with an audio segment without voice, the CNN-LSTM model will output the result of no emotion classification, and the PAD value is assigned to 0.

[0019] As a further improvement of one embodiment of the present invention, in step S2, if the YOLO model does not detect the face in the video frame, it outputs no emotion classification result; if the YOLO model detects the face in the video frame, it obtains features that effectively represent different emotions, and processes the obtained features to obtain a sequence of emotion confidence.

[0020] As a further improvement to one embodiment of the present invention, in step S3, if the frame rate of the input video is f, then each If s corresponds to a frame in the video, then each video slice contains L*f frames. The YOLO model detects these L*f frames and obtains a set of no more than L*f confidence vectors, i.e. The confidence sequence is further processed as follows:

[0021] (3) Remove the none value to form a new sequence. Since 0 ≤ none quantity ≤ L*f, therefore 0 ≤ m ≤ L*f. When none quantity = L*f, the emotion result of this video slice is also none.

[0022] (4) Statistics The emotion that appears most frequently for each emotion category is selected as the sequence. The baseline sentiment, and then extract that sentiment from the sequence. Extract from it to form a new sequence.

[0023] will sequence The median of the confidence scores for the seven emotions was used as the confidence score for the video slice.

[0024] As a further improvement to one embodiment of the present invention, in step S4, the calculation method for incorporating the confidence scores of Disgusted and Fear emotions into the confidence scores of Angry, Sad, and Surprised emotions is as follows:

[0025]

[0026] Where n is a positive integer, t nL Indicates the Lth, 2nd, 3rd, ..., n*Lth seconds. represents the confidence level of Angry at the n*Lth second; R represents the maximum range of values ​​for the PAD dimension; Arousal is the predicted value for the continuous speech emotion dimension in the audio modality.

[0027] As a further improvement to one embodiment of the present invention, in step S5, the calculation method for fusing the emotion recognition results of the audio and video dual modalities is as follows:

[0028]

[0029] Where n is a positive integer, t 2n Indicates the Lth, 2nd, 3rd, ..., n*Lth seconds. Let represent the confidence level of video modality emotion recognition at the n*Lth second. This represents the confidence level of audio modality emotion recognition at the n*Lth second. w1 represents the confidence level of emotion recognition by audio-video dual-modal fusion at the n*Lth second, w2 represents the fusion weight coefficient of the video modality, and w2 represents the fusion weight coefficient of the audio modality.

[0030] As a further improvement of one embodiment of the present invention, it also includes S6, correction of the audio-video dual-modal emotion fusion result: correction is made for the case where there is no voice in the continuous segment or no face is detected in the continuous segment, which solves the problem that the audio-video modal emotion recognition result cannot be fused when it is none.

[0031] As a further improvement to one embodiment of the present invention, in step S6, the fusion rules are as follows:

[0032] Rule 1: When the recognition results of audio and video modalities are both emotion confidence scores, the output result is a linearly weighted confidence score with PAD dimension values;

[0033] Rule 2: When the recognition result of the video modality is none and the recognition result of the audio modality is emotion confidence, the output result is the emotion confidence of the unfused audio modality, and it has a PAD dimension value;

[0034] Rule 3: When the recognition result of the video modality is the emotion confidence score and the recognition result of the audio modality is none, the output result is the emotion confidence score of the unfused video modality, without the PAD dimension value;

[0035] Rule 4: When the recognition results of all video modalities are none, the output result is none, with no PAD dimension value.

[0036] The above technical solution has the following beneficial effects:

[0037] (1) Synchronization of audio and video bimodal emotion recognition was achieved. For the same time step, the audio and video modal can output confidence results synchronously, resulting in better audio and video bimodal emotion fusion results.

[0038] (2) It achieved 7-category emotion alignment between video modality and 5-category emotion alignment between audio modality;

[0039] (3) It effectively solves the problem of fusion where the emotion recognition result of audio and video modal is none. Attached Figure Description

[0040] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0042] Figure 1 This is a schematic diagram of decision-level fusion provided by the present invention.

[0043] Figure 2 The Valence-Arousal emotion model provided by this invention.

[0044] Figure 3 The fusion rules for video and audio modalities provided by this invention.

[0045] Figure 4 This is a schematic diagram of the process provided by the present invention. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0048] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0049] Example

[0050] See Figure 4 As shown, a dual-modal emotion recognition method for audio and video based on decision-level fusion includes the following steps:

[0051] S1, emotion recognition in audio unimodal mode.

[0052] Based on the CNN-LSTM model, the audio sequence is predicted to achieve discrete five-emotion recognition (Angry, Happy, Neutral, Sad, Surprised) and continuous speech dimension value prediction (PAD, pleasure-displeasure, arousal-nonarousal, dominance-submissiveness).

[0053] Among them, CNN-LSTM (Convolutional Neural Network-Long Short-Term Memory Network) is a hybrid model that combines a convolutional neural network (CNN) and a long short-term memory network (LSTM). In natural language processing, a CNN is first used to extract features from the input text, and then the extracted features are input into an LSTM for sequence modeling and prediction.

[0054] This invention employs the PAD three-dimensional emotion model, which differs from discrete emotion categories that merely classify basic emotions and neglect the relationships between different emotions. The PAD three-dimensional emotion space can continuously and smoothly express different categories of emotions, and thus represent the relationships between emotions, where P represents pleasure-displeasure, A represents arousal-nonarousal, and D represents dominance-submissiveness.

[0055] In practice, the CNN-LSTM model extracts Mel-spectral features from the input audio, and then feeds these features into convolutional neural networks (CNNs), CNN-LSTMs, and other models. Based on the input speech features, the models provide discrete emotion category predictions and continuous emotion intensity predictions. In the process of audio unimodal emotion recognition, the real-time audio is sliced ​​according to the length of Ls (Last Sensitive Time). The sliced ​​audio is then input into models such as CNN-LSTMs to obtain a time series of emotion confidence and a time series of PAD (Patient Deficit Angle).

[0056] In practical applications, there may be continuous audio segments without voice. If these segments are directly input into a CNN-LSTM model for recognition, a large number of false positives will occur. Therefore, it is essential to perform VAD (Voice Activity Detection) before the CNN-LSTM model predicts emotion. VAD detection can distinguish between static audio segments without voice and non-static audio segments with voice. When an audio segment without voice is input into a CNN-LSTM model, the model will output a no-emotion classification result, i.e., "none," and the PAD value will be set to 0.

[0057] The table below shows examples of discrete and continuous emotion recognition results:

[0058]

[0059]

[0060] Table 1 Examples of discrete and continuous emotion recognition results

[0061] S2, emotion recognition in video monomodality.

[0062] Video monomodal facial emotion recognition based on the YOLO model can accurately identify seven types of facial expressions: Angry, Happy, Neutral, Sad, Surprised, Disgusted, and Fear.

[0063] The aforementioned single-modal facial emotion recognition in video mainly covers key stages such as face preprocessing, feature extraction, and expression recognition. Among these, face preprocessing plays a fundamental role in the entire process. Especially when facing facial expression recognition tasks in complex environments, prioritizing face detection is particularly essential.

[0064] Face detection, as the first step in face preprocessing, has the core task of accurately locating facial features in the input video frames. However, in some cases, no face may appear in the input video frame. In this case, the YOLO model will output a no-emotion classification result, labeled "none". Only when a face is successfully detected in the input video frame will the YOLO model proceed to further classify facial expressions.

[0065] During the feature extraction stage, the YOLO model extracts key features from detected facial images that effectively represent different emotions. These features may include facial muscle movement patterns, changes in the shape of facial features, subtle changes in skin color, and so on. Through the precise capture and analysis of these features, the YOLO model can provide rich and valuable information for subsequent expression recognition.

[0066] In the facial expression recognition stage, the YOLO model uses previously extracted features, combined with deep learning algorithms and a large amount of training data, to classify and judge facial expressions. Finally, it outputs a sequence of emotion confidence, as follows:

[0067]

[0068] Among them, f n This represents the nth frame of the video stream; p is the confidence level. This represents the confidence vector of the nth frame of the video stream.

[0069] This sequence reflects the model's level of certainty for each possible emotion classification. For example, if the YOLO model has a high confidence level in classifying a face expression as "Happy" but a low confidence level in classifying other emotions, then it can be considered with relatively high certainty that the face expresses the emotion of happiness.

[0070] S3, synchronization of audio and video dual-modal emotion recognition results.

[0071] In the process of video-based single-modal emotion recognition, to ensure the synchronization of audio and video recognition results, this patent uses the length of the audio segment as a benchmark and slices the real-time video according to the length of Ls. This method cleverly establishes a correlation between audio and video, laying the foundation for achieving accurate synchronous recognition.

[0072] Assuming the video output by the surveillance camera has a frame rate of f, then each For a given frame in the video, each video slice contains L*f frames. The YOLO model detects these L*f frames and obtains a set of no more than L*f confidence vectors, i.e. (In this embodiment, f is set to 25fps and L is set to 2 seconds)

[0073] The confidence sequence is further processed as follows:

[0074] (5) Remove the none value to form a new sequence. Since 0 ≤ none quantity ≤ L*f, therefore 0 ≤ m ≤ L*f. When none quantity = L*f, the emotion result of this video slice is also none.

[0075] (6) Statistics The emotion that appears most frequently for each emotion category is selected as the sequence. The baseline sentiment, and then extract that sentiment from the sequence. Extract from it to form a new sequence.

[0076] (7) Change the sequence The median of the confidence scores for the seven emotions was used as the confidence score for the video slice.

[0077] Using the above method, a confidence vector can be obtained for each video slice, thus achieving synchronization of audio and video bimodal emotion recognition results.

[0078] This synchronous slicing and detection mechanism has several advantages. First, it ensures the temporal consistency of audio and video, enabling accurate comparison and analysis of the information they carry on the same timeline. This avoids information deviations and misinterpretations caused by time asynchrony. Second, by precisely slicing the video, key image information can be effectively extracted, reducing the complexity and computational load of data processing and improving the overall system efficiency.

[0079] The table below is an example of facial expression recognition results:

[0080]

[0081]

[0082] Table 2 Examples of Facial Expression Recognition Results

[0083] S4, Alignment of audio and video bimodal emotion recognition results.

[0084] Steps S1, S2, and S3 yield the emotion confidence and PAD dimension values ​​for the audio and video segments, which together form a time series with a step size of Ls. However, the emotion recognition results for audio and video modalities are 5-class and 7-class respectively, requiring alignment to output a unified 5-class result.

[0085] This patent utilizes the Arousal-Nonarousal dimension value in the three-dimensional emotion of PAD predicted by a continuous speech emotion recognition model, combined with the Valence-Arousal emotion model (such as...). Figure 2 As shown, the confidence scores of the two additional emotions in the video modality—Disgusted and Fear—are weighted and added to the confidence scores of the three emotions: Angry, Sad, and Surprised.

[0086] This patent employs dynamic weighting to categorize the confidence scores of Disgusted and Fear emotions into the confidence scores of Angry, Sad, and Surprised emotions, respectively, as detailed below:

[0087]

[0088]

[0089] Where n is a positive integer, t nL Indicates the Lth, 2nd, 3rd, ..., n*Lth seconds. represents the confidence level of Angry at the n*Lth second; R represents the maximum range of values ​​for the PAD dimension (3 in this embodiment); Arousal is the predicted value of the continuous speech emotion dimension in the audio modality.

[0090] The above method aligns the 7-category emotion classification of the video modality with the 5-category emotion classification of the audio modality, improving the accuracy and reliability of emotion recognition. By aligning the classification results of the two different modalities (video and audio), the unique information they each carry can be comprehensively utilized. The video modality may focus on capturing visual cues such as facial expressions and body movements, while the audio modality is better at reflecting acoustic features such as tone, speech rate, and volume. Combining the two can compensate for the limitations of a single modality, thus enabling more comprehensive and accurate emotion recognition.

[0091] S5, fusion of audio and video dual-modal emotion recognition results.

[0092] After aligning the emotion results of the audio and video modalities, this patent employs an enumeration weighting method to fuse the emotion recognition results of the audio and video dual modalities. The specific method is as follows:

[0093]

[0094] Where n is a positive integer, t 2n Indicates the Lth, 2nd, 3rd, ..., n*Lth seconds. Let represent the confidence level of video modality emotion recognition at the n*Lth second. This represents the confidence level of audio modality emotion recognition at the n*Lth second. This represents the confidence level of emotion recognition achieved through audio-video bimodal fusion at the n*Lth second. In this embodiment, the fusion weight coefficient w1 for the video modality is 0.7, and the fusion weight coefficient w2 for the audio modality is 0.3.

[0095] After the above processing, the final result is It is a time series of sentiment confidence, The emotion category corresponding to the maximum value is taken as the final emotion after audio-video bimodal fusion.

[0096] By fusing emotion results aligned with both audio and video modalities, the comprehensiveness and accuracy of emotion recognition are enhanced. The video modality provides rich visual information, such as facial expressions and body posture; the audio modality focuses on vocal features, such as tone, speech rate, and timbre. Fusing the classification results from these two modalities fully leverages their respective strengths, avoiding potential information gaps or misjudgments that may occur with a single modality, thus enabling a more comprehensive and accurate capture and understanding of emotions.

[0097] S6, a correction to the audio-visual bimodal emotion fusion results.

[0098] Since in practical applications there may be consecutive segments without voices or consecutive segments without detected faces, resulting in an "none" emotion recognition result for audio or video modalities, this patent establishes the following rules to further address the "none" value fusion problem in practical applications, such as... Figure 3 As shown.

[0099] The fusion rules are explained as follows:

[0100] Rule 1: When the recognition results of audio and video modalities are both emotion confidence scores, the output result is a linearly weighted confidence score with PAD dimension values;

[0101] Rule 2: When the recognition result of the video modality is none and the recognition result of the audio modality is emotion confidence, the output result is the emotion confidence of the unfused audio modality, that is, the emotion confidence of the 5-class speech classification, and has PAD dimension value;

[0102] Rule 3: When the recognition result of the video modality is the emotion confidence score and the recognition result of the audio modality is the none value, the output result is the emotion confidence score of the unfused video modality, that is, the emotion confidence score of the 7-class facial expression, without PAD dimension value, specifically, all PAD dimension values ​​are assigned to 0;

[0103] Rule 4: When the recognition results of the video modal are all none, the output result is none, with no PAD dimension value. Specifically, all PAD dimension values ​​are assigned the value 0.

[0104] The above effective fusion rules solve the problem of fusion failure when the audio and video modal emotion recognition result is none.

[0105] The table below shows an example of an audio-video bimodal fusion emotion recognition result:

[0106] Time / s Emotional fusion results Confidence Pleasure value Arousal value Dominance value 1L Happy 54.39％ 1.53 2.98 -0.52 2L Happy 41.65％ 58.30％ -2.76 -1.05 3L Sad 53.39％ -2.32 -0.05 -2.65 4L Happy 51.05％ 1.03 0.88 2.96 5L Happy 51.38％ 2.55 -2.00 -2.04 6L Happy 39.52％ -2.03 0.16 -0.60 7L Sad 47.09％ 1.60 -2.43 -2.75 8L Happy 50.22％ 0.11 1.67 -2.85 9L None None 0.00 0.00 0.00 10L Happy 57.31％ -1.88 0.50 -2.42 11L Happy 36.62％ -1.69 -1.47 2.77 12L Happy 31.18％ 38.37％ -1.58 -0.35 13L Neutral 53.28％ 38.13％ 1.06 0.60 14L Neutral 26.09％ 46.20％ 1.50 -0.49 15L Neutral 58.45％ 43.69％ 0.76 1.17 16L Neutral 61.85％ 41.85％ 2.95 -0.60 17L Neutral 76.26％ 44.47％ 2.07 1.36 18L None None 0.00 0.00 0.00 19L Neutral 32.06％ -2.28 1.81 -0.61 20L Neutral 39.96％ 1.45 -2.48 2.81

[0107] Table 3. Examples of Audio-Video Dual-Modal Fusion Emotion Recognition Results. The specific parameters involved in this embodiment are as follows:

[0108]

[0109]

[0110] By correcting the audio-video bimodal emotion fusion results, the problem of inability to fuse when the audio-video modal emotion recognition result is none is solved, minimizing recognition errors caused by special circumstances and improving the accuracy and reliability of emotion recognition.

[0111] In summary, the embodiments of the present invention achieve the following technical effects: Audio monomodal processing captures rich emotion recognition features from elements such as tone, speed, volume, and rhythm; video monomodal processing captures emotion recognition features from facial expressions, body posture, and eye movements. By processing the emotion recognition of both, the limitations of each are overcome. For example, audio may be affected by background noise, while video may be inaccurate due to lighting, angle, or other issues. By comprehensively considering the continuity and correlation of the audio and video bimodal processing along the time axis, the recognition results of both are synchronized, aligned, fused, and corrected, ultimately obtaining more comprehensive and accurate emotion changes, bringing greater convenience and value to human life and work.

[0112] The key technical features and protection points of this patent are as follows:

[0113] (1) Based on the Valence-Arousal emotion model, dynamic weights are used to classify the confidence scores of Disgusted and Fear emotions of the face into the confidence scores of Angry, Sad and Surprised emotions, respectively, thus achieving the alignment of 7 categories of video modality with 5 categories of audio modality emotion.

[0114] (2) An effective set of fusion rules was invented, which solved the problem that fusion was not possible when the audio and video modal emotion recognition results were none;

[0115] (3) An effective method for synchronizing audio and video emotion recognition results was developed, namely, removing the none value to construct a sequence, extracting the emotion category with the highest confidence score and the largest number of statistics to construct a new sequence, and taking the median of the sequence as the confidence score of the video slice.

[0116] Obviously, the embodiments described above are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0117] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0118] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for audio-visual bimodal emotion recognition based on decision-level fusion, characterized in that: Comprise the following steps: S1, audio single mode emotion recognition: the CNN-LSTM model carries out audio sequence processing to the input audio, slices the real-time audio according to the length of Ls, the CNN-LSTM model carries out prediction to each slice audio, realizes discrete 5 class emotion recognition and continuous voice dimension value PAD prediction, obtains an emotion confidence time sequence and PAD time sequence; S2, video single mode emotion recognition: the YOLO model carries out face emotion recognition to the input video, obtains a sequence about emotion confidence: wherein f n represents the n-th image of the video stream; p is the confidence, p fn represents the confidence vector of the n-th image of the video stream; S3, audio and video dual mode emotion recognition result synchronization: taking the length of audio segment as the benchmark, the real-time video is sliced according to the length of Ls, each video slice can obtain a confidence vector, and the audio and video dual mode emotion recognition result synchronization is realized; S4, audio and video dual mode emotion recognition result alignment: the emotion confidence and PAD dimension value of audio slice and video slice jointly constitute a time sequence with a step of Ls, combined with Valence-Arousal emotion model and adopting dynamic weight, the confidence of face Disgusted and Fear emotion is respectively attributed to the confidence of Angry, Sad and Surprised emotion, and the 7 classification of video mode is aligned with the 5 classification emotion of audio mode; S5, audio and video dual mode emotion recognition result fusion: after aligning the emotion results of audio mode and video mode, the enumeration weight method is adopted to fuse the audio and video dual mode emotion recognition results, and a time sequence of emotion confidence and the final emotion of audio and video dual mode fusion are obtained.

2. The audio-video dual modal emotion recognition method of claim 1, wherein: In step S1, the CNN-LSTM model carries out VAD detection before predicting the audio sequence, and distinguishes the audio segments without static human voice from the audio segments with non-static human voice;When the CNN-LSTM model inputs the audio segment without human voice, the CNN-LSTM model will output the result without emotion classification, and the PAD value is all assigned as 0. 3.The audio-video bimodal emotion recognition method of claim 1, wherein: In step S2, the YOLO model outputs the result without emotion classification when it does not detect the face part in the video frame, and outputs a sequence about emotion confidence according to the obtained features.

4. The audio-video dual modal emotion recognition method of claim 1, wherein: In step S3, the frame rate of the input video is f, then every For a frame of the video, every video slice contains L*f frame images; the YOLO model detects the L*f frame images, and a group of confidence vectors with no more than L*f is obtained, i.e. Further processing of the confidence sequence is as follows: (1) If none is not equal to the value of the first element of the sequence, then the value of the first element of the sequence is removed, and a new sequence is formed. Since 0nonequantityL*f, 0mL*f, when nonequantity=L*f, the mood result of the video slice is also none.​ (2) Statistics The number of times of corresponding emotion categories, the most frequently occurring emotion as a sequence The reference emotion of the sequence and extract the emotion from the sequence The sequence The median of the seven emotional confidences in the middle as the confidence of the video slice.

5. The audio-video dual modal emotion recognition method of claim 1, wherein: In step S4, the calculation method of attributing the confidence of face Disgusted and Fear emotion to the confidence of Angry, Sad and Surprised emotion is as follows: wherein n is a positive integer, t nL represents the L, 2L, 3L, …, n*L seconds, represents the confidence of Angry at the n*L second; R represents the maximum value range of the PAD dimension value; Arousal is the continuous speech emotion dimension prediction value under the audio mode.

6. The audio-video dual modal emotion recognition method of claim 1, wherein: In step S5, the calculation method of fusing the audio and video dual mode emotion recognition results is as follows: wherein n is a positive integer, t nL denotes the Lth, 2Lth, 3Lth, …, n*Lth second, denotes the confidence of the video modality emotion recognition at the n*Lth second, denotes the confidence of the audio modality emotion recognition at the n*Lth second, denotes the confidence of the audio-video bimodality fusion emotion recognition at the n*Lth second, w1 denotes the fusion weight coefficient of the video modality, and w2 denotes the fusion weight coefficient of the audio modality.

7. The audio-video dual modal emotion recognition method of claim 1, wherein: It also includes S6, correction of audio and video dual mode emotion fusion result: for the case that there is no human voice in the continuous segment or the face is not detected in the continuous segment, the problem that the audio and video mode emotion recognition results cannot be fused when they are none value is solved.

8. The audio-video dual modal emotion recognition method of claim 7, wherein: In step S6, the fusion rule is as follows: Rule 1: when the recognition results of audio and video modes are all emotion confidence, the output result is the linearly weighted confidence, and there is PAD dimension value. Rule 2: When the recognition result of the video modality is the none value, and the recognition result of the audio modality is the emotion confidence, the output result is the un-fused audio modality emotion confidence with the PAD dimension value; Rule 3: When the recognition result of the video modality is the emotion confidence, and the recognition result of the audio modality is the none value, the output result is the un-fused video modality emotion confidence without the PAD dimension value; Rule 4: When the recognition result of the video modality is all the none value, the output result is the none value without the PAD dimension value.

Images