A Multimodal Emotion Recognition Method and System for Voice and Facial Expressions of People in Smart Buildings
By employing a multimodal emotion recognition method based on speech and facial expressions, and utilizing residual networks, 3D-CNN, and MLP models for feature extraction and fusion, and designing a loss function that increases the inter-class scatter matrix, the problem of low recognition rate in traditional single-modal methods is solved, achieving higher recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CENT CHINA BRANCH OF CHINA DATANG CORP SCI & TECH RES INST CO LTD
- Filing Date
- 2022-07-28
- Publication Date
- 2026-05-26
AI Technical Summary
Traditional emotion recognition technology mainly relies on unimodal methods, which have low recognition rates and are difficult to accurately identify people's emotional states, especially when applied in smart buildings, where they lack comprehensiveness.
A multimodal emotion recognition method based on speech and facial expressions is adopted. High-order emotion features are extracted through residual networks and 3D-CNN models, and multimodal fusion is performed using MLP models. A loss function that increases the inter-class scatter matrix is designed to enhance feature differences and achieve end-to-end training.
It improves the recognition rate of multimodal emotion recognition based on voice, facial expressions, and images, enhances the ability to distinguish similar emotion categories, and improves the accuracy of recognition.
Smart Images
Figure CN115273905B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of smart building technology, and more specifically, relates to a method and system for multimodal emotion recognition of human voice and facial expressions in smart buildings. Background Technology
[0002] Smart buildings, by utilizing next-generation information technologies such as computers, artificial intelligence, the Internet of Things, big data, and cloud computing, are changing the way people interact with building systems. With continuous societal progress and the ongoing development of the intelligent era, office buildings are gradually evolving into smart buildings. Furthermore, emotional management has become an indispensable human resource management tool for contemporary enterprises. It is a management model where managers motivate employees by showing respect, understanding, and warmth towards them, thereby promoting enterprise development.
[0003] However, traditional corporate emotion management relies primarily on managers manually observing employees' emotional states, which is insufficient for accurate identification. Therefore, smart buildings, as a major trend in future office building development, could effectively motivate employees and promote rapid corporate growth by utilizing technologies such as artificial intelligence, big data, and 6G to automatically identify employees' emotional states and promptly report them to managers. Over the past few decades, emotion recognition has developed a widely adopted classification standard. This standard categorizes emotional states into six types: anger, disgust, fear, sadness, surprise, and happiness. Emotion recognition primarily includes voice, facial expression / image, and physiological signal emotion recognition. Voice emotion recognition identifies a person's emotional state by analyzing their speech rate, tone, and semantics; facial expression / image emotion recognition analyzes facial expressions and muscle movements; and physiological signal emotion recognition identifies a person's emotional state by analyzing physiological signals such as respiration, heart rate, and body temperature. However, single-modal emotion recognition does not express complete emotional information, and its recognition rate falls far short of expectations.
[0004] Therefore, to improve the recognition rate of emotion recognition, the research focus has gradually shifted from single-modal emotion recognition to multimodal emotion recognition. Among these, the voice and facial expression image modalities of individuals can be extracted non-contactly using cameras and voice sensors, making them simple, convenient, and flexible for application in smart buildings. Therefore, this invention mainly studies a method and system for multimodal emotion recognition of human voice and facial expressions in smart buildings, focusing on improving the recognition rate of multimodal emotion recognition of voice and facial expression images by designing a loss function that increases the inter-class divergence matrix. Summary of the Invention
[0005] The purpose of this invention is to propose a method and system for multimodal emotion recognition of voice and facial expressions in smart buildings, thereby improving the recognition rate of multimodal emotion recognition of voice, facial expressions and images.
[0006] In a first aspect, this invention proposes a multimodal emotion recognition method for the voice and facial expressions of people in smart buildings, including:
[0007] The collected voice information and facial expression video are preprocessed, including: extracting MFCC features from the voice information, dividing the facial expression video into frames of facial expression images and extracting facial expression regions from the facial expression images;
[0008] The extracted MFCC features are input into a residual network model to extract high-order sentiment features of the speech information;
[0009] The framed facial expression images and the extracted facial expression regions are input into a 3D-CNN model to extract high-order emotional features of the facial expression images;
[0010] The MLP model is used to perform multimodal fusion of the high-order emotional features of the speech information and the facial expression image to identify the emotional state of the person. In the multimodal fusion process, a pre-designed loss function is used to complete the end-to-end training of the multimodal feature fusion of speech and facial expression image, and to increase the difference between the high-order emotional features of speech information and facial expression image.
[0011] Secondly, this invention proposes a smart building personnel voice and facial expression multimodal emotion recognition system, including: a field data acquisition module, a communication module, a main control module, and an emotion display module;
[0012] The on-site data acquisition module is used to collect voice information and facial expression video data of people inside the building;
[0013] The communication module is used to upload the data collected by the field data acquisition module to the main control module;
[0014] The main control module is used for:
[0015] The collected voice information and facial expression video are preprocessed, including: extracting MFCC features from the voice information, dividing the facial expression video into frames of facial expression images and extracting facial expression regions from the facial expression images;
[0016] The extracted MFCC features are input into a residual network model to extract high-order sentiment features of the speech information;
[0017] The framed facial expression images and the extracted facial expression regions are input into a 3D-CNN model to extract high-order emotional features of the facial expression images;
[0018] The MLP model is used to perform multimodal fusion of the high-order emotional features of the speech information and the facial expression image to identify the emotional state of the person. In the process of multimodal fusion, a pre-designed loss function is used to complete the end-to-end training of the multimodal feature fusion of speech and facial expression image, and to increase the difference between the high-order emotional features of speech information and facial expression image.
[0019] The emotion display module is used to display the emotional state of the person identified by the main control module.
[0020] The beneficial effects of this invention are as follows:
[0021] The speech-expression-image multimodal emotion recognition method of the present invention is divided into preprocessing, residual network and 3D-CNN and feature fusion. Among them, the preprocessing mainly extracts the MFCC features of speech information and divides the expression video into frames and extracts the expression region. The residual network extracts the high-order emotion features of speech information, and the 3D-CNN extracts the high-order emotion features of expression images. The MLP feature fusion is used to complete the fusion of the high-order emotion features of speech and expression images and identify the emotional state of the person. In the process of feature layer fusion, a loss function that can increase the inter-class divergence matrix is designed. It not only realizes the end-to-end training of speech-expression-image multimodal feature fusion, but also increases the difference between the high-order emotion features of speech and expression images, effectively improving the ability to distinguish similar emotion categories, and thus effectively improving the recognition rate of speech-expression-image multimodal emotion recognition.
[0022] The system of the present invention has other features and advantages that will be apparent from or will be set forth in detail in the accompanying drawings and following detailed description, which together serve to explain the particular principles of the invention. Attached Figure Description
[0023] The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0024] Figure 1 The diagram illustrates the steps of a multimodal emotion recognition method for human voice and facial expressions in smart buildings, according to Embodiment 1 of the present invention.
[0025] Figure 2 The diagram illustrates the principle of a multimodal emotion recognition method for human voice and facial expressions in smart buildings, according to Embodiment 1 of the present invention.
[0026] Figure 3The diagram shows a spectrum of voice information in a multimodal emotion recognition method for human voice expressions in a smart building according to Embodiment 1 of the present invention.
[0027] Figure 4 The diagram shows the MFCC feature map of voice information in a smart building personnel voice expression multimodal emotion recognition method according to Embodiment 1 of the present invention.
[0028] Figure 5 and Figure 6 The experimental results of the multi-recognition method of Embodiment 1 of the present invention on the RML and BAUM-1s datasets are shown respectively.
[0029] Figure 7 The diagram shows a structural schematic of a smart building personnel voice and facial expression multimodal emotion recognition system according to Embodiment 2 of the present invention. Detailed Implementation
[0030] The invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0031] Example 1
[0032] like Figure 1 and Figure 2 As shown in the figure, this embodiment provides a method for multimodal emotion recognition of voice and facial expressions of personnel in smart buildings. The method includes:
[0033] Step S1: Preprocess the collected voice information and facial expression video of the personnel. The preprocessing includes: extracting MFCC features of the voice information, dividing the facial expression video into frames into facial expression images and extracting the facial expression regions in the facial expression images.
[0034] Specifically, the MFCC features of speech information conform to human auditory habits and are an effective method for speech feature extraction. Specifically, the method first calculates the spectrum of the speech information using preprocessing and FFT, then calculates the Mel spectrum of the speech information using a Mel triangular filter bank, and finally extracts the MFCC features of the speech information through cepstral analysis.
[0035] In this embodiment, the first-order MFCC features and second-order MFCC features of the speech information are calculated as follows:
[0036]
[0037] Where, d t C represents the first-order difference of the MFCC feature of the t-th speech information; tLet represent the t-th speech information MFCC; Q represent the order of the speech information MFCC; and K represent the time difference of the first-order difference of the speech information MFCC features. Furthermore, the second-order difference of the speech information MFCC features is based on equation (1) and the first-order difference d of the speech information MFCC features. t Calculated.
[0038] The MFCC features, first-order MFCC features, and second-order MFCC features of speech information are compressed into Mel-Cepstral, first-order Mel-Cepstral, and second-order Mel-Cepstral, respectively. The Mel-Cepstral, first-order Mel-Cepstral, and second-order Mel-Cepstral of speech information are then modeled after the RGB three channels of an image, resulting in a speech MFCC feature with a dimension of 64×64×3. Figure 3 A voice message was displayed. Figure 4 It demonstrates the MFCC features, first-order MFCC features, and second-order MFCC features of speech information, as well as Mel cepstral plots, first-order Mel cepstral plots, and second-order Mel cepstral plots.
[0039] For the facial expression image modality, the facial expression video clip is first divided into continuous facial expression image information, then OpenCV is used to automatically extract the facial expression region of the facial expression image, and finally the size of each frame of facial expression image is modified to 240×240×3.
[0040] Step S2: Input the extracted MFCC features into the residual network model to extract high-order sentiment features of the speech information;
[0041] Specifically, high-order emotional features of speech are extracted using residual networks. For example... Figure 2 As shown, the residual network contains 5 residual blocks and 2 fully connected layers (FC6-FC7). Each residual block contains 2 convolutional layers (CL1-CL2), 3 ReLU activation function layers, 2 normalization layers (BN), and one addition operation. The number of neurons in the fully connected layer FC7 is 2048, which means that the dimension of the speech MFCC features input to the residual network is 64×64×3, and the output speech high-order emotion features are...
[0042] Step S3: Input the framed facial expression images and the extracted facial expression regions into the 3D-CNN model to extract high-order emotional features of the facial expression images;
[0043] Specifically, high-order emotional features of facial expression images are extracted using a 3D-CNN model. The 3D-CNN model was originally developed for motion recognition, and later it was also applied to video processing.
[0044] like Figure 2 As shown, the 3D-CNN model includes:
[0045] Seven convolutional layers CL1-CL2-CL3-CL4a-CL4b-CL5a-CL5, five pooling layers (Pool1-Pool2-Pool3-Pool4-Pool5), and two fully connected layers (FC6-FC7).
[0046] The facial expression video is first segmented into frames, and then input into... Figure 2 The high-order emotional features of facial expressions are extracted using a 3D-CNN model. The FC7 layer of the 3D-CNN model has 2048 neurons, meaning the dimension of the high-order emotional features extracted by the 3D-CNN is [missing information].
[0047] Step S4: Use an MLP model to perform multimodal fusion of high-order emotional features of speech information and facial expression images to identify the emotional state of people. In the process of multimodal fusion, use a pre-designed loss function to complete the end-to-end training of multimodal feature fusion of speech and facial expression images, and increase the difference between high-order emotional features of speech information and facial expression images.
[0048] Specifically, in classification optimization, to improve the recognition rate of multimodal emotion recognition for speech, facial expressions, and images, the network should be optimized to increase the differences between emotion state categories. The differences between categories can be measured using the inter-class scatter matrix; the larger the inter-class scatter matrix, the more significant the differences between emotion state categories, which is more beneficial for multimodal emotion recognition.
[0049] The principle for increasing the inter-class scatter matrix is as follows:
[0050] For higher-order emotional features of speech The pattern class for emotional states is defined as {W1,…,W c ,…,W C This is a C-class classification problem. First, define a weighting matrix K. X This represents the similarity between sentiment classes that represent higher-order sentiment features of speech. The specific definition is as follows:
[0051]
[0052] Where i,j∈[1,C], the weighting matrix K X It is a C×C dimensional matrix; Represents the pattern class W i and W j The cosine similarity of the average vectors of the higher-order emotional feature samples of speech; Represents the pattern class W i The average vector is defined as:
[0053]
[0054] in, Represents the pattern class W i The l-th speech sample vector.
[0055] As can be seen from the definition of similarity in equation (2), the more similar the categories of higher-order emotional features in speech are, the higher the corresponding weighting matrix K. X The larger.
[0056] Inter-class scatter matrix S of higher-order emotional features of speech X Defined as:
[0057]
[0058] Where T denotes the transpose of the matrix. The pattern class is W. i The total number of samples. X represents the overall average vector of higher-order emotional features of speech.
[0059] Inter-class scatter matrix S X Describes pattern class W i The average vector of higher-order emotional features of speech The degree of dispersion relative to the overall average vector X. Therefore, the between-class scatter matrix can also be used to describe the degree of dispersion between any two pattern classes, and it is defined as:
[0060]
[0061] in, Describes pattern class W i and W j The degree of dispersion between them. The larger the size, the more pattern class W i and W j The more obvious the differences between them, the better it is for multimodal emotion recognition.
[0062] In addition, to increase the differences between categories, a weighting matrix K is used. X between-class scatter matrix Optimization yields a new inter-class scatter matrix. It is defined as:
[0063]
[0064] As can be seen from equation (6), the more similar the higher-order emotional features of speech are, the better K X The larger it is, the less easily it is distinguishable. However, K X The larger, Also bigger. The larger the value of the weighting matrix K, the more pronounced the differences between categories, which is more beneficial for multimodal emotion recognition. Therefore, by using the weighting matrix K... X Add the matrix to the inter-class scatter matrix In this way, the ability to distinguish similar emotion categories can be effectively improved, which effectively improves the recognition rate of multimodal emotion recognition.
[0065] For higher-order emotional features of facial expressions The pattern class for emotional states is defined as {W1,…,W c ,…,W C This is also a C-classification problem. First, define a weighting matrix K. Y This represents the similarity between high-order emotion feature classes in facial expression images. The specific definition is as follows:
[0066]
[0067] Where i,j∈[1,C], the weighting matrix K Y It is a C×C dimensional matrix; Represents the pattern class W i and W j The cosine similarity of the average vectors of high-order emotion feature samples from facial expression images; Represents the pattern class W i The average vector is defined as:
[0068]
[0069] in, Represents the pattern class W i The l-th facial expression image sample vector.
[0070] Based on equation (5), the high-order emotion feature pattern class W of facial expression images i With pattern class W i Inter-class scatter matrix Defined as:
[0071]
[0072] In addition, to increase the differences between categories, a weighting matrix K is used. Y between-class scatter matrix Optimization yields a new inter-class scatter matrix. It is defined as:
[0073]
[0074] The more similar the higher-order emotional features between facial expression images, the better. Y The larger, The larger the weighting matrix K, the more pronounced the differences between categories, which is more beneficial for multimodal emotion recognition. Therefore, by using the weighting matrix K... Y Add the matrix to the inter-class scatter matrix This can also effectively improve the recognition rate of multimodal emotion recognition.
[0075] like Figure 2 As shown, multimodal fusion is implemented using an MLP, where the first layer of the MLP has 4096 neurons, the second layer has 1024 neurons, and the last layer has C neurons. The ReLU activation function is chosen for the first two layers, and the softmax function is chosen for the last layer. To complete the C classification task, the objective function is defined as:
[0076]
[0077] Where θ represents the network parameters, K represents the number of training samples, X and Y represent the speech and facial expression image training samples, and y i Let Y(·) represent the label, Υ(·) represent the features input to the softmax function in the MLP, softmax(·) represent the softmax function, and l(·) represent the cross-entropy loss function, which is defined as:
[0078]
[0079] Where, y′ i Expressed as softmax(Υ((X,Y);θ)) in equation (11), y i,j Let y′ represent the label of the j-th class of the i-th sample data. i,j This represents the output of the softmax layer for the j-th class of the i-th sample data.
[0080] Furthermore, to increase the difference between categories, the inter-class scatter matrix should be a large value, which can be transformed into the following optimization problem:
[0081]
[0082] Where tr[·] represents the trace of the matrix.
[0083] The larger Equation (13) is, the larger the inter-class scatter matrix is, and the more obvious the differences between classes are. In order to simultaneously complete the classification task and increase the inter-class scatter matrix, Equations (11)-(13) are combined, and a C-classification objective function that can increase the inter-class scatter matrix is defined as follows:
[0084]
[0085] Here, α is a coefficient.
[0086] Based on equation (14), the newly defined loss function in this paper is:
[0087]
[0088] Equation (15) completes the C-classification training in an end-to-end manner. Meanwhile, and It effectively increases the inter-class divergence matrix between high-order emotional features of speech and facial expression images, effectively improves the ability to distinguish similar emotional categories, and thus effectively improves the recognition rate of multimodal emotion recognition of speech and facial expression images.
[0089] Application example:
[0090] The RML and BAUM-1s datasets are publicly available datasets commonly used for multimodal emotion recognition of speech, facial expressions, and images; therefore, they were also selected to evaluate the performance of this method.
[0091] The RML dataset is a public emotion recognition dataset collected by Leisen Multimedia Lab from eight subjects with different languages, accents, and cultural backgrounds. It is approximately 4.2GB in size and contains 720 video clips. Six basic emotions are expressed: anger, disgust, fear, sadness, surprise, and happiness. All video clips were captured using a digital camera in a quiet, well-lit environment and are in AVI format. The sampling rate is 22050Hz, the frame rate is 30fps, the resolution is 720×480×3, and the video length ranges from 3s to 6s. Each video clip provides emotional features for both speech and facial expressions.
[0092] The BAUM-1s dataset is a public emotion recognition dataset collected from 31 subjects. It is approximately 1GB in size and contains 1222 video clips. All subjects recorded their videos in Turkish. Of the 31 subjects, 55% were female and 45% were male. Six emotions were expressed: anger, disgust, fear, sadness, surprise, and happiness. Only 521 video clips expressed these six basic emotions. All video clips were recorded in a studio in MP4 format. The frame rate was 30fps, the resolution was 720×576×3, and the video length was 3-4 seconds. The BAUM-1s dataset represents the subjects' spontaneous expression of emotions after watching the relevant videos.
[0093] The experimental results of the method in this embodiment on the RML and BAUM-1s datasets are shown in the figure. The recognition rates for the six emotion states are as follows: Figure 5 and Figure 6 As shown in Table 1, the comparison results with other algorithms are as follows.
[0094] Table 1 Comparison of experimental results
[0095]
[0096]
[0097] Experimental results show that the method of this invention achieves recognition rates of 98.75% and 76.62% on the RML and BAUM-1s datasets, respectively. These results are superior to several existing emotion state recognition methods. The reason for this is:
[0098] The method of this invention designs a loss function that increases the inter-class divergence matrix during the high-order emotion fusion process of speech and facial expression images. This loss function not only enables end-to-end training of multimodal feature fusion of speech and facial expression images, but also increases the differences between high-order emotion features of speech and facial expression images, effectively improving the ability to distinguish similar emotion categories, and thus effectively improving the recognition rate of multimodal emotion recognition of speech and facial expression images.
[0099] Example 2
[0100] like Figure 7 As shown in the figure, this embodiment provides a smart building personnel voice and facial expression multimodal emotion recognition system, including: a field data acquisition module, a communication module, a main control module, and an emotion display module;
[0101] The on-site data acquisition module is used to collect voice information and facial expression video data of people inside the building;
[0102] The communication module is used to upload the data collected by the field data acquisition module to the main control module;
[0103] The main control module is used for:
[0104] The collected voice information and facial expression video are preprocessed, including: extracting MFCC features from the voice information, dividing the facial expression video into frames into facial expression images and extracting the facial expression regions from the facial expression images;
[0105] The extracted MFCC features are input into the residual network model to extract high-order sentiment features of the speech information;
[0106] The framed facial expression images and the extracted facial expression regions are input into a 3D-CNN model to extract high-order emotional features from the facial expression images.
[0107] Multimodal fusion of high-order emotional features of speech information and facial expression images is performed using an MLP model to identify the emotional state of a person. In the process of multimodal fusion, a pre-designed loss function is used to complete the end-to-end training of multimodal feature fusion of speech and facial expression images, and to increase the difference between high-order emotional features of speech information and facial expression images.
[0108] The emotion display module is used to display the emotional state of the personnel identified by the main control module.
[0109] In this embodiment, the field data acquisition module is located at the field layer of the system, including a sensor cluster such as video cameras and audio sensors. The communication module is located at the communication network layer, mainly including communication network equipment such as fiber optic ring networks and fiber optic switches. The main control module and the emotion display module are located at the control layer. The main control module includes a server or host device. The main control module is used to run the software system of the smart building personnel voice and facial expression multimodal emotion recognition method of Embodiment 1. The emotion display module can be a terminal device such as a PC or mobile phone.
[0110] The intelligent building personnel emotion recognition system in this embodiment first uses cameras and voice sensors to collect voice and image information of personnel, which is then transmitted to the main control module via the communication network layer. Next, the main control module's multimodal emotion method identifies the emotional state of personnel and sends the identified emotional state to the emotion display module to help managers analyze personnel's emotional states and motivate them to promote rapid enterprise development. Furthermore, the core module of this system is the voice, facial expression, and image multimodal emotion recognition method, which designs a loss function that increases the inter-class divergence matrix during the high-order emotion fusion process of voice and facial expression images. This loss function not only enables end-to-end training of multimodal feature fusion of voice, facial expression, and images but also increases the differences between high-order emotion features of voice and facial expression images, effectively improving the ability to distinguish similar emotion categories, and thus effectively improving the recognition rate of multimodal emotion recognition.
[0111] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. A method for multimodal emotion recognition of voice and facial expressions of personnel in smart buildings, characterized in that, include: The collected voice information and facial expression video are preprocessed, including: extracting MFCC features from the voice information, dividing the facial expression video into frames of facial expression images and extracting facial expression regions from the facial expression images; The extracted MFCC features are input into a residual network model to extract high-order sentiment features of the speech information; The framed facial expression images and the extracted facial expression regions are input into a 3D-CNN model to extract high-order emotional features of the facial expression images; The MLP model is used to perform multimodal fusion of the high-order emotional features of the speech information and the facial expression image to identify the emotional state of the person. In the process of multimodal fusion, a pre-designed loss function is used to complete the end-to-end training of the multimodal feature fusion of speech and facial expression image, and to increase the difference between the high-order emotional features of speech information and facial expression image. The loss function is: Where k is the total number of training samples, and C is the total number of emotion state categories. This represents the model's predicted output vector for the i-th training sample. This represents the true sentiment category label of the i-th training sample. Indicates the first i The first sample data j Class tags, This represents the probability that the model predicts the i-th sample data belongs to the j-th class. It is a coefficient. Represents the trace of a matrix. This represents the inter-class scatter matrix of high-order sentiment features of speech information. This is the inter-class scatter matrix of the high-order emotional features of facial expression images.
2. The method according to claim 1, characterized in that, The extraction of MFCC features from the speech information includes: The spectrum of the speech information is calculated, and the spectrum is converted into Mel cepstrum using a Mel triangular filter bank. The Mel cepstrum is then analyzed to extract the MFCC features of the speech information.
3. The method according to claim 2, characterized in that, The first-order MFCC features of the speech information are calculated using the following formula: (1) in, Indicates the first t First-order difference of MFCC features of each speech information; Indicates the first t One voice information MFCC; Q Indicates the order of the MFCC of the speech information; K The time difference represents the first-order difference of the MFCC features of speech information; The second-order difference of the MFCC features of the speech information is based on equation (1) and the first-order difference of the MFCC features of the speech information. Calculated.
4. The method according to claim 3, characterized in that, The process of converting the spectrum to Mel cepstrum using a Mel triangular filter bank and analyzing the Mel cepstrum to extract MFCC features of the speech information includes: The MFCC features, first-order MFCC features, and second-order MFCC features of speech information are compressed into Mel cepstral graphs, first-order Mel cepstral graphs, and second-order Mel cepstral graphs using the Mel triangular filter bank. The Mel-Cepstral, First-Order Mel-Cepstral, and Second-Order Mel-Cepstral of speech information are analogous to the RGB three channels of an image to obtain a dimension... The speech MFCC features.
5. The method according to claim 1, characterized in that, The step of segmenting the facial expression video into facial expression images and extracting the facial expression regions from the facial expression images includes: The facial expression video is divided into frames of continuous facial expression images. OpenCV is used to automatically extract the facial expression regions from each image, and the size of each frame is modified accordingly. .
6. The method according to claim 1, characterized in that, The residual network model includes five residual blocks arranged sequentially from the input to the output and two fully connected layers. Each residual block contains: 2 convolutional layers, 3 ReLU activation function layers, 2 normalization layers, and one addition operation; the last fully connected layer has 2048 neurons.
7. The method according to claim 1, characterized in that, The 3D-CNN model includes 7 convolutional layers, 5 pooling layers, and 2 fully connected layers arranged sequentially from the input to the output. The last fully connected layer has 2048 neurons.
8. The method according to claim 1, characterized in that, The design process of the loss function includes: Define the weighting matrix The weighted matrix represents the similarity between sentiment classes that represent higher-order sentiment features of speech information. Defined by the following formula: (2) in, Weighted matrix It is 3D matrix C The total number of emotional state categories; Representation pattern class and The cosine similarity of the average vectors of high-order emotional feature samples of speech information; in, Representation pattern class The average vector is defined as: (3) in, Representation pattern class The A speech sample vector, m Representation pattern class The number of samples in the sample; Inter-class scatter matrix defining higher-order sentiment features of speech information for: (4) in, T To represent the transpose of a matrix, The pattern class is The total number of samples, The overall average vector representing the higher-order emotional features of speech information; Define a pattern class and The degree of dispersion between for: (5) Using weighted matrices between-class scatter matrix Optimization yields a new inter-class scatter matrix. Define a new inter-class scatter matrix. for: (6) Define the weighting matrix The weighted matrix represents the similarity between high-order emotion feature classes of facial expression images. Defined by the following formula: (7) in, Weighted matrix It is 3D matrix; Representation pattern class and The cosine similarity of the average vectors of high-order emotion feature samples from facial expression images; Representation pattern class The average vector is defined as: (8) in, Representation pattern class The One facial expression image sample vector; Based on equation (5), a high-order emotion feature pattern class for facial expression images is defined. With pattern classes Inter-class scatter matrix for: (9) Using weighted matrices between-class scatter matrix Optimization yields a new inter-class scatter matrix. Define a new inter-class scatter matrix. for: (10)。 9. The method according to claim 8, characterized in that, The MLP model has 4096 neurons in the first layer, 1024 neurons in the second layer, and C neurons in the last layer; the ReLU activation function is selected for the first two layers, and the softmax function is selected for the last layer. The design process of the loss function also includes: The objective function for the C classification task is defined as follows: (11) in, Represents the parameters of the network. Indicates the number of training samples. This represents training samples of speech and facial expression images. Indicates a label, This represents the characteristics input to the softmax function in an MLP. This represents the softmax function. The cross-entropy loss function is defined as follows: (12) in, Represented as in equation (11) , Indicates the first i The first sample data j Class tags, Indicates the first i The first sample data j The output of the softmax layer of the class; The optimization formula is defined as follows: (13) in, Represents the trace of a matrix; Combining equations (11)-(13), we define a C-classification objective function that can increase the inter-class scatter matrix: (14) in, It is a coefficient; Based on equation (14), the new loss function is defined as: (15) C-classification training is completed in an end-to-end manner using Equation (15).
10. A multimodal emotion recognition system for voice and facial expressions of personnel in smart buildings, characterized in that, include: The system includes a field data acquisition module, a communication module, a main control module, and an emotion display module. The on-site data acquisition module is used to collect voice information and facial expression video data of people inside the building; The communication module is used to upload the data collected by the field data acquisition module to the main control module; The main control module is used for: The collected voice information and facial expression video are preprocessed, including: extracting MFCC features from the voice information, dividing the facial expression video into frames of facial expression images and extracting facial expression regions from the facial expression images; The extracted MFCC features are input into a residual network model to extract high-order sentiment features of the speech information; The framed facial expression images and the extracted facial expression regions are input into a 3D-CNN model to extract high-order emotional features of the facial expression images; Multimodal fusion of high-order emotional features from the speech information and facial expression images is performed using an MLP model to identify the emotional state of a person. During the multimodal fusion process, a pre-designed loss function is used to complete end-to-end training of the speech-facial expression image multimodal feature fusion, thereby increasing the difference between the high-order emotional features of the speech information and facial expression images. The loss function is as follows: Where k is the total number of training samples, and C is the total number of emotion state categories. This represents the model's predicted output vector for the i-th training sample. This represents the true sentiment category label of the i-th training sample. Indicates the first i The first sample data j Class tags, This represents the probability that the model predicts the i-th sample data belongs to the j-th class. It is a coefficient. Represents the trace of a matrix. This represents the inter-class scatter matrix of high-order sentiment features of speech information. The inter-class scatter matrix represents the high-order emotional features of facial expression images; The emotion display module is used to display the emotional state of the person identified by the main control module.