Multi-mode prelingual emotion recognition system and method for deaf people

Abstract

The invention relates to a multi-modal prelingual emotion recognition system for a deaf person, and the system comprises an information collection and preprocessing unit, a functional brain network emotion recognition unit, a facial expression recognition unit, and an optimal decision fusion unit. The emotion of a deaf person subject is predicted through the methods of expression and electroencephalogram signal collection, signal preprocessing and calculation, facial area expression image processing, facial expression recognition, functional brain network emotion recognition and optimal decision fusion. Through fusion of a functional brain network and facial expression recognition, complex dynamic characteristics in electroencephalogram signals of the deaf people can be deeply analyzed, subtle emotional changes of the deaf people before speech are captured, the recognition accuracy and robustness are improved through an algorithm of adaptively adjusting the weight of each mode, and the method has relatively high precision and adaptability, and is suitable for popularization and application. And a more effective technical support is provided for emotion recognition of the deaf people.

Description

[Technical Field]

[0001] This invention belongs to the field of emotion recognition technology, specifically relating to a multimodal emotion recognition system and method for prelingual deaf individuals. [Background Technology]

[0002] Prelingual deaf individuals are those who are born without hearing or lose their hearing during infancy. Due to severe impairment of their auditory pathways, prelingual deaf individuals rely more heavily on nonverbal methods, such as facial expressions and body language, to express and understand emotions. Because of potential social isolation and communication barriers, the deaf community may experience more emotional and psychological challenges compared to the hearing community.

[0003] Emotions are a crucial factor in interpersonal communication and social interaction, and also an important indicator of human mental health. In recent years, with the rapid development of artificial intelligence technology and hardware computing devices, emotional intelligence systems such as virtual assistants and social robots have emerged, and the field of emotion recognition has received more attention. However, traditional emotion recognition methods that rely on a single modality are gradually reaching bottlenecks in terms of recognition accuracy and the ability to address individual differences in emotions. They also often have significant limitations, such as the inability to accurately identify hidden or disguised emotions.

[0004] Existing technologies disclose methods for emotion recognition that involve acquiring images of an individual's face and limbs, extracting features from all captured facial and limb images to obtain facial expression and movement features for all individuals in the dataset, and finally inputting the limb movement and facial expression features into an emotion classification model for training, ultimately outputting different emotion labels for each individual.

[0005] Existing technologies also disclose a method for emotion recognition that extracts differential entropy of different frequency bands in EEG signals as features through discrete wavelet transform and empirical mode decomposition, and then uses a multi-class support vector machine for three-class classification.

[0006] However, unable to rely on spoken language, deaf people rely more on facial expressions, gestures, and body language for emotional communication, which increases the complexity and difficulty of emotion recognition in deaf individuals. Existing emotion recognition methods for hearing people largely rely on the comprehensive analysis of information such as speech, tone of voice, and facial expressions. However, these methods have significant limitations when applied to the deaf population. Furthermore, prelingual deaf individuals may exhibit differences in emotional expression compared to hearing individuals, requiring more refined and personalized recognition models. To overcome the shortcomings of existing emotion classification models, this invention proposes a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial micro-expressions. This method is expected to provide effective technical support for a deeper understanding and accurate capture of the emotional states of prelingual deaf individuals. [Summary of the Invention]

[0007] The purpose of this invention is to provide a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions. It can overcome the shortcomings of existing technologies and is a targeted and easy-to-implement emotion recognition system and method.

[0008] The technical solution of this invention is a multimodal prelingual deaf emotion recognition system, characterized in that it includes an information acquisition and preprocessing unit, a functional brain network emotion recognition unit, a facial expression recognition unit, and an optimal decision fusion unit; wherein, the input end of the information acquisition and preprocessing unit acquires facial expression images and EEG signals of the deaf subject, and its output end is connected to the input ends of the functional brain network emotion recognition unit and the facial expression recognition unit respectively; the input end of the optimal decision fusion unit receives the output signals of the functional brain network emotion recognition unit and the facial expression recognition unit respectively, and its output end outputs the emotion recognition result of the deaf subject.

[0009] The information acquisition and preprocessing unit consists of a camera module, an EEG acquisition module, and a preprocessing submodule. The camera module is used to capture facial expression images of deaf subjects, and its output is connected to the input of the preprocessing submodule. The EEG acquisition module is used to record the EEG signals of deaf subjects, and its output is connected to the input of the preprocessing submodule. The preprocessing submodule is used to perform key point localization on the acquired facial images and to filter, segment, and perform baseline correction and artifact removal on the acquired EEG signals. Its output is connected to the input of the functional brain network emotion recognition unit and the facial expression recognition unit, respectively.

[0010] The EEG acquisition module consists of an EEG sensor and a signal amplifier; the EEG sensor is used to acquire the EEG signals of deaf subjects; the input of the signal amplifier is connected to the EEG sensor, and its output is connected to the input of the preprocessing submodule.

[0011] The functional brain network emotion recognition unit consists of an EEG feature extraction module, a functional brain network construction module, and an emotion recognition module. The input end of the EEG feature extraction module is connected to the output end of the information acquisition and preprocessing unit, and extracts key features of frequency, amplitude, and phase from the preprocessed EEG signals of the deaf subject. Its output end is connected to the input end of the functional brain network construction module. The functional brain network construction module is used to construct a functional brain network model based on the EEG signal features extracted by the EEG feature extraction module to reflect the connection relationship and information flow between brain regions. The emotion recognition optimal decision fusion unit is used to output the prelingual deaf person's emotion classification label. Its input end is the recognition results output by the unconstructed functional brain network model and the facial expression recognition model, respectively. Its output end outputs the final prediction result of emotion recognition.

[0012] The facial expression recognition unit selects facial features such as the degree of opening and closing of the eyes and mouth, and the slant of the eyebrows as emotional representations of different subjects for further emotion recognition. The optimal decision fusion unit proposes an improved decision algorithm that assigns weights based on the credibility of each modality. By assigning higher weights to modalities with high credibility and lower weights to modalities with low credibility, the algorithm significantly improves the accuracy and robustness of the fusion results.

[0013] A multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, characterized by the following steps:

[0014] (1) Collect facial expression images and EEG signals of deaf subjects and send them to the information acquisition and preprocessing unit;

[0015] In step (1), facial expression images are captured using a high-speed camera to ensure that subtle changes in micro-expressions are recorded. Electroencephalogram (EEG) signals are simultaneously recorded using a Neuroscan 64-channel EEG acquisition device to obtain accurate signals of brain activity.

[0016] The determination of the sliding window size in the sliding window method takes into account the frequency characteristics of the EEG signal, the time scale of the emotional changes of interest, and the expected time resolution. In an exemplary embodiment of this application, the facial expression image acquisition uses the first frame of the image acquired per second as the corresponding deaf person's expression image for each second, and the EEG signal feature extraction is divided according to a 1-second non-overlapping sliding window.

[0017] (2) The functional brain network emotion recognition unit preprocesses and analyzes the EEG signals obtained in step (1). EEGLAB is used for EEG signal preprocessing, including removal of electrooculogram artifacts, removal of useless electrodes, filtering, and principal component analysis (PCA). The brain functional network characteristics, power spectral density characteristics, and differential entropy characteristics of the subject's EEG signals are calculated. The phase lag coefficient, mutual information, phase-locked value, Granger causality, and Pearson correlation coefficient are selected to construct a brain functional connectivity weighting matrix. Then, the preprocessed EEG signals are input into an improved classification model based on convolutional neural networks and long short-term memory networks to realize the recognition of emotions of deaf subjects by the functional brain network emotion recognition unit.

[0018] In step (2), the functional brain network signal feature fusion adopts feature-level fusion. To solve the problem that the time complexity of the classification model will increase exponentially due to the multidimensional information after feature fusion, and also to prevent sample overfitting and reduce classification time, this invention uses principal component analysis to reduce feature dimensionality, thereby ensuring that the decomposed principal components are mutually orthogonal and that the variance between data is maximized. The functional brain network emotion recognition unit defines 62 network nodes and covers 62 channels of the scalp. The phase-locked value (PLV) connection matrix construction method is used to express the phase-locked value synchronization relationship between two nodes as follows:

[0019]

[0020] φ ab (t)=φ a (t)-φ b (t)

[0021] Where < > represents the average time interval, and the instantaneous phase difference between nodes a(t) and b(t) at time t is denoted by φ. ab It means that φ a (t) and φ b (t) represents the instantaneous phase of the two points at that moment.

[0022] The structure diagram of the deaf brain network emotion classification model based on convolutional neural network and long short-term memory network in step (2) is as follows: Figure 3 As shown, the model consists of two convolutional layers, two pooling layers, a long short-term memory network, and two fully connected layers.

[0023] (3) Facial expression image processing. The facial expression image obtained in step (1) is adjusted to a size of 35×35 pixels, and the weights are updated using the VGGNet network based on the stochastic gradient descent algorithm. The eyebrows, eyes and mouth are selected as important key point areas. The tilt of the eyebrows and the degree of opening and closing of the eyes and mouth are used as facial emotion features of deaf subjects to train the classifier. An end-to-end facial key point deaf facial expression recognition model is established, and the classification results of the deaf facial expression recognition unit are output through the facial expression classifier.

[0024] In step (3), the facial expression classifier uses the Dlib face detection and recognition model to achieve key point localization and sample cropping. The network framework of the facial expression classifier includes four convolutional levels and three fully connected layers. Each convolutional level includes a max pooling layer and two convolutional blocks, which integrate the ReLU activation function, batch normalization layer, and convolutional layer. The first two fully connected layers are connected to the ReLU activation function, while the last fully connected layer is responsible for the final classification task. This invention uses the SoftMax activation function and introduces batch normalization to improve the learning efficiency of the neural network, thereby reducing internal covariance shift and effectively preventing gradient vanishing or gradient exploding.

[0025] (4) Based on the characteristics of prelingual deaf people who rely on lip reading and pay more attention to facial, especially mouth, emotional expression, we use modal weight enumeration method and multimodal fusion prediction arousal score evaluation index to adaptively adjust the weight of each modality, and update the current optimal weight distribution of each modality through cyclic enumeration algorithm.

[0026] In step (4), the performance index for adaptively adjusting the weights of each modality is the root mean square error. The optimal decision weight fusion achieves the minimum root mean square error when searching for the optimal weight distribution of each modality.

[0027] (5) After obtaining the optimal weight distribution of each modality, it is input into the optimal decision fusion unit, which performs optimal decision fusion on the prediction results of the facial expression recognition unit and the functional brain network recognition unit of the deaf subject, and outputs the final predicted emotion type of the deaf subject.

[0028] The advantages of this invention: This invention innovatively integrates functional brain networks and facial expression recognition technology in the field of deaf emotion recognition. Utilizing a model combining an improved convolutional neural network and a long short-term memory network, this invention can deeply analyze the complex dynamic characteristics of deaf people's electroencephalogram (EEG) signals and capture subtle pre-verbal emotional changes using micro-expression recognition technology that emphasizes lip reading. Furthermore, through an algorithm that adaptively adjusts the weights of each modality, this technical solution can dynamically optimize weight allocation according to the characteristics of deaf people's emotional expression, thereby significantly improving the accuracy and robustness of recognition. These optimizations in detail enable this invention to exhibit higher accuracy and adaptability in multimodal emotion recognition, providing more effective technical support for deaf emotion recognition. [Attached Image Description]

[0029] Figure 1 This is a system overall structure diagram of a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, which is involved in this invention.

[0030] Figure 2This is a block diagram of the functional brain network recognition unit of a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, which is part of this invention.

[0031] Figure 3 This is a schematic diagram of the emotion recognition process in a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, as described in this invention.

[0032] Figure 4 This is a schematic diagram of the facial key point localization of the subject's face in a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, as described in this invention.

[0033] Figure 5 This is a schematic diagram of the optimal weight distribution search algorithm for multimodal facial expression recognition in a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, as described in this invention.

Detailed Implementation Methods

[0034] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0035] This invention proposes a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, which includes:

[0036] (1) Collect facial expression images and EEG signals of deaf subjects and send them to the information acquisition and preprocessing unit;

[0037] In step (1), facial expression images are captured using a high-speed camera to ensure that subtle changes in micro-expressions are recorded. Electroencephalogram (EEG) signals are simultaneously recorded using a Neuroscan 64-channel EEG acquisition device to obtain accurate signals of brain activity.

[0038] The determination of the sliding window size in the sliding window method takes into account the frequency characteristics of the EEG signal, the time scale of the emotional changes of interest, and the expected time resolution. In an exemplary embodiment of this application, the facial expression image acquisition uses the first frame of the image acquired per second as the corresponding deaf person's expression image for each second, and the EEG signal feature extraction is divided according to a 1-second non-overlapping sliding window.

[0039] (2) The functional brain network emotion recognition unit preprocesses and analyzes the EEG signals obtained in step (1). EEGLAB is used for EEG signal preprocessing, including removal of electrooculogram artifacts, removal of useless electrodes, filtering, and principal component analysis (PCA). The brain functional network characteristics, power spectral density characteristics, and differential entropy characteristics of the subject's EEG signals are calculated. The phase lag coefficient, mutual information, phase-locked value, Granger causality, and Pearson correlation coefficient are selected to construct a brain functional connectivity weighting matrix. Then, the preprocessed EEG signals are input into an improved classification model based on convolutional neural networks and long short-term memory networks to realize the recognition of emotions of deaf subjects by the functional brain network emotion recognition unit.

[0040] In step (2), the feature fusion of functional brain network signals adopts feature-level fusion. In order to solve the problem that the time complexity of the classification model will increase exponentially after feature fusion due to multidimensional information, and also to prevent sample overfitting and reduce classification time, this invention uses principal component analysis to reduce feature dimensionality, thereby ensuring that the principal components after decomposition are mutually orthogonal and that the variance between data is maximized.

[0041] The structure diagram of the deaf brain network emotion classification model based on convolutional neural network and long short-term memory network in step (2) is as follows: Figure 3 As shown, the model consists of two convolutional layers, two pooling layers, a long short-term memory network, and two fully connected layers.

[0042] (3) Facial expression image processing. The facial expression image obtained in step (1) is adjusted to a size of 35×35 pixels, and the weights are updated using the VGGNet network based on the stochastic gradient descent algorithm. The eyebrows, eyes and mouth are selected as important key point areas. The tilt of the eyebrows and the degree of opening and closing of the eyes and mouth are used as facial emotion features of deaf subjects to train the classifier. An end-to-end facial key point deaf facial expression recognition model is established, and the classification results of the deaf facial expression recognition unit are output through the facial expression classifier.

[0043] In step (3), the facial expression classifier uses the Dlib face detection and recognition model to achieve key point localization and sample cropping. The network framework of the facial expression classifier includes four convolutional levels and three fully connected layers. Each convolutional level includes a max pooling layer and two convolutional blocks, which integrate the ReLU activation function, batch normalization layer, and convolutional layer. The first two fully connected layers are connected to the ReLU activation function, while the last fully connected layer is responsible for the final classification task. This invention uses the SoftMax activation function and introduces batch normalization to improve the learning efficiency of the neural network, thereby reducing internal covariance shift and effectively preventing gradient vanishing or gradient exploding.

[0044] (4) Based on the characteristics of prelingual deaf people who rely on lip reading and pay more attention to facial, especially mouth, emotional expression, we use modal weight enumeration method and multimodal fusion prediction arousal score evaluation index to adaptively adjust the weight of each modality, and update the current optimal weight distribution of each modality through cyclic enumeration algorithm.

[0045] In step (4), the performance index for adaptively adjusting the weights of each modality is the root mean square error. The optimal decision weight fusion is achieved by searching for the optimal weight distribution of each modality, where the root mean square error is minimized.

[0046] (5) After obtaining the optimal weight distribution of each modality, it is input into the optimal decision fusion unit, which performs optimal decision fusion on the prediction results of the facial expression recognition unit and the functional brain network recognition unit of the deaf subject, and outputs the final predicted emotion type of the deaf subject.

[0047] Figure 1 This is a system architecture diagram of a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions, as described in this invention. It illustrates the overall scheme for emotion recognition in prelingual deaf individuals. (See also...) Figure 2 The diagram shown is a block diagram of the functional brain network recognition unit of this invention. Brain signals are acquired using a Neuroscan 64-channel EEG signal acquisition device. Prelingual deaf adults whose hearing impairment occurred before language formation were selected as subjects. The acquired EEG signals were preprocessed to filter out noise data caused by power line interference, blinking, ECG, and EMG. The functional brain network emotion recognition unit defines 62 network nodes and covers 62 channels of the scalp. A brain functional connectivity weighted matrix is ​​constructed using phase lag coefficient, mutual information, phase-locked value, Granger causality, and Pearson correlation coefficient. The phase-locked value (PLV)-based connectivity matrix construction method represents the synchronization relationship between two nodes as follows:

[0048]

[0049] Where <> represents the average time interval, and φ represents the instantaneous phase difference between nodes a(t) and b(t) at time t. ab It means that φ a (t) and φ b (t) represents the instantaneous phase of the two points at that moment. The value of PLV is between 0 and 1. The closer it is to 1, the stronger the synchronization between the two points.

[0050] This invention utilizes clustering coefficients, node degree, node strength, local efficiency, and node betweenness centrality to evaluate the properties of the brain functional network for emotion recognition in deaf individuals, describing its randomness and global connectivity indices. It extracts brain functional network features, power spectral density features, and differential entropy features for feature fusion, thereby comprehensively utilizing frequency and spatial domain features to improve classifier recognition accuracy. The quality of features plays a crucial role in determining the model's generalization ability. If the extracted features contain too many irrelevant features, it will lead to feature redundancy, thus weakening the classifier's performance. To optimize the feature set and improve its utility, this invention employs Principal Component Analysis (PCA) algorithm, effectively reducing feature set redundancy.

[0051] The flowchart of emotion recognition in the functional brain network emotion recognition unit is as follows: Figure 3 As shown, based on the differences in changes exhibited by the constructed functional brain network, this invention precisely segments and trims EEG signals at points of event change. Subsequently, the carefully trimmed EEG signals are input into a classification model based on a convolutional neural network (CNN) and a long short-term memory network (LSTM) to achieve accurate emotion recognition. EEG features extracted at 1-second intervals are first fed into the CNN network, passing through two convolutional layers and pooling layers, then into the LSTM network for training, and finally through two fully connected layers to obtain the final emotion recognition result for the deaf subject. This process fully utilizes the dynamic characteristics of the brain network and improves the accuracy and reliability of emotion recognition through machine learning.

[0052] A schematic diagram of facial key point localization for deaf subjects in a facial expression recognition unit is shown below. Figure 4 As shown in the figure, this invention selects facial features such as the degree of opening and closing of the eyes and mouth, and the angle of the eyebrows to represent different emotions of the subjects. The calculation method for the corresponding features is as follows:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] k1 = polyfit[(x 18 ,x 19 ,x 20 ,x 21 ,x 22 ),(y 18 ,y 19 ,y20 ,y 21 ,y 22 ),1] (7)

[0059] k2 = polyfit[(x 23 ,x 24 ,x 25 ,x 26 ,x 27 ),(y 23 ,y 24 ,y 25 ,y 26 ,y 27 ),1] (8)

[0060] In this invention, facial expression features are mainly calculated using formulas (2)-(8), where d1 is the size of the mouth opening, i.e., the distance between the upper and lower lips, d2 and d3 represent the opening and closing states of the left and right eyes, θ1 and θ2 are the opening amplitudes of the left and right corners of the mouth, and k1 and k2 are the fitting slopes of the left and right eyebrows, respectively. To achieve facial expression feature extraction and recognition, this invention uses a deep convolutional neural network. On the one hand, an end-to-end network framework is constructed, and on the other hand, a VGGNet network based on the stochastic gradient descent algorithm is used to update the weights, thereby achieving high-precision recognition and classification of facial expressions of deaf subjects. The facial expression recognition network framework is built in a structure containing 4 convolutional levels and 3 fully connected layers. Each convolutional level includes a max pooling layer and two convolutional blocks. These convolutional blocks integrate the ReLU activation function, batch normalization layer, and convolutional layer to ensure efficient model operation. The first two fully connected layers are connected to the ReLU activation function, while the last fully connected layer is responsible for the final classification task. This invention employs the SoftMax activation function and introduces batch normalization to improve the learning efficiency of the neural network, thereby reducing internal covariance shift and effectively preventing gradient vanishing or exploding. The convolutional layer plays a crucial role in feature extraction, feature dimensionality reduction, and nonlinear processing.

[0061] To further optimize the recognition accuracy of the facial expression recognition unit, a batch of samples was generated using a portion of the training data. The weights and offsets of these samples were updated using stochastic gradient descent, as shown in formula (9):

[0062]

[0063] Where η is the learning efficiency, m is the number of batch samples generated, Δw is the weight strength, Δb is the offset gradient, the model is updated through backpropagation, and finally the input facial image is classified and recognized based on the extracted features.

[0064] Decision fusion units play a crucial role in the accuracy of multimodal prelingual deaf emotion recognition. After obtaining the emotion recognition results of each modality, this invention proposes a decision-level fusion-based emotion recognition method for prelingual deaf individuals, aiming to provide more accurate and comprehensive emotion recognition results for deaf subjects. Traditional decision-level fusion methods typically employ equal-weighted fusion, which, while considering the emotion recognition results of multiple modalities, neglects the credibility of each modality. Furthermore, traditional methods fail to fully consider the unique characteristics of prelingual deaf emotional expression, resulting in limited robustness improvement. To address these shortcomings, this invention proposes an improved decision algorithm that assigns weights based on the credibility of each modality. By assigning higher weights to modalities with high credibility and lower weights to modalities with low credibility, this algorithm significantly improves the accuracy and robustness of the fusion results. This method not only fully considers the credibility of each modality but also optimizes for the emotional expression characteristics of prelingual deaf individuals, thereby achieving higher robustness and accuracy in multimodal emotion recognition.

[0065] In a multimodal emotion recognition decision fusion unit, there are x modalities and x corresponding regression models. A total of H trials are conducted for prediction, where the average predicted arousal score of the i-th model in the h-th trial is A. hi i∈{1,2,3,…,x}, h∈{1,2,3,…,H}. Here, this invention sets a weight set ω{0.00,0.01,0.02,…,0.98,0.99,1.00} starting from 0.00 and ending at 1.00, with a step size of 0.01. The method for evaluating the performance of the current weight distribution is as follows: the root mean square error R is used as a metric. When each mode reaches the optimal weight distribution, the root mean square error R should be at its minimum value, denoted as R0. min The multimodal facial expression recognition optimal weight distribution search algorithm proposed in this invention mainly consists of three stages, the flowchart of which can be found in the attached diagram. Figure 5 As shown in the image.

[0066] In the first stage, the weights of x modalities in multimodal expression recognition are enumerated in the weight set. Let the weight of the i-th modality be ω. i The condition for entering the second stage is that the sum of all weights satisfies equation (10). When the loop enumeration ends, the optimal weight distribution will be saved and the algorithm will end.

[0067]

[0068] In the second stage, the algorithm calculates the predicted arousal score obtained from the emotion recognition fusion under the current multimodal weights. Assume the predicted arousal score in experiment h is... Its calculation method is shown in equation (11):

[0069]

[0070] In the third stage, the algorithm calculates the root mean square error R of the H trials under the current multimodal weights in the manner shown in equation (12), denoted as Rm. P , where y h The true wake-up score for experiment h. By comparing R... P and R min To evaluate the performance of the weight distribution in current multimodal emotion recognition models. When R... P ≥R min At that time, not R min Update. If R P <R min Then update R min For R P After the comparison is completed, return to the first phase and execute the task of enumerating the next set of weight distributions again.

[0071]

[0072] Through this search algorithm, the proposed scheme can accurately find the optimal weight distribution for each modality, thereby improving the accuracy and robustness of the multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions. After obtaining the optimal weight distribution for each modality, the final predicted emotion type of the deaf subject can be output by combining the current emotion recognition results of each modality. In summary, this invention proposes a multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions. This method combines the advantages of EEG signal and facial expression analysis to achieve accurate recognition of prelingual deaf emotions. By constructing a functional brain network model, extracting key features, and using deep convolutional neural networks for facial expression recognition, this invention effectively improves the accuracy and reliability of emotion recognition. Furthermore, by using a decision-level fusion algorithm to allocate weights according to the credibility of each modality, the accuracy and robustness of the fusion results are further improved.

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element. Additionally, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the stated number; "above," "below," "within," etc., are understood to include the stated number.

[0074] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multimodal pre-linguistic emotion recognition system for deaf individuals, characterized in that... It includes an information acquisition and preprocessing unit, a functional brain network emotion recognition unit, a facial expression recognition unit, and an optimal decision fusion unit. The information acquisition and preprocessing unit acquires facial expression images and EEG signals from the deaf subject at its input end, and its output end is connected to the input ends of the functional brain network emotion recognition unit and the facial expression recognition unit, respectively. The optimal decision fusion unit receives the output signals from the functional brain network emotion recognition unit and the facial expression recognition unit at its input end, and outputs the emotion recognition result of the deaf subject at its output end.

2. The multimodal pre-linguistic emotion recognition system for deaf people according to claim 1, characterized in that... The information acquisition and preprocessing unit consists of a camera module, an EEG acquisition module, and a preprocessing submodule; the camera module is used to capture facial expression images of deaf subjects, and its output is connected to the input of the preprocessing submodule. The EEG acquisition module is used to record the EEG signals of deaf subjects, and its output is connected to the input of the preprocessing submodule. The preprocessing submodule is used to perform key point localization on the acquired facial region image and to filter, segment, and perform baseline correction and artifact removal on the acquired EEG signals. Its output is connected to the input of the functional brain network emotion recognition unit and the facial expression recognition unit, respectively.

3. The multimodal prelingual emotion recognition system for deaf people according to claim 1, characterized in that... The EEG acquisition module consists of an EEG sensor and a signal amplifier; the EEG sensor is used to acquire the EEG signals of deaf subjects; the input of the signal amplifier is connected to the EEG sensor, and its output is connected to the input of the preprocessing submodule.

4. The multimodal pre-linguistic emotion recognition system for deaf people according to claim 1, characterized in that... The functional brain network emotion recognition unit consists of an EEG feature extraction module, a functional brain network construction module, and an emotion recognition module. The input end of the EEG feature extraction module is connected to the output end of the information acquisition and preprocessing unit, and extracts key features of frequency, amplitude, and phase from the preprocessed EEG signals of the deaf subject. Its output end is connected to the input end of the functional brain network construction module. The functional brain network construction module is used to construct a functional brain network model based on the EEG signal features extracted by the EEG feature extraction module to reflect the connection relationship and information flow between brain regions. The emotion recognition optimal decision fusion unit is used to output prelingual deaf emotion classification labels. Its input end is the recognition results output by the constructed functional brain network model and the facial expression recognition model, respectively, and its output end outputs the final prediction result of emotion recognition.

5. The multimodal prelingual deaf emotion recognition system according to claim 1, characterized in that... The facial expression recognition unit selects facial features such as the degree of opening and closing of the eyes and mouth and the slope of the eyebrows as emotional representations of different subjects for further emotion recognition; the optimal decision fusion unit proposes an improved decision algorithm that can assign weights according to the credibility of each modality; by giving higher weights to modalities with high credibility and lower weights to modalities with low credibility, the algorithm significantly improves the accuracy and robustness of the fusion results.

6. A multimodal prelingual emotion recognition method for deaf individuals based on functional brain networks and facial expressions, characterized in that... It includes the following steps: (1) Collect facial expression images and EEG signals of deaf subjects and send them to the information acquisition and preprocessing unit; (2) The functional brain network emotion recognition unit preprocesses and calculates the EEG signal obtained in step (1). EEGLAB is used to preprocess the EEG signal, including removal of electrooculogram artifacts, removal of useless electrodes, filtering and principal component analysis. The brain functional network characteristics, power spectral density characteristics, and differential entropy characteristics of the subject's EEG signals were calculated. The phase lag coefficient, mutual information, phase-locked value, Granger causality, and Pearson correlation coefficient were selected to construct a brain functional connectivity weighting matrix. Then, the EEG signals that had been preprocessed were input into an improved classification model based on convolutional neural networks and long short-term memory networks to enable the functional brain network emotion recognition unit to recognize the emotions of deaf subjects. (3) Facial expression image processing: The facial expression image obtained in step (1) is adjusted to 35×35 pixels in size, and the weights are updated using the VGGNet network based on the stochastic gradient descent algorithm; the eyebrows, eyes and mouth are selected as important key point areas, and the eyebrow tilt and the degree of opening and closing of the eyes and mouth are used as facial emotion features of deaf subjects to train the classifier, establish an end-to-end facial key point deaf facial expression recognition model, and output the classification results of the deaf facial expression recognition unit through the facial expression classifier; (4) Based on the characteristics of prelingual deaf people who rely on lip reading and pay more attention to facial, especially mouth, emotional expression, we use modal weight enumeration method and multimodal fusion prediction arousal score evaluation index to adaptively adjust the weight of each modality, and update the current optimal weight distribution of each modality through cyclic enumeration algorithm. (5) After obtaining the optimal weight distribution of each modality, it is input into the optimal decision fusion unit, which performs optimal decision fusion on the prediction results of the facial expression recognition unit and the functional brain network recognition unit of the deaf subject, and outputs the final predicted emotion type of the deaf subject.

7. The multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions according to claim 6, characterized in that... In step (1), facial expression images are captured by a high-speed camera to ensure that subtle changes in micro-expressions can be recorded. EEG signals are synchronously recorded using a Neuroscan 64-channel EEG signal acquisition device to obtain real signals of brain activity; The determination of the sliding window size in the sliding window method takes into account the frequency characteristics of the EEG signal, the time scale of the emotional changes of interest, and the expected time resolution. In an exemplary embodiment of this application, the facial expression image acquisition uses the first frame of the image acquired per second as the corresponding deaf person's expression image for each second, and the EEG signal feature extraction is divided according to a 1-second non-overlapping sliding window.

8. The multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions according to claim 6, characterized in that... In step (2) described above, the feature fusion of the functional brain network signal adopts feature-level fusion. To solve the problem that the time complexity of the classification model will increase exponentially due to the multidimensional information after feature fusion, and also to prevent sample overfitting and reduce classification time, this invention uses principal component analysis to reduce the dimensionality of features, thereby ensuring that the principal components after decomposition are mutually orthogonal and that the variance between data is maximized; the functional brain network emotion recognition unit defines 62 network nodes and covers 62 channels of the scalp; the phase-locked value synchronization relationship between two nodes is expressed as follows based on the phase-locked value connection matrix construction method: f ab (t)=φ a (t)-φ b (t) Where < > represents the average time interval, and the instantaneous phase difference between nodes a(t) and b(t) at time t is denoted by φ. ab It means that φ a (t) and φ b (t) represents the instantaneous phase between the two points at that moment; The emotion classification model of the deaf brain network based on convolutional neural network and long short-term memory network in step (2) consists of two convolutional layers, two pooling layers, long short-term memory network and two fully connected layers.

9. The multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions according to claim 6, characterized in that... In step (3), the facial expression classifier uses the Dlib face detection and recognition model to achieve key point localization and sample cropping. The network framework of the facial expression classifier includes four convolutional levels and three fully connected layers. Each convolutional level includes a max pooling layer and two convolutional blocks, which integrate the ReLU activation function, batch normalization layer, and convolutional layer. The first two fully connected layers are connected to the ReLU activation function, while the last fully connected layer is responsible for the final classification task. This invention uses the SoftMax activation function and introduces batch normalization operation to improve the learning efficiency of the neural network, thereby reducing internal covariance shift and effectively preventing gradient vanishing or gradient explosion.

10. The multimodal prelingual deaf emotion recognition method based on functional brain networks and facial expressions according to claim 6, characterized in that... In step (4), the performance index for adaptively adjusting the weights of each modality is the root mean square error. The optimal decision weight fusion achieves the minimum root mean square error when searching for the optimal weight distribution of each modality.

Images