Emotion recognition device based on multi-modal convolutional neural network

Abstract

The invention relates to the field of intelligent recognition, and particularly discloses an emotion recognition device based on a multi-modal convolutional neural network. Carrying out convolutional coding on the facial expression image by adopting an image analysis and signal processing technology based on a deep convolutional neural network model, and carrying out semantic coding on the facial expression image; therefore, the emotion type is automatically judged according to the sparse interaction fusion feature between the facial expression image convolution coding feature and the electroencephalogram signal waveform semantic feature. Therefore, emotional states can be captured more comprehensively and features which are crucial to emotion recognition are highlighted, so that the accuracy of emotion recognition is improved, meanwhile, multi-modal recognition can better adapt to emotion expressions under different individuals and situations, and better generalization ability is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent recognition, and more specifically, to an emotion recognition device based on a multimodal convolutional neural network. Background Art

[0002] As one of the core technologies of advanced human-computer interaction, emotion recognition computing has attracted much attention with the development of deep learning and artificial intelligence. Emotional expression can be divided into two categories: physiological signals (such as EEG and ECG) and non-physiological signals (such as language and facial expressions). Facial expressions are the most intuitive way to express emotions, and EEG signals are closely related to emotions.

[0003] However, traditional sentiment analysis methods often focus on a single data modality, such as relying solely on facial expressions or EEG signals to judge emotional states. This approach has some limitations and shortcomings. Specifically, single-modality analysis may not fully utilize all available information, resulting in low recognition accuracy. Although facial expressions are a direct manifestation of emotions, they may be intentionally concealed or invisible due to occlusion, and are also affected by individual differences and cultural backgrounds. Although EEG signals can reflect the inner emotional state, it is difficult to capture complete emotional information when used alone. In addition, single-modality analysis is easily interfered by noise, such as background sound in speech or facial occlusion, which makes the analysis results unreliable. Therefore, an emotion recognition device based on a multimodal convolutional neural network is desired. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an emotion recognition device based on a multimodal convolutional neural network, which obtains the facial expression image and EEG signal of the object under test, and uses image analysis and signal processing technology based on a deep convolutional neural network model to perform convolution encoding on the facial expression image, and semantic encoding on the facial expression image, so as to automatically judge the emotion type according to the sparse interactive fusion features between the convolution encoding features of the facial expression image and the semantic features of the EEG signal waveform. In this way, by fusing the information of the two modes of facial expression images and EEG signals, the emotional state can be captured more comprehensively and those features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations, and has better generalization ability.

[0005] According to one aspect of the present application, there is provided an emotion recognition device based on a multimodal convolutional neural network, comprising: A facial expression and EEG signal acquisition module is used to acquire facial expression images and EEG signals of the subject under test; A facial expression EEG signal encoding module is used to perform convolution encoding on the facial expression image and the EEG signal to obtain facial expression convolution encoding features and EEG signal waveform semantic features; A facial expression-EEG signal fusion module is used to perform multimodal feature significant joint interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain facial expression-EEG signal significant sparse interactive fusion features, wherein the facial expression-EEG signal fusion module includes: a facial expression-EEG signal feature encoding unit, used to perform core feature association and global encoding on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal kernel feature global encoding matrix; a facial expression-EEG signal interactive fusion unit, used to perform position-by-position association interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal position-by-position interactive fusion feature map; a facial expression-EEG signal sparse mask unit, used to perform sparse mask significant fusion on the facial expression-EEG signal kernel feature global encoding matrix and the facial expression-EEG signal position-by-position interactive fusion feature map to obtain a facial expression-EEG signal significant sparse interactive fusion feature; The recognition result generation module is used to obtain the recognition result based on the facial expression-EEG signal significant sparse interaction fusion feature, and the recognition result is used to represent the emotion type label.

[0006] Compared with the prior art, the present application provides an emotion recognition device based on a multimodal convolutional neural network, which obtains the facial expression image and EEG signal of the subject, and uses image analysis and signal processing technology based on a deep convolutional neural network model to convolutionally encode the facial expression image, and semantically encode the facial expression image, so as to automatically judge the emotion type according to the sparse interactive fusion features between the convolutional coding features of the facial expression image and the semantic features of the EEG signal waveform. In this way, by fusing the information of the two modes of facial expression images and EEG signals, the emotional state can be captured more comprehensively and those features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations, and has better generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 is a block diagram of an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application, Figure 2 is a schematic diagram of data flow of an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application, Figure 3 is a block diagram of a facial expression EEG signal encoding module in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application, Figure 4 is a block diagram of a facial expression-EEG signal fusion module in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application, Figure 5 is a block diagram of a facial expression-EEG signal feature encoding unit in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application, Figure 6 The present invention is a block diagram of a facial expression-EEG signal sparse masking unit in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. DETAILED DESCRIPTION

[0009] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0010] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0011] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0012] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0013] As one of the core technologies of advanced human-computer interaction, emotion recognition computing has attracted much attention with the development of deep learning and artificial intelligence. Emotional expression can be divided into two categories: physiological signals (such as EEG and ECG) and non-physiological signals (such as language and facial expressions). Facial expressions are the most intuitive way to express emotions, and EEG signals are closely related to emotions.

[0014] However, traditional sentiment analysis methods often focus on a single data modality, such as relying solely on facial expressions or EEG signals to judge emotional states. This approach has some limitations and shortcomings. Specifically, single-modality analysis may not fully utilize all available information, resulting in low recognition accuracy. Although facial expressions are a direct reflection of emotions, they may be intentionally concealed or invisible due to occlusion, and are also affected by individual differences and cultural backgrounds. Although EEG signals can reflect the inner emotional state, it is difficult to capture complete emotional information when used alone. In addition, single-modality analysis is easily affected by noise, such as background sound in speech or facial occlusion, which makes the analysis results unreliable.

[0015] Therefore, in response to the above technical problems, the technical concept of the present application is to obtain the facial expression image and EEG signal of the object under test, and use image analysis and signal processing technology based on a deep convolutional neural network model to perform convolution coding on the facial expression image, and perform semantic coding on the facial expression image, so as to automatically judge the emotion type according to the sparse interactive fusion features between the convolution coding features of the facial expression image and the semantic features of the EEG signal waveform. In this way, by fusing the information of the two modes of facial expression images and EEG signals, the emotional state can be captured more comprehensively and those features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations, and has better generalization ability.

[0016] Figure 1 It is a block diagram of an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of data flow of an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the emotion recognition device 100 based on the multimodal convolutional neural network includes: a facial expression EEG signal acquisition module 110, which is used to obtain the facial expression image and EEG signal of the object under test; a facial expression EEG signal encoding module 120, which is used to perform convolution encoding on the facial expression image and the EEG signal respectively to obtain facial expression convolution encoding features and EEG signal waveform semantic features; a facial expression-EEG signal fusion module 130, which is used to perform multimodal feature significance joint interaction on the facial expression convolution encoding features and the EEG signal waveform semantic features to obtain facial expression-EEG signal significant sparse interaction fusion features; a recognition result generation module 140, which is used to obtain a recognition result based on the facial expression-EEG signal significant sparse interaction fusion features, and the recognition result is used to represent the emotion type label.

[0017] In an embodiment of the present application, the facial expression EEG signal acquisition module 110 is used to acquire facial expression images and EEG signals of the object under test. It should be understood that the facial expression image is one of the important ways for humans to express emotions and can provide intuitive emotional clues. The EEG signal reflects the electrical characteristics of brain activity. Emotional states can cause changes in activity in specific areas of the brain, and brain activity patterns related to emotions can be captured through electroencephalograms. Based on this, in order to comprehensively evaluate emotional states in order to better understand the emotional needs of the object under test, in the technical solution of the present application, facial expression images and EEG signals of the object under test are acquired.

[0018] Specifically, in the embodiment of the present application, the facial expression EEG signal encoding module 120 is used to perform convolution encoding on the facial expression image and the EEG signal respectively to obtain facial expression convolution encoding features and EEG signal waveform semantic features. Figure 3It is a block diagram of a facial expression EEG signal encoding module in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Specifically, in the embodiment of the present application, the facial expression EEG signal encoding module 120 includes: a facial expression feature extraction unit 121, which is used to input the facial expression image into the first convolution branch of the multimodal convolutional neural network model to obtain a facial expression convolutional encoding feature map as the facial expression convolutional encoding feature; an EEG signal feature extraction unit 122, which is used to input the EEG signal into the second convolution branch of the multimodal convolutional neural network model to obtain an EEG signal waveform semantic encoding feature map as the EEG signal waveform semantic feature. Accordingly, considering that the facial expression image contains a large amount of spatial information, such as the position and shape of the eyes, eyebrows, and mouth, as well as slight changes in other parts of the face. The multimodal convolutional neural network is a deep learning model specially designed to process and fuse multiple types of input data. Unlike traditional unimodal models (such as models that only process images or text), multimodal models are designed to process data of multiple modalities (such as images, text, signals, etc.) at the same time, so that the model can understand and analyze problems from multiple angles, improving its performance and generalization ability. Based on this, in order to capture and mine the local subtleties of facial expressions from facial expression images, in the technical solution of the present application, the facial expression image is input into the first convolution branch of the multimodal convolutional neural network model to obtain a facial expression convolutional encoding feature map. That is, the first convolution branch can capture and refine the complex facial expression patterns in facial expressions, so as to gradually abstract higher-level features and better characterize the different facial emotion features in facial expressions. At the same time, considering that the EEG signal, as an important physiological signal, contains intrinsic information about the emotional state, in order to effectively extract emotion-related features and thus improve the overall performance of emotion recognition, in the technical solution of the present application, the EEG signal is input into the second convolution branch of the multimodal convolutional neural network model to obtain a semantic coding feature map of the EEG signal waveform. In this way, the local patterns in the signal and their changes in the time dimension are captured to enhance the ability to recognize and capture emotion-related brain activities.

[0019] In an embodiment of the present application, the facial expression-EEG signal fusion module 130 is used to perform multimodal feature significance joint interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain facial expression-EEG signal significant sparse interaction fusion features. Figure 4 is a block diagram of a facial expression-EEG signal fusion module in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 4As shown, the facial expression-EEG signal fusion module 130 includes: a facial expression-EEG signal feature encoding unit 131, which is used to perform core feature association and global encoding on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal kernel feature global coding matrix; a facial expression-EEG signal interactive fusion unit 132, which is used to perform position-by-position association and interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal position-by-position interactive fusion feature map; a facial expression-EEG signal sparse mask unit 133, which is used to perform sparse mask significant fusion on the facial expression-EEG signal kernel feature global coding matrix and the facial expression-EEG signal position-by-position interactive fusion feature map to obtain a facial expression-EEG signal significant sparse interactive fusion feature. Furthermore, considering that emotion recognition not only depends on the information of a single modality, but also requires the integration of data from multiple modalities. The facial expression convolutional coding features and the EEG signal waveform semantic features provide the external manifestation and internal state information of emotional expression, respectively. However, not all features are equally important, and some of them may be noise or insignificant. Therefore, a method is needed to highlight those features that are crucial to emotion recognition and to effectively fuse and interact these two features. In the technical solution of the present application, the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map are subjected to multimodal feature significance joint interaction to obtain a facial expression-EEG signal significant sparse interaction fusion feature map as a facial expression-EEG signal significant sparse interaction fusion feature.

[0020] Specifically, first, the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map are subjected to core feature association and global coding to obtain a global coding matrix between facial expression-EEG signal kernel features, so as to find the implicit core correlation between facial expression and EEG signal features to capture the global pattern between different features. Specifically, first, the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map are subjected to extraction by retaining the original data (essential features) through a kernel feature distillation network while filtering out the influence of redundant information to obtain a facial expression kernel feature vector and an EEG signal waveform semantic kernel feature vector. Afterwards, the global correlation between the facial expression kernel feature vector and the EEG signal waveform semantic kernel feature vector is calculated to reveal the relationship between different features, which helps to identify hidden patterns in the data and lays the foundation for subsequent sparse processing.

[0021] At the same time, on another data processing link, in order to capture and depict subtle changes in facial expressions and EEG signals in time and space more carefully and meticulously, the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map are associated and interacted position by position to improve the model's sensitivity to subtle feature changes, thereby obtaining a facial expression-EEG signal position-by-position interactive fusion feature map.

[0022] Next, the global coding matrix between the facial expression-EEG signal kernel features and the facial expression-EEG signal position-by-position interactive fusion feature map are sparsely masked and significantly fused to emphasize the key features that best reflect the emotional state, while suppressing or ignoring those less important features, thereby enhancing the focus of the model and obtaining a facial expression-EEG signal significant sparse interactive fusion feature map. Specifically, in order to eliminate those minor connections or weights and emphasize those feature information that is crucial to information transmission, a sparse mask unit is used to perform hierarchical sparse masking on the global coding matrix between the facial expression-EEG signal kernel features to obtain a facial expression-EEG signal kernel feature-related sparse weight matrix, which can reduce invalid calculations to highlight important features and improve the model running speed. Finally, based on the associated sparse weight matrix as a weight matrix, the facial expression-EEG signal position-by-position interactive fusion feature map is subjected to feature-significant sparse modulation to adjust the contribution of each part according to the importance level indicated by the weight matrix, and the final facial expression-EEG signal significant sparse interactive fusion feature map is obtained.

[0023] Figure 5 : is a block diagram of a facial expression-EEG signal feature encoding unit in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 5 As shown, the facial expression-EEG signal feature encoding unit 131 includes: a facial expression-EEG signal kernel feature distillation subunit 1311, which is used to input the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map into a kernel feature distillation network to obtain a facial expression kernel feature vector and an EEG signal waveform semantic kernel feature vector; a facial expression-EEG signal global encoding subunit 1312, which is used to perform global association coding on the facial expression kernel feature vector and the EEG signal waveform semantic kernel feature vector to obtain the facial expression-EEG signal kernel feature global coding matrix.

[0024] Specifically, in an embodiment of the present application, the facial expression EEG signal kernel feature distillation subunit is used to: perform feature fine-grained decomposition on the facial expression convolutional coding feature map to obtain a set of facial expression feature vectors; calculate the feature difference coefficient of each facial expression feature vector in the set of facial expression feature vectors relative to other facial expression feature vectors to obtain a set of facial expression feature difference coefficients; normalize the set of facial expression feature difference coefficients to obtain a set of normalized facial expression feature difference coefficients; use each normalized facial expression feature difference coefficient in the set of normalized facial expression feature difference coefficients as a weight, and calculate the weighted sum of the set of facial expression feature vectors to obtain the facial expression kernel feature vector.

[0025] More specifically, in an embodiment of the present application, the facial expression convolution coding feature map is subjected to fine-grained feature decomposition to obtain a set of facial expression feature vectors, including: expanding each feature matrix of the facial expression convolution coding feature map along the channel dimension into feature vectors to obtain the set of facial expression feature vectors.

[0026] More specifically, in an embodiment of the present application, the characteristic difference coefficient of each facial expression feature vector in the set of facial expression feature vectors relative to other facial expression feature vectors is calculated to obtain a set of facial expression characteristic difference coefficients, including: extracting a predetermined facial expression feature vector from the set of facial expression feature vectors; subtracting the predetermined facial expression feature vector from other facial expression feature vectors in the set of facial expression feature vectors by position, and calculating a norm of each facial expression feature difference vector in the obtained set of facial expression feature difference vectors to obtain a set of facial expression feature norm values; calculating the global mean of the set of facial expression feature norm values ​​to obtain the facial expression feature difference coefficient.

[0027] More specifically, in an embodiment of the present application, the set of facial expression feature difference coefficients is normalized to obtain a set of normalized facial expression feature difference coefficients, including: calculating an exponential function with a natural constant e as the base and the negative of each facial expression feature difference coefficient in the set of facial expression feature difference coefficients as the exponent to obtain a set of facial expression feature difference indices; calculating the position-by-position summation of the set of facial expression feature difference indices to obtain a total sum value of facial expression feature difference indices; and dividing the set of facial expression feature difference indices by the total sum value of facial expression feature difference indices by position to obtain the set of normalized facial expression feature difference coefficients.

[0028] More specifically, in an embodiment of the present application, the facial expression EEG signal global encoding subunit is used to: use the matrix obtained by multiplying the facial expression kernel feature vector with the transposed vector of the EEG signal waveform semantic kernel feature vector as the global encoding matrix between the facial expression-EEG signal kernel features.

[0029] Specifically, in an embodiment of the present application, the facial expression-EEG signal interactive fusion unit 132 is used to: perform positional multiplication of each feature matrix along the channel dimension in the facial expression convolutional coding feature map with the feature values ​​of the corresponding positions in each feature matrix along the channel dimension in the EEG signal waveform semantic coding feature map to obtain the facial expression-EEG signal position-by-position interactive fusion feature map.

[0030] Figure 6 is a block diagram of a facial expression-EEG signal sparse masking unit in an emotion recognition device based on a multimodal convolutional neural network according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 6 As shown, the facial expression-EEG signal sparse mask unit 133 includes: a facial expression-EEG signal associated sparse weight mask subunit 1331, which is used to input the global coding matrix between the facial expression-EEG signal kernel features into the sparse mask unit to obtain the facial expression-EEG signal kernel feature associated sparse weight matrix; a facial expression-EEG signal associated interaction subunit 1332, which is used to perform feature significant sparse modulation on the facial expression-EEG signal position-by-position interactive fusion feature map based on the facial expression-EEG signal kernel feature associated sparse weight matrix to obtain the facial expression-EEG signal significant sparse interactive fusion feature map as the facial expression-EEG signal significant sparse interactive fusion feature.

[0031] More specifically, in an embodiment of the present application, the facial expression-EEG signal associated sparse weight mask subunit 1331 is used to: compare the eigenvalues ​​of each position in the global coding matrix between the facial expression-EEG signal kernel features with a preset threshold to obtain the facial expression-EEG signal kernel feature associated sparse weight matrix composed of multiple updated eigenvalues; wherein, in response to the eigenvalue being greater than a second preset threshold, the product of the eigenvalue and a constant three is used as the updated eigenvalue; in response to the eigenvalue being greater than a first preset threshold and less than or equal to the second preset threshold, the original value of the eigenvalue is used as the updated eigenvalue; in response to the eigenvalue being less than or equal to the first preset threshold, the value obtained by dividing the eigenvalue by the constant three is used as the updated eigenvalue.

[0032] More specifically, in an embodiment of the present application, the facial expression-EEG signal association interaction subunit 1332 is used to: matrix multiply each feature matrix along the channel dimension in the facial expression-EEG signal position-by-position interaction fusion feature map with the sparse weight matrix associated with the facial expression-EEG signal kernel features to obtain the facial expression-EEG signal significant sparse interaction fusion feature map.

[0033] In the embodiment of the present application, specifically, the multimodal feature significance joint interaction is performed on the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map, which can be expressed by the formula: , , , , , , in, and are respectively the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map, For the feature map Perform kernel feature distillation. and are the facial expression kernel feature vector and the EEG signal waveform semantic kernel feature vector, respectively. For the feature map Perform feature fine-grained decomposition operations. is the first, second, ..., first eigenvector in the set of decomposed eigenvectors , ..., , the After decomposition, the eigenvectors represents the one-norm of the eigenvector, is the set of decomposed eigenvectors, represents the number of vectors in the set of decomposed eigenvectors minus one, yes The corresponding characteristic difference coefficient, Expressed as a natural constant The exponential function with base , is the number of vectors in the set of decomposed eigenvectors, is the kernel eigenvector, yes The transposed vector of is matrix multiplication, is the global encoding matrix between the facial expression and EEG signal kernel features, is a sparse masking operation, yes The eigenvalues ​​at each position in , is the preset threshold, is the sparse weight matrix of the correlation between the facial expression and the EEG signal kernel features, For point-by-position multiplication, is the facial expression-EEG signal position-by-position interactive fusion feature map, It is the facial expression-EEG signal significant sparse interaction fusion feature map.

[0034] In an embodiment of the present application, the recognition result generation module 140 is used to obtain a recognition result based on the facial expression-EEG signal significant sparse interaction fusion feature, and the recognition result is used to represent the emotion type label. Specifically, in an embodiment of the present application, the recognition result generation module is used to: input the facial expression-EEG signal significant sparse interaction fusion feature map into the classifier-based emotion recognition module to obtain the recognition result, and the recognition result is used to represent the emotion type label. That is, the facial expression-EEG signal significant sparse interaction fusion feature obtained by the significant sparse interaction of the facial expression convolution coding feature map and the EEG signal waveform semantic coding feature map is used for classification processing to automatically determine the emotion type. In this way, by fusing the information of the two modalities of facial expression image and EEG signal, the emotional state can be captured more comprehensively and the features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations, and has better generalization ability.

[0035] It should be understandable that, considering that the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map respectively represent the facial expression image semantic coding features of the facial expression image and the signal waveform semantic coding features of the EEG signal, when they are input into the feature significant joint interaction coding module based on kernel-correlated sparse mask, the joint interaction between the image semantic features and signal waveform semantic features of different modalities will have the kernel-correlated sparse mask perceptual difference. Therefore, it is expected to improve the complex spatial structure regression convergence and generalization effect of the facial expression-EEG signal significant sparse interaction fusion feature map based on the joint perceptual interaction of different spatial distribution structures.

[0036] Preferably, inputting the facial expression-EEG signal significant sparse interaction fusion feature map into a classifier-based emotion recognition module to obtain a recognition result comprises: Calculate the absolute value sum of all eigenvalues ​​of the facial expression-EEG signal significant sparse interaction fusion feature map and the square root of the sum of squares to obtain the first facial expression-EEG signal significant sparse interaction fusion spatial structure value and the second facial expression-EEG signal significant sparse interaction fusion spatial structure value, that is: , , in, The first one represents the facial expression-EEG signal significant sparse interaction fusion feature map eigenvalues, Indicates the spatial structure value of the first facial expression-EEG signal significant sparse interaction fusion, Indicates the spatial structure value of the significant sparse interaction fusion of the second facial expression and EEG signal; Determine the total number of eigenvalues ​​of all eigenvalues ​​of the set of the facial expression-EEG signal significant sparse interaction fusion feature map ; For each eigenvalue of the facial expression-EEG signal significant sparse interaction fusion feature map, calculate the first facial expression-EEG signal significant sparse interaction fusion spatial structure value minus the product of the eigenvalue and the total number of eigenvalues ​​to obtain the first facial expression-EEG signal significant sparse interaction fusion long-range dependency value ,in, Indicates the spatial structure value of the first facial expression-EEG signal significant sparse interaction fusion, The first one represents the facial expression-EEG signal significant sparse interaction fusion feature map eigenvalues, The number of eigenvalues ​​representing all eigenvalues ​​of the facial expression-EEG signal significant sparse interaction fusion feature map, The first facial expression-EEG signal significant sparse interaction fusion long-range dependency value is represented, and the second facial expression-EEG signal significant sparse interaction fusion long-range dependency value is obtained by multiplying the square root of the total number of eigenvalues ​​by the product of the eigenvalues ​​minus the second facial expression-EEG signal significant sparse interaction fusion spatial structure value. ,in, Indicates the spatial structure value of the significant sparse interaction fusion of the second facial expression and EEG signal, represents the second facial expression-EEG signal significant sparse interaction fusion long-range dependency value; taking the first facial expression-EEG signal significant sparse interaction fusion long-range dependency value as the exponent of the natural constant and the inverse of the second facial expression-EEG signal significant sparse interaction fusion long-range dependency value, weighted summing them to obtain the optimized eigenvalue, , in, represents a natural constant, and represents the weighted hyperparameter, Representing the optimized eigenvalue corresponding to each eigenvalue; inputting the optimized facial expression-EEG signal significant sparse interactive fusion feature map composed of the optimized eigenvalues ​​into the classifier-based emotion recognition module to obtain the recognition result.

[0037] Based on this, the feature set of the facial expression-EEG signal significant sparse interaction fusion feature map may lack spatial structure consistency in high-dimensional space, which leads to the weight matrix of the classifier implicitly inferring spatial structure information based on the feature, resulting in inconsistent convergence. By establishing a long-distance feature dependency relationship based on the overall feature scale of the facial expression-EEG signal significant sparse interaction fusion feature map relative to the spatial structure representation of the facial expression-EEG signal significant sparse interaction fusion feature map, the feature local connectivity of the facial expression-EEG signal significant sparse interaction fusion feature map is established, and the spatial ambiguous information of the object feature value is captured by predicting the unstructured feature value points of the facial expression-EEG signal significant sparse interaction fusion feature map, thereby improving the convergence and generalization effect of the classification regression of the feature set of the facial expression-EEG signal significant sparse interaction fusion feature map under the complex spatial structure representation, and improving the accuracy of the recognition result obtained by inputting it into the emotion recognition module based on the classifier. In this way, by fusing the information of the two modalities of facial expression image and EEG signal, the emotional state can be captured more comprehensively and the features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations and has better generalization ability.

[0038] In summary, the emotion recognition device 100 based on the multimodal convolutional neural network according to the embodiment of the present application is explained, which obtains the facial expression image and EEG signal of the object under test, and uses image analysis and signal processing technology based on a deep convolutional neural network model to perform convolution encoding on the facial expression image, and semantic encoding on the facial expression image, so as to automatically judge the emotion type according to the sparse interactive fusion features between the convolution encoding features of the facial expression image and the semantic features of the EEG signal waveform. In this way, by fusing the information of the two modes of facial expression images and EEG signals, the emotional state can be captured more comprehensively and those features that are crucial to emotion recognition can be highlighted, thereby improving the accuracy of emotion recognition. At the same time, multimodal recognition can better adapt to the emotional expression of different individuals and situations, and has better generalization ability.

[0039] As described above, the emotion recognition device 100 based on a multimodal convolutional neural network according to the embodiment of the present application can be implemented in various wireless terminals, such as a server having an emotion recognition algorithm based on a multimodal convolutional neural network. In one possible implementation, the emotion recognition device 100 based on a multimodal convolutional neural network according to the embodiment of the present application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the emotion recognition device 100 based on a multimodal convolutional neural network can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the emotion recognition device 100 based on a multimodal convolutional neural network can also be one of the many hardware modules of the wireless terminal.

[0040] Alternatively, in another example, the emotion recognition device 100 based on a multimodal convolutional neural network and the wireless terminal may also be separate devices, and the emotion recognition device 100 based on a multimodal convolutional neural network may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0041] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. The disclosure is not limited to the disclosed implementations, and many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terms used herein are selected to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various practical methods disclosed herein.

Claims

1. An emotion recognition device based on a multimodal convolutional neural network, characterized in that: include: A facial expression and EEG signal acquisition module is used to acquire facial expression images and EEG signals of the subject under test; A facial expression EEG signal encoding module is used to perform convolution encoding on the facial expression image and the EEG signal to obtain facial expression convolution encoding features and EEG signal waveform semantic features; A facial expression-EEG signal fusion module is used to perform multimodal feature significant joint interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain facial expression-EEG signal significant sparse interactive fusion features, wherein the facial expression-EEG signal fusion module includes: a facial expression-EEG signal feature encoding unit, used to perform core feature association and global encoding on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal kernel feature global encoding matrix; a facial expression-EEG signal interactive fusion unit, used to perform position-by-position association interaction on the facial expression convolutional coding features and the EEG signal waveform semantic features to obtain a facial expression-EEG signal position-by-position interactive fusion feature map; a facial expression-EEG signal sparse mask unit, used to perform sparse mask significant fusion on the facial expression-EEG signal kernel feature global encoding matrix and the facial expression-EEG signal position-by-position interactive fusion feature map to obtain a facial expression-EEG signal significant sparse interactive fusion feature; The recognition result generation module is used to obtain the recognition result based on the facial expression-EEG signal significant sparse interaction fusion feature, and the recognition result is used to represent the emotion type label.

2. The emotion recognition device based on a multimodal convolutional neural network according to claim 1, characterized in that: The facial expression EEG signal encoding module comprises: A facial expression feature extraction unit, used for inputting the facial expression image into the first convolution branch of the multimodal convolutional neural network model to obtain a facial expression convolutional coding feature map as the facial expression convolutional coding feature; The EEG signal feature extraction unit is used to input the EEG signal into the second convolution branch of the multimodal convolutional neural network model to obtain an EEG signal waveform semantic encoding feature map as the EEG signal waveform semantic feature.

3. The emotion recognition device based on a multimodal convolutional neural network according to claim 2, characterized in that: The facial expression-EEG signal feature encoding unit comprises: A facial expression EEG signal kernel feature distillation subunit is used to input the facial expression convolutional coding feature map and the EEG signal waveform semantic coding feature map into a kernel feature distillation network to obtain a facial expression kernel feature vector and an EEG signal waveform semantic kernel feature vector; The facial expression and EEG signal global encoding subunit is used to perform global correlation encoding on the facial expression kernel feature vector and the EEG signal waveform semantic kernel feature vector to obtain the global encoding matrix between the facial expression and EEG signal kernel features.

4. The emotion recognition device based on a multimodal convolutional neural network according to claim 3, characterized in that: The facial expression EEG signal kernel feature distillation subunit is used to: Performing feature fine-grained decomposition on the facial expression convolutional coding feature map to obtain a set of facial expression feature vectors; Calculating a characteristic difference coefficient of each facial expression feature vector in the set of facial expression feature vectors relative to other facial expression feature vectors to obtain a set of facial expression characteristic difference coefficients; Normalizing the set of facial expression feature difference coefficients to obtain a set of normalized facial expression feature difference coefficients; The weighted sum of the set of facial expression feature vectors is calculated using each normalized facial expression feature difference coefficient in the set of normalized facial expression feature difference coefficients as a weight to obtain the facial expression kernel feature vector.

5. The emotion recognition device based on a multimodal convolutional neural network according to claim 4, characterized in that: The facial expression EEG signal global encoding subunit is used to: use the matrix obtained by multiplying the facial expression kernel feature vector with the transposed vector of the EEG signal waveform semantic kernel feature vector as the facial expression-EEG signal kernel feature global encoding matrix.

6. The emotion recognition device based on a multimodal convolutional neural network according to claim 5, characterized in that: The facial expression-EEG signal interactive fusion unit is used to: perform position-wise multiplication of the feature matrices along the channel dimension in the facial expression convolutional coding feature map with the feature values ​​of the corresponding positions in the feature matrices along the channel dimension in the EEG signal waveform semantic coding feature map to obtain the facial expression-EEG signal position-by-position interactive fusion feature map.

7. The emotion recognition device based on a multimodal convolutional neural network according to claim 6, characterized in that: The facial expression-EEG signal sparse mask unit comprises: A facial expression-EEG signal associated sparse weight mask subunit, used for inputting the facial expression-EEG signal kernel feature global encoding matrix into the sparse mask unit to obtain a facial expression-EEG signal kernel feature associated sparse weight matrix; The facial expression-EEG signal correlation interaction subunit is used to perform feature significant sparse modulation on the facial expression-EEG signal position-by-position interaction fusion feature map based on the correlation sparse weight matrix between the facial expression-EEG signal kernel features to obtain the facial expression-EEG signal significant sparse interaction fusion feature map as the facial expression-EEG signal significant sparse interaction fusion feature.

8. The emotion recognition device based on a multimodal convolutional neural network according to claim 7, characterized in that: The facial expression EEG signal associated sparse weight mask subunit is used to: Compare the eigenvalues ​​of each position in the global encoding matrix between the facial expression and EEG signal kernel features with a preset threshold to obtain a sparse weight matrix of the facial expression and EEG signal kernel feature associations composed of a plurality of updated eigenvalues; wherein, in response to the eigenvalue being greater than a second preset threshold, the product of the eigenvalue and a constant three is used as the updated eigenvalue; In response to the feature value being greater than a first preset threshold and less than or equal to the second preset threshold, taking the original value of the feature value as the updated feature value; In response to the eigenvalue being less than or equal to the first preset threshold, a value obtained by dividing the eigenvalue by constant three is used as the updated eigenvalue.

9. The emotion recognition device based on a multimodal convolutional neural network according to claim 8, characterized in that: The facial expression-EEG signal correlation interaction subunit is used to: perform matrix multiplication of each feature matrix along the channel dimension in the facial expression-EEG signal position-by-position interaction fusion feature map with the facial expression-EEG signal kernel feature correlation sparse weight matrix to obtain the facial expression-EEG signal significant sparse interaction fusion feature map.

10. The emotion recognition device based on a multimodal convolutional neural network according to claim 9, characterized in that: The recognition result generation module is used to: input the facial expression-EEG signal significant sparse interaction fusion feature map into a classifier-based emotion recognition module to obtain the recognition result, and the recognition result is used to represent the emotion type label.

Images