Expression recognition method and device based on multi-modal data, edge equipment and medium

Abstract

The invention provides an expression recognition method and device based on multi-modal data, edge computing equipment and a medium, and the method comprises the steps: obtaining a to-be-recognized image and an electroencephalogram signal of a testee, the to-be-recognized image comprising a human face image of the testee, the electroencephalogram signals are acquired by electroencephalogram signal acquisition equipment when the testee is in different emotional states; preprocessing the to-be-recognized image to obtain a facial feature map; preprocessing the electroencephalogram signal to obtain an electroencephalogram feature map; performing data fusion on the facial feature map and the electroencephalogram feature map to obtain multi-modal fusion data; and performing expression recognition based on the facial feature map, the electroencephalogram feature map and the multi-modal fusion data. The method provided by the invention is helpful for improving the accuracy of expression recognition, capturing subtle expression differences, enhancing the reliability and stability of expression recognition, and better adapting to differences of different individuals.

Description

Technical Field

[0001] The present application relates to the field of expression recognition technology, and in particular to an expression recognition method, apparatus, edge computing device and medium based on multimodal data. Background Art

[0002] Expression recognition plays an important role in intelligent human-computer interaction. In related technologies, facial expression recognition is performed through image-based expression recognition algorithms. However, image-based expression recognition algorithms only rely on single-modality data (facial images) to judge expressions, and facial images are easily affected by factors such as lighting, angle, and occlusion, resulting in low accuracy and reliability of expression recognition. Summary of the invention

[0003] The present application provides a method, apparatus, edge computing device and medium for expression recognition based on multimodal data.

[0004] In a first aspect, the present application provides a method for facial expression recognition based on multimodal data, comprising:

[0005] Acquire an image to be identified and an EEG signal of a subject, wherein the image to be identified includes a facial image of the subject, and the EEG signal is acquired by an EEG signal acquisition device when the subject is in different emotional states;

[0006] Preprocess the image to be identified to obtain a facial feature map;

[0007] Preprocess the EEG signal to obtain an EEG characteristic map;

[0008] The facial feature map and the EEG feature map are fused to obtain multimodal fusion data;

[0009] Expression recognition is performed based on the facial feature map, the EEG feature map and the multimodal fusion data.

[0010] In one possible implementation, the image to be recognized is preprocessed to obtain a facial feature map, including:

[0011] Adjusting the brightness of the image to be identified according to a preset brightness threshold range to obtain a first image;

[0012] Detecting and cropping a facial image of a subject in the first image;

[0013] The face image of the subject is processed by size normalization, denoising and contrast enhancement in sequence to obtain a facial feature map.

[0014] In one possible implementation, the EEG signal is preprocessed to obtain an EEG characteristic graph, including:

[0015] Filtering the EEG signal;

[0016] Based on the EEG signals after filtering, the first, second and third cycles with the top three amplitude values ​​are extracted in the target area, and the amplitude values ​​are arranged from high to low;

[0017] The filtered EEG signal is evenly divided into a plurality of first subsequences, the length of the first subsequence is the length of the first period, and the plurality of first subsequences are arranged into a first two-dimensional tensor, each row in the first two-dimensional tensor corresponds to a different first subsequence, and each column in the first two-dimensional tensor corresponds to a different moment in the first subsequence;

[0018] The filtered EEG signal is evenly divided into a plurality of second subsequences, the length of the second subsequence is the length of the second period, and the plurality of second subsequences are arranged into a second two-dimensional tensor, each row in the second two-dimensional tensor corresponds to a different second subsequence, and each column in the second two-dimensional tensor corresponds to a different moment in the second subsequence;

[0019] The filtered EEG signal is evenly divided into a plurality of third subsequences, the length of the third subsequence is the length of the third period, and the plurality of third subsequences are arranged into a third two-dimensional tensor, each row in the third two-dimensional tensor corresponds to a different third subsequence, and each column in the third two-dimensional tensor corresponds to a different moment in the third subsequence;

[0020] Feature extraction is performed on the first two-dimensional tensor, the second two-dimensional tensor and the third two-dimensional tensor to obtain an EEG feature map.

[0021] In one possible implementation, the target area is an area adjacent to the alpha wave and the beta wave, and the first cycle, the second cycle, and the third cycle with the top three amplitude values ​​are extracted in the target area based on the EEG signal after filtering, including:

[0022] The frequency domain information of the EEG signal after filtering is extracted by fast Fourier transform, and the frequency domain information includes the amplitude values ​​corresponding to the data points of different frequencies in the EEG signal;

[0023] Based on the frequency domain information of the EEG signal, the first frequency data point, the second frequency data point and the third frequency data point with the top three amplitude values ​​are extracted in the area adjacent to the α wave and the β wave. The first cycle is the cycle corresponding to the first frequency data point, the second cycle is the cycle corresponding to the second frequency data point, and the third cycle is the cycle corresponding to the third frequency data point.

[0024] In one possible implementation, the facial feature map and the EEG feature map are fused to obtain multimodal fusion data, including:

[0025] The facial feature map and the EEG feature map are channel-merged to obtain multimodal fusion data.

[0026] One possible implementation method is to perform expression recognition based on facial feature maps, EEG feature maps and multimodal fusion data, including:

[0027] The facial feature map, the EEG feature map and the multimodal fusion data are input into the preset expression recognition model for recognition, and the expression recognition result of the subject is obtained;

[0028] The structure of the preset expression recognition model includes a deep fusion extraction layer, a fully connected layer, an expression prediction layer and multiple feature fusion extraction layers. The multiple feature fusion extraction layers use deep separable convolution to extract feature information of different depths based on facial feature maps, EEG feature maps and multimodal fusion data respectively; the deep fusion extraction layer is used to reduce the number of channels of the feature maps output by the multiple feature fusion extraction layers; the fully connected layer is used to perform feature fusion on the feature maps output by the deep fusion extraction layer; the expression prediction layer is used to map the output results of the fully connected layer to expression categories.

[0029] In one possible implementation, the multiple feature fusion extraction layers include an initial feature fusion extraction layer, a secondary feature fusion extraction layer, a third-level feature fusion extraction layer, a fourth-level feature fusion extraction layer, a fifth-level feature fusion extraction layer, and a sixth-level feature fusion extraction layer. The facial feature map, the EEG feature map, and the multimodal fusion data are input into a preset expression recognition model for recognition, and the expression recognition results of the subject are obtained, including:

[0030] In the initial feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on facial feature map and EEG feature map respectively, and the P1 feature map and P2 feature map with dimensions of N / 2×N / 2×12 are obtained; 12 1×1 convolution kernels are used to extract fusion features of multimodal fusion data, and convolution down-sampling is performed to obtain the P3 feature map with dimensions of N / 2×N / 2×12; the P1 feature map, P2 feature map and P3 feature map are channel-merged to obtain the P4 feature map with dimensions of N / 2×N / 2×36;

[0031] In the secondary feature fusion extraction layer, grouped convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P1 feature map and P2 feature map respectively, and the P5 feature map and P6 feature map with dimensions of N / 4×N / 4×24 are obtained; 24 1×1 convolution kernels are used to extract fusion features of P4 feature map, and convolution down-sampling is performed to obtain P7 feature map with dimensions of N / 4×N / 4×24; P5 feature map, P6 feature map and P7 feature map are channel-merged to obtain P8 feature map with dimensions of N / 4×N / 4×72;

[0032] In the third-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P5 feature map and P6 feature map respectively, and the P9 feature map and P10 feature map with dimensions of N / 8×N / 8×48 are obtained; 48 1×1 convolution kernels are used to extract fusion features of P8 feature map, and convolution down-sampling is performed to obtain P11 feature map with dimensions of N / 8×N / 8×48; P9 feature map, P10 feature map and P11 feature map are merged by channel, and P12 feature map with dimensions of N / 8×N / 8×144 is obtained;

[0033] In the fourth-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P9 feature map and P10 feature map respectively, and the P13 feature map and P14 feature map with dimensions of N / 16×N / 16×96 are obtained; 96 1×1 convolution kernels are used to extract fusion features of P12 feature map, and convolution down-sampling is performed to obtain P15 feature map with dimensions of N / 16×N / 16×96; P13 feature map, P14 feature map and P15 feature map are channel-merged to obtain P16 feature map with dimensions of N / 16×N / 16×288;

[0034] In the five-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P13 feature map and P14 feature map respectively, and the dimensions of P17 feature map and P18 feature map are obtained. The P16 feature map is fused and extracted using 192 1×1 convolution kernels, and convolution down-sampled to obtain the P19 feature map with the dimension of N / 32×N / 32×192. The P17 feature map, P18 feature map and P19 feature map are channel-merged to obtain the P20 feature map with the dimension of N / 32×N / 32×576.

[0035] In the six-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with compression and excitation attention mechanism on P17 and P18 feature maps, respectively, to obtain P21 and P22 feature maps with dimensions of N / 64×N / 64×384; 384 1×1 convolution kernels are used to extract fusion features from the P20 feature map, and convolution down-sampling is performed to obtain a P23 feature map with dimensions of N / 64×N / 64×384; the P21 feature map, P22 feature map, and P23 feature map are channel-merged to obtain a P24 feature map with dimensions of N / 64×N / 64×1152;

[0036] In the deep fusion extraction layer, group convolution is used to extract features from the P21 feature map, the P22 feature map, and the P24 feature map using 128 1×1 convolution kernels, respectively, to obtain the P25 feature map, the P26 feature map, and the P27 feature map with dimensions of 1×1×128; the P25 feature map, the P26 feature map, and the P27 feature map are channel-merged to obtain the P28 feature map with a dimension of 1×1×384;

[0037] In the fully connected layer, the P28 feature map is weighted connected to 64 neurons;

[0038] In the expression prediction layer, the result after weight connection in the fully connected layer is weighted connected with M neurons to obtain the expression recognition result of the subject, where M represents the number of expression categories.

[0039] In a second aspect, the present application provides an expression recognition device based on multimodal data, comprising:

[0040] An acquisition module is used to acquire an image to be recognized and an EEG signal of a subject. The image to be recognized includes a facial image of the subject. The EEG signal is acquired by an EEG signal acquisition device when the subject is in different emotional states.

[0041] A first preprocessing module is used to preprocess the image to be identified to obtain a facial feature map;

[0042] The second preprocessing module is used to preprocess the EEG signal to obtain an EEG characteristic graph;

[0043] A data fusion module is used to fuse the facial feature map and the EEG feature map to obtain multimodal fusion data;

[0044] The expression recognition module is used to perform expression recognition based on facial feature maps, EEG feature maps and multimodal fusion data.

[0045] In a third aspect, the present application provides an edge computing device, comprising: a processor and a memory, the memory being used to store a computer program; the processor being used to run the computer program to implement the expression recognition method based on multimodal data as described in the first aspect.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is executed on a computer, the computer implements the expression recognition method based on multimodal data as described in the first aspect.

[0047] Compared with the prior art, this application has at least the following technical effects:

[0048] The present application provides a method for facial expression recognition based on multimodal data, including obtaining an image to be recognized and an EEG signal of a subject, and preprocessing the image to be recognized and the EEG signal of the subject respectively to obtain a facial feature map and an EEG feature map; fusing the facial feature map and the EEG feature map to obtain multimodal fusion data; and performing facial expression recognition based on the facial feature map, the EEG feature map and the multimodal fusion data. The method can achieve the following technical effects:

[0049] (1) Improving the accuracy of expression recognition: By fusing multimodal data (facial image data and EEG image data) for expression recognition, we can make full use of the information carried by different modal data and make up for the limitations of single modal data. Facial images can provide intuitive expression appearance features, while EEG image data can reflect the relationship between neural activity and emotional state in the brain. Combining facial image data with EEG image data helps to analyze expressions from multiple angles, thereby improving the accuracy of expression recognition.

[0050] (2) Capturing subtle differences in facial expressions: EEG image data is highly sensitive to emotions and can capture subtle differences in facial expressions that may be overlooked by facial images.

[0051] (3) Enhance the reliability and stability of expression recognition: Facial image data is easily affected by environmental factors such as lighting, angle, and occlusion, which reduces the reliability of expression recognition. However, EEG image data is relatively stable and is not directly affected by external environmental factors. After fusing the two, the impact of environmental factors on expression recognition can be reduced to a certain extent, thereby enhancing the reliability and stability of expression recognition.

[0052] (4) Adapting to individual differences: Different individuals may have different facial expression features and EEG activity patterns. Single-modality expression recognition methods may find it difficult to adapt to such individual differences, resulting in unstable performance on different individuals. Multimodal data fusion can comprehensively consider individual facial features and EEG activities, better adapt to the differences between different individuals, and improve the stability and reliability of expression recognition on different individuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flowchart of a method for facial expression recognition based on multimodal data provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the image preprocessing process provided in the embodiment of the present application;

[0055] Figure 3 A schematic diagram of the process of preprocessing EEG signals provided in an embodiment of the present application;

[0056] Figure 4A schematic diagram of a process for performing recognition using a preset expression recognition model provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of an expression recognition device based on multimodal data provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In the embodiments of the present application, unless otherwise specified, the character " / " indicates that the objects before and after the association are in an or relationship. For example, A / B can represent A or B. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone.

[0060] It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used to distinguish the description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor can they be understood as indicating or implying order.

[0061] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. In addition, "at least one of the following" or similar expressions refers to any combination of these items, which may include any combination of single items or plural items. For example, at least one of A, B, or C may represent: A, B, C, A and B, A and C, B and C, or A, B and C. Among them, each of A, B, and C may be an element itself, or a set containing one or more elements.

[0062] In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment", etc. are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0063] In the embodiments of the present application, "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. In the embodiments of the present application, communication and transmission can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. For example, transmission can include sending and / or receiving, which can be a noun or a verb.

[0064] The equal to involved in the embodiments of the present application can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. It should be noted that when equal to is used in conjunction with greater than, it cannot be used in conjunction with less than; when equal to is used in conjunction with less than, it cannot be used in conjunction with greater than.

[0065] The inventors found in the process of implementing the embodiments of the present application that the image-based expression recognition algorithms, such as VGG-Face, FaceNet and other algorithms, use single-modal facial data for expression recognition, and such algorithms have certain limitations: on the one hand, human expression is a complex process involving multiple aspects of information. Although facial images are the most intuitive form of expression, relying solely on facial features may be affected by factors such as illumination, angle, and occlusion, and the algorithm may fail when the illumination changes or occlusion occurs. For example, under low-light conditions, facial details may be difficult to capture clearly, thereby affecting the accuracy of expression recognition. At the same time, different people may have similar facial features, but the true emotions behind their expressions may be completely different; on the other hand, physiological signals such as EEG, heart rate, and skin conductivity can also reflect a person's emotional state; when a person is in a state of tension or excitement, changes such as enhanced EEG beta waves and suppressed alpha waves will occur, and heart rate will increase and skin conductivity will change; the comprehensive use of these multimodal data can provide more comprehensive and accurate information for expression recognition. In addition, the model parameters of VGG-Face and FaceNet are large, and they cannot run in real time on low-computing embedded devices.

[0066] Based on the above problems, the embodiment of the present application proposes an expression recognition method based on multimodal data, which helps to improve the accuracy of expression recognition, capture subtle expression differences, enhance the reliability and stability of expression recognition, and better adapt to the differences between different individuals.

[0067] Now combined Figure 1-Figure 4 The expression recognition method based on multimodal data provided in an embodiment of the present application is described.

[0068] Figure 1 The flowchart of the expression recognition method based on multimodal data provided in the embodiment of the present application specifically includes the following steps:

[0069] Step S11, obtaining the image to be recognized and the EEG signal of the subject.

[0070] Specifically, a high-definition camera can be used to obtain an image to be identified, where the image to be identified includes a facial image of the subject. Using a high-definition camera can ensure that the resolution of the image is high enough to clearly capture the details of the facial expression.

[0071] While acquiring the image, the EEG signals of the subject are synchronously collected. The EEG signals can be collected using an EEG signal collection device. After ensuring that the collection device is worn correctly and has good signal quality, the EEG signals are collected when the subject is in different emotional states (such as happiness, sadness, anger, etc.) to obtain EEG data related to different expressions.

[0072] Step S12, preprocessing the image to be recognized to obtain a facial feature map.

[0073] Specifically, image preprocessing includes image brightness adjustment, face detection and cropping, image normalization, denoising and contrast enhancement.

[0074] like Figure 2 As shown, Figure 2 The following is a flow chart of image preprocessing provided in an embodiment of the present application. In the present application, step S12 may specifically include the following steps:

[0075] Step S121, adjusting the brightness of the image to be identified according to a preset brightness threshold range to obtain a first image.

[0076] In this step, by adjusting the brightness of the image to be within a preset brightness threshold range, the impact of lighting changes on expression recognition can be reduced.

[0077] Step S122, detecting and cropping the facial image of the subject in the first image.

[0078] Specifically, the Haar feature cascade classifier is used to quickly and accurately detect the face position in the first image, and the face part of the image is further cropped to obtain the face image of the subject. Among them, the Haar cascade classifier is a face detection method based on Haar-like features, which gradually screens the face area through cascaded weak classifiers.

[0079] Step S123, performing size normalization, denoising and contrast enhancement processing on the face image of the subject in sequence to obtain a facial feature map.

[0080] Specifically, the collected facial images of the subjects are resized and adjusted to a fixed size of N×N to facilitate subsequent processing and model input; Gaussian filtering technology is used to remove noise in facial images; and the histogram equalization method is used to enhance the contrast of the image to make the expression features more obvious. By resizing, denoising and enhancing the contrast of the facial images of the subjects in turn, a facial feature map with a dimension of N×N×3 can be obtained, where "N×N" represents the resolution of the image, that is, the width and height of the image are both N pixels. And "×3" means that each pixel has three color channels, namely RGB three components. The three-channel RGB facial feature map has richer color information.

[0081] Step S13, preprocessing the EEG signal to obtain an EEG characteristic graph.

[0082] Specifically, the EEG signal preprocessing includes filtering, period extraction, sequence folding to form a two-dimensional tensor, and feature extraction of the two-dimensional tensor.

[0083] like Figure 3 As shown, Figure 3 The schematic diagram of the process of preprocessing the EEG signal provided in the embodiment of the present application. In the present application, step S13 may specifically include the following steps:

[0084] Step S131, filtering the EEG signal.

[0085] Specifically, the collected EEG signals are filtered to remove low-frequency drift and high-frequency noise. Usually, a bandpass filter is used to retain signals within a specific frequency range, such as 1-40 Hz. This frequency range contains important information related to brain activity.

[0086] Step S132, extracting the first period, the second period and the third period with the top three amplitude values ​​in the target area based on the filtered EEG signal, and arranging the amplitude values ​​from high to low.

[0087] Specifically, the EEG signal processed in step S131 is expressed as a one-dimensional signal with a time series length of T. The time series is analyzed in the frequency domain using FFT (Fast Fourier Transform), that is, the frequency domain information of the EEG signal after filtering is extracted by fast Fourier transform, and the frequency domain information includes the amplitude values ​​corresponding to the different frequency data points in the EEG signal. Based on the frequency domain information of the EEG signal, the first frequency data point, the second frequency data point and the third frequency data point with the top three amplitude values ​​are extracted in the target area, where the amplitude values ​​are arranged from high to low. Each frequency corresponds to a cycle length, and the cycle corresponding to each frequency data point can be calculated by "cycle length = 1 / frequency". That is, the first cycle corresponding to the first frequency data point, the second cycle corresponding to the second frequency data point, and the third cycle corresponding to the third frequency data point are obtained.

[0088] Taking into account the sparsity of the frequency domain and the need to avoid noise caused by meaningless high frequencies, the target area is selected as the area adjacent to the α wave and the β wave. The first three frequencies with larger amplitude values ​​in the target area are selected. These three frequencies correspond to the three most significant cycle lengths, thus obtaining the main cycle information of the EEG data.

[0089] Step S133: Evenly divide the filtered electroencephalogram signal into a plurality of first subsequences, where the length of the first subsequence is the length of the first period, and arrange the plurality of first subsequences into a first two-dimensional tensor.

[0090] In this step, based on the first cycle obtained in the above step S132, the EEG signal after filtering is subjected to a sequence folding operation. Specifically, the EEG signal after filtering is evenly divided into a plurality of first subsequences, and the length of the first subsequence is the length of the first cycle. If the length of the time series corresponding to the EEG signal cannot be divided by the length of the first cycle, it is necessary to add 0 at the end of the time series so that the sequence length can be divided by the length of the cycle. Next, the plurality of first subsequences are arranged into a first two-dimensional tensor, wherein each row in the first two-dimensional tensor corresponds to a different first subsequence, and each column in the first two-dimensional tensor corresponds to a different moment in the first subsequence.

[0091] For example, the length of the time series of the EEG signal after filtering is T, and the length of the first cycle is 2 / T. Then the EEG signal after filtering is evenly divided into two first subsequences, the length of the first subsequence is 2 / T, and the two first subsequences are arranged into a first two-dimensional tensor. The first two-dimensional tensor has two rows, each row includes one first subsequence, and each column corresponds to a different moment in the first subsequence.

[0092] Step S134: evenly divide the filtered electroencephalogram signal into a plurality of second subsequences, wherein the length of the second subsequence is the length of the second period, and arrange the plurality of second subsequences into a second two-dimensional tensor.

[0093] The same technical principle as step S133, based on the second cycle obtained in the above step S132, the EEG signal after filtering is subjected to sequence folding operation. Specifically, the EEG signal after filtering is evenly divided into a plurality of second subsequences, the length of the second subsequence is the length of the second cycle, and the plurality of second subsequences are arranged into a second two-dimensional tensor, each row in the second two-dimensional tensor corresponds to a different second subsequence, and each column in the second two-dimensional tensor corresponds to a different moment in the second subsequence.

[0094] Step S135: evenly divide the filtered electroencephalogram signal into a plurality of third subsequences, wherein the length of the third subsequence is the length of the third period, and arrange the plurality of third subsequences into a third two-dimensional tensor.

[0095] The same technical principle as step S133, based on the third period obtained in the above step S132, the EEG signal after filtering is subjected to sequence folding operation. Specifically, the EEG signal after filtering is evenly divided into a plurality of third subsequences, the length of the third subsequence is the length of the third period, the plurality of third subsequences are arranged into a third two-dimensional tensor, each row in the third two-dimensional tensor corresponds to a different third subsequence, and each column in the third two-dimensional tensor corresponds to a different moment in the third subsequence.

[0096] It should be noted that steps S133, S134, and S135 can be executed simultaneously (in parallel) or sequentially. Execution simultaneously can increase execution speed and shorten data processing time. When executed sequentially, the execution order can be S133, S134, S135, or S134, S135, S133, etc. This application does not limit the execution order. Figure 3 The flowchart shown is only a schematic diagram of one implementation method.

[0097] Step S136, performing feature extraction on the first two-dimensional tensor, the second two-dimensional tensor and the third two-dimensional tensor to obtain an EEG feature map.

[0098] Through the above steps S133, S134, and S135, the one-dimensional time series EEG signal is converted into three two-dimensional tensors, and the three two-dimensional tensors are feature extracted using a fixed parameter encoder to obtain an EEG feature map with a dimension of N×N×3. "N×N" represents the spatial dimension of the EEG signal, and the "N" here is consistent with the "N" in the dimension of the facial feature map obtained by image preprocessing, which meets the input requirements of the subsequent expression recognition model. "×3" represents the three feature channels of the EEG signal, specifically referring to the results of feature extraction of the three two-dimensional tensors.

[0099] Step S14, fusing the facial feature map and the EEG feature map to obtain multimodal fusion data.

[0100] Specifically, the facial feature map with a dimension of N×N×3 obtained in the above step S12 and the EEG feature map with a dimension of N×N×3 obtained in step S13 are channel-merged to obtain multimodal fusion data with a dimension of N×N×6.

[0101] In the data fusion stage, through the above steps S12 and S13, the acquired images to be recognized and EEG signals are first preprocessed, and the EEG data is converted into two-dimensional EEG image data with multiple channels, so that the EEG data and facial image data have the same feature dimension, and then data fusion is performed. The unification of the data format of EEG data and facial image data is conducive to the rapid calculation of embedded hardware and accelerates the processing speed of expression recognition.

[0102] Step S15, performing expression recognition based on the facial feature map, the EEG feature map and the multimodal fusion data.

[0103] Specifically, the facial feature map, the EEG feature map and the multimodal fusion data are input into a preset expression recognition model for recognition to obtain the expression recognition result of the subject.

[0104] Among them, the structure of the preset expression recognition model includes a deep fusion extraction layer, a fully connected layer, an expression prediction layer and multiple feature fusion extraction layers. The multiple feature fusion extraction layers use deep separable convolution to extract feature information of different depths based on facial feature maps, EEG feature maps and multimodal fusion data respectively; the deep fusion extraction layer is used to reduce the number of channels of the feature maps output by the multiple feature fusion extraction layers; the fully connected layer is used to perform feature fusion on the feature maps output by the deep fusion extraction layer, and the expression prediction layer is used to map the output results of the fully connected layer to expression categories.

[0105] This application uses a preset expression recognition model for expression recognition. The model has a simple structure, in which multiple feature fusion extraction layers are combined with depth-separable convolutions. Feature information of different depths is extracted based on facial feature maps, EEG feature maps and multimodal fusion data, which helps to reduce the number of model parameters. While ensuring a high recognition accuracy, it can greatly reduce the demand for computing resources and the complexity of the model, so that it can run on resource-constrained devices, such as mobile devices, embedded devices, etc., expanding the application scope of expression recognition technology and realizing lightweight and efficient expression recognition.

[0106] In some optional embodiments, such as Figure 4 As shown, the structure of the preset expression recognition model includes an initial feature fusion extraction layer, a secondary feature fusion extraction layer, a tertiary feature fusion extraction layer, a quaternary feature fusion extraction layer, a quinary feature fusion extraction layer, a VI feature fusion extraction layer, a deep fusion extraction layer, a fully connected layer and an expression prediction layer.

[0107] It can be understood that in other embodiments, the structure of the preset expression recognition model can also include 5 or 7 or 8 multiple feature fusion extraction layers. The present application uses 6 feature fusion extraction layers to ensure the best expression recognition effect. If the model includes more than 6 feature fusion extraction layers, it will lead to a large amount of calculation of the model. If the model includes less than 6 feature fusion extraction layers, the prediction effect cannot be guaranteed.

[0108] Figure 4 The schematic diagram of the process of using a preset expression recognition model for recognition provided in the embodiment of the present application specifically includes:

[0109] In the initial feature fusion extraction layer, group convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the facial feature map and the EEG feature map, respectively, to obtain P1 feature map and P2 feature map with dimensions of N / 2×N / 2×12; 12 1×1 convolution kernels are used to extract fusion features of the multimodal fusion data, and convolution downsampling is performed (referring to downsampling using ordinary convolution or standard convolution) to obtain a P3 feature map with a dimension of N / 2×N / 2×12; the P1 feature map, P2 feature map and P3 feature map are channel-merged to obtain a P4 feature map with a dimension of N / 2×N / 2×36.

[0110] In the secondary feature fusion extraction layer, grouped convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the P1 and P2 feature maps, respectively, to obtain P5 and P6 feature maps with dimensions of N / 4×N / 4×24; 24 1×1 convolution kernels are used to extract fusion features of the P4 feature map, and convolution down-sampling is performed to obtain a P7 feature map with a dimension of N / 4×N / 4×24; the P5, P6 and P7 feature maps are channel-merged to obtain a P8 feature map with a dimension of N / 4×N / 4×72.

[0111] In the third-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the P5 and P6 feature maps, respectively, to obtain the P9 and P10 feature maps with dimensions of N / 8×N / 8×48; 48 1×1 convolution kernels are used to extract fusion features of the P8 feature map, and convolution down-sampling is performed to obtain the P11 feature map with dimensions of N / 8×N / 8×48; the P9, P10 and P11 feature maps are channel-merged to obtain the P12 feature map with dimensions of N / 8×N / 8×144.

[0112] In the fourth-level feature fusion extraction layer, grouped convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the P9 feature map and the P10 feature map, respectively, to obtain the P13 feature map and the P14 feature map with dimensions of N / 16×N / 16×96; 96 1×1 convolution kernels are used to extract fusion features of the P12 feature map, and convolution down-sampling is performed to obtain the P15 feature map with dimensions of N / 16×N / 16×96; the P13 feature map, the P14 feature map and the P15 feature map are channel-merged to obtain the P16 feature map with dimensions of N / 16×N / 16×288.

[0113] In the five-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the P13 and P14 feature maps, respectively, to obtain the P17 and P18 feature maps with dimensions of N / 32×N / 32×192; 192 1×1 convolution kernels are used to extract fusion features of the P16 feature map, and convolution down-sampling is performed to obtain the P19 feature map with dimensions of N / 32×N / 32×192; the P17, P18 and P19 feature maps are channel-merged to obtain the P20 feature map with dimensions of N / 32×N / 32×576.

[0114] In the six-level feature fusion extraction layer, grouped convolution is used to perform deep separable convolution with SE (Squeeze-and-Excitation) attention mechanism on the P17 and P18 feature maps, respectively, to obtain the P21 and P22 feature maps with dimensions of N / 64×N / 64×384; 384 1×1 convolution kernels are used to extract fusion features of the P20 feature map, and convolution down-sampling is performed to obtain the P23 feature map with dimensions of N / 64×N / 64×384; the P21, P22 and P23 feature maps are channel-merged to obtain the P24 feature map with dimensions of N / 64×N / 64×1152.

[0115] In the deep fusion extraction layer, group convolution is used to extract features from the P21, P22, and P24 feature maps using 128 1×1 convolution kernels, reducing the number of channels of the P21, P22, and P24 feature maps to 128 dimensions, and obtaining the P25, P26, and P27 feature maps with dimensions of 1×1×128; the P25, P26, and P27 feature maps are merged through their channels to obtain the P28 feature map with a dimension of 1×1×384.

[0116] In the fully connected layer, the P28 feature map is weighted connected to 64 neurons. Weighted connection refers to the connection weight between the input feature and the neuron, that is, the 1×1×384 input feature is connected to each of the 64 neurons through a weight. Each neuron calculates the sum of a weighted input and then generates the output of the neuron through an activation function (usually nonlinear, such as ReLU). The output of these 64 neurons can be regarded as a transformation and combination of the input features, capturing higher-level features or patterns.

[0117] In the expression prediction layer, the result after weight connection in the fully connected layer is weighted connected with M neurons to obtain the expression recognition result of the subject, where M represents the number of expression categories. Optionally, the expression categories include joy, sadness, anger, surprise, disgust, fear, shyness and neutral. Then M can be any integer between 1 and 8.

[0118] In one embodiment, M=8, then the outputs of the 64 neurons will be weighted connected to 8 neurons, that is, the output of each of the 64 neurons will be connected to each of the 8 output neurons. Ultimately, these 8 output neurons will generate a probability distribution representing the probability that the input feature belongs to each of the 8 emotion categories. This is usually achieved through a softmax activation function, which ensures that the output of the output layer can be interpreted as a probability and that the sum of all outputs is 1.

[0119] In this application, each stage of recognition using the preset expression recognition model includes facial feature extraction, EEG feature extraction, and fusion feature extraction of the two modalities, so that data fusion permeates all aspects of the algorithm; and the model uses a variety of convolutions such as deep separable convolution, grouped convolution, and 1×1 convolution based on the attention mechanism, which greatly reduces the amount of model calculation, speeds up the model reasoning speed, and can perform multimodal expression recognition in real time. And the model has a simple structure and is easy to deploy on terminals.

[0120] This application performs expression recognition through the fusion of multimodal data (facial image data and EEG image data), improves the accuracy, reliability and stability of expression recognition, and can capture subtle differences in expression and adapt to different individual differences. The facial image data, EEG image data and multimodal fusion data are input into a preset expression recognition model for recognition. The model uses a variety of convolutions such as deep separable convolution, grouped convolution, and 1×1 convolution based on the attention mechanism, which greatly reduces the amount of model calculation and speeds up the model reasoning. It can perform multimodal expression recognition in real time and realize lightweight and efficient expression recognition.

[0121] Based on the same idea, the embodiment of the present application also provides an expression recognition device based on multimodal data, such as Figure 5 As shown, Figure 5 The schematic diagram of the structure of the expression recognition device based on multimodal data provided in the embodiment of the present application, the expression recognition device based on multimodal data 50 may include:

[0122] An acquisition module 51 is used to acquire an image to be recognized and an EEG signal of a subject, wherein the image to be recognized includes a face image of the subject, and the EEG signal is acquired by an EEG signal acquisition device when the subject is in different emotional states;

[0123] A first preprocessing module 52, used for preprocessing the image to be recognized to obtain a facial feature map;

[0124] The second preprocessing module 53 is used to preprocess the EEG signal to obtain an EEG characteristic graph;

[0125] The data fusion module 54 is used to fuse the facial feature map and the EEG feature map to obtain multi-modal fusion data;

[0126] The expression recognition module 55 is used to perform expression recognition based on the facial feature map, the EEG feature map and the multimodal fusion data.

[0127] In one possible implementation, the first preprocessing module 52 may also be used to:

[0128] Adjusting the brightness of the image to be identified according to a preset brightness threshold range to obtain a first image;

[0129] Detecting and cropping a facial image of a subject in the first image;

[0130] The face image of the subject is processed by size normalization, denoising and contrast enhancement in sequence to obtain a facial feature map.

[0131] In one possible implementation, the second preprocessing module 53 may also be used to:

[0132] Filtering the EEG signal;

[0133] Based on the EEG signals after filtering, the first, second and third cycles with the top three amplitude values ​​are extracted in the target area, and the amplitude values ​​are arranged from high to low;

[0134] The filtered EEG signal is evenly divided into a plurality of first subsequences, the length of the first subsequence is the length of the first period, and the plurality of first subsequences are arranged into a first two-dimensional tensor, each row in the first two-dimensional tensor corresponds to a different first subsequence, and each column in the first two-dimensional tensor corresponds to a different moment in the first subsequence;

[0135] The filtered EEG signal is evenly divided into a plurality of second subsequences, the length of the second subsequence is the length of the second period, and the plurality of second subsequences are arranged into a second two-dimensional tensor, each row in the second two-dimensional tensor corresponds to a different second subsequence, and each column in the second two-dimensional tensor corresponds to a different moment in the second subsequence;

[0136] The filtered EEG signal is evenly divided into a plurality of third subsequences, the length of the third subsequence is the length of the third period, and the plurality of third subsequences are arranged into a third two-dimensional tensor, each row in the third two-dimensional tensor corresponds to a different third subsequence, and each column in the third two-dimensional tensor corresponds to a different moment in the third subsequence;

[0137] Feature extraction is performed on the first two-dimensional tensor, the second two-dimensional tensor and the third two-dimensional tensor to obtain an EEG feature map.

[0138] In one possible implementation, the target area is an area adjacent to the alpha wave and the beta wave, and the first cycle, the second cycle, and the third cycle with the top three amplitude values ​​are extracted in the target area based on the EEG signal after filtering, including:

[0139] The frequency domain information of the EEG signal after filtering is extracted by fast Fourier transform, and the frequency domain information includes the amplitude values ​​corresponding to the data points of different frequencies in the EEG signal;

[0140] Based on the frequency domain information of the EEG signal, the first frequency data point, the second frequency data point and the third frequency data point with the top three amplitude values ​​are extracted in the area adjacent to the α wave and the β wave. The first cycle is the cycle corresponding to the first frequency data point, the second cycle is the cycle corresponding to the second frequency data point, and the third cycle is the cycle corresponding to the third frequency data point.

[0141] In one possible implementation, the data fusion module 54 may also be used to:

[0142] The facial feature map and the EEG feature map are channel-merged to obtain multimodal fusion data.

[0143] In one possible implementation, the expression recognition module 55 can also be used to: input the facial feature map, the EEG feature map and the multimodal fusion data into a preset expression recognition model for recognition, and obtain the expression recognition result of the subject;

[0144] The structure of the preset expression recognition model includes a deep fusion extraction layer, a fully connected layer, an expression prediction layer and multiple feature fusion extraction layers. The multiple feature fusion extraction layers use deep separable convolution to extract feature information of different depths based on facial feature maps, EEG feature maps and multimodal fusion data respectively; the deep fusion extraction layer is used to reduce the number of channels of the feature maps output by the multiple feature fusion extraction layers; the fully connected layer is used to perform feature fusion on the feature maps output by the deep fusion extraction layer; the expression prediction layer is used to map the output results of the fully connected layer to expression categories.

[0145] In one possible implementation, the multiple feature fusion extraction layers include an initial feature fusion extraction layer, a secondary feature fusion extraction layer, a third-level feature fusion extraction layer, a fourth-level feature fusion extraction layer, a fifth-level feature fusion extraction layer, and a sixth-level feature fusion extraction layer. The expression recognition module 55 can also be used for:

[0146] In the initial feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on facial feature map and EEG feature map respectively, and the P1 feature map and P2 feature map with dimensions of N / 2×N / 2×12 are obtained; 12 1×1 convolution kernels are used to extract fusion features of multimodal fusion data, and convolution down-sampling is performed to obtain the P3 feature map with dimensions of N / 2×N / 2×12; the P1 feature map, P2 feature map and P3 feature map are channel-merged to obtain the P4 feature map with dimensions of N / 2×N / 2×36;

[0147] In the secondary feature fusion extraction layer, grouped convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P1 feature map and P2 feature map respectively, and the P5 feature map and P6 feature map with dimensions of N / 4×N / 4×24 are obtained; 24 1×1 convolution kernels are used to extract fusion features of P4 feature map, and convolution down-sampling is performed to obtain P7 feature map with dimensions of N / 4×N / 4×24; P5 feature map, P6 feature map and P7 feature map are channel-merged to obtain P8 feature map with dimensions of N / 4×N / 4×72;

[0148] In the third-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P5 feature map and P6 feature map respectively, and the P9 feature map and P10 feature map with dimensions of N / 8×N / 8×48 are obtained; 48 1×1 convolution kernels are used to extract fusion features of P8 feature map, and convolution down-sampling is performed to obtain P11 feature map with dimensions of N / 8×N / 8×48; P9 feature map, P10 feature map and P11 feature map are merged by channel, and P12 feature map with dimensions of N / 8×N / 8×144 is obtained;

[0149] In the fourth-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P9 feature map and P10 feature map respectively, and the P13 feature map and P14 feature map with dimensions of N / 16×N / 16×96 are obtained; 96 1×1 convolution kernels are used to extract fusion features of P12 feature map, and convolution down-sampling is performed to obtain P15 feature map with dimensions of N / 16×N / 16×96; P13 feature map, P14 feature map and P15 feature map are channel-merged to obtain P16 feature map with dimensions of N / 16×N / 16×288;

[0150] In the five-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with attention mechanism of compression and excitation on P13 feature map and P14 feature map respectively, and the dimensions of P17 feature map and P18 feature map are obtained. The P16 feature map is fused and extracted using 192 1×1 convolution kernels, and convolution down-sampled to obtain the P19 feature map with the dimension of N / 32×N / 32×192. The P17 feature map, P18 feature map and P19 feature map are channel-merged to obtain the P20 feature map with the dimension of N / 32×N / 32×576.

[0151] In the six-level feature fusion extraction layer, group convolution is used to perform deep separable convolution with compression and excitation attention mechanism on P17 and P18 feature maps, respectively, to obtain P21 and P22 feature maps with dimensions of N / 64×N / 64×384; 384 1×1 convolution kernels are used to extract fusion features from the P20 feature map, and convolution down-sampling is performed to obtain a P23 feature map with dimensions of N / 64×N / 64×384; the P21 feature map, P22 feature map, and P23 feature map are channel-merged to obtain a P24 feature map with dimensions of N / 64×N / 64×1152;

[0152] In the deep fusion extraction layer, group convolution is used to extract features from the P21 feature map, the P22 feature map, and the P24 feature map using 128 1×1 convolution kernels, respectively, to obtain the P25 feature map, the P26 feature map, and the P27 feature map with dimensions of 1×1×128; the P25 feature map, the P26 feature map, and the P27 feature map are channel-merged to obtain the P28 feature map with a dimension of 1×1×384;

[0153] In the fully connected layer, the P28 feature map is weighted connected to 64 neurons;

[0154] In the expression prediction layer, the result after weight connection in the fully connected layer is weighted connected with M neurons to obtain the expression recognition result of the subject, where M represents the number of expression categories.

[0155] Figure 5 The expression recognition device 50 based on multimodal data provided in the illustrated embodiment can be used to execute the technical solution of the method embodiment shown in the present application. Its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

[0156] It should be understood that the above Figure 5 The division of the various modules of the expression recognition device 50 based on multimodal data shown is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or it can be integrated in a chip of the edge computing device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0157] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0158] In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0159] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided by the embodiment shown in the present application.

[0160] Combine the following Figure 6 The exemplary edge computing device provided in the embodiments of the present application is further introduced. Figure 6 A schematic diagram of the structure of an edge computing device 6000 is shown.

[0161] The above-mentioned edge computing device 6000 may include: at least one processor; and at least one memory communicatively connected to the above-mentioned processor, wherein: the above-mentioned memory stores program instructions that can be executed by the above-mentioned processor, and the processor calls the above-mentioned program instructions to execute the expression recognition method based on multimodal data provided in the embodiment shown in this application.

[0162] Figure 6 A block diagram of an exemplary edge computing device 6000 suitable for implementing the embodiments of the present application is shown. Figure 6 The edge computing device 6000 shown is merely an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present application.

[0163] like Figure 6 As shown, the edge computing device 6000 is in the form of a general computing device. The components of the edge computing device 6000 may include, but are not limited to: one or more processors 6010, a memory 6020, a communication bus 6040 connecting different system components (including the memory 6020 and the processor 6010), and a communication interface 6030.

[0164] The communication bus 6040 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0165] The edge computing device 6000 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the edge computing device, including volatile and non-volatile media, removable and non-removable media.

[0166] The memory 6020 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The edge computing device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Figure 6Not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 6040 via one or more data medium interfaces. The memory 6020 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0167] A program / utility having a set (at least one) of program modules may be stored in the memory 6020, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0168] The edge computing device 6000 may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the edge computing device, and / or any device that enables the edge computing device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via the communication interface 6030. In addition, the edge computing device 6000 may also communicate with the network adapter ( Figure 6 The network adapter may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via the communication bus 6040. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the edge computing device 6000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Drives; hereinafter referred to as: RAID) systems, tape drives, and data backup storage systems.

[0169] The processor 6010 executes various functional applications and data processing by running the programs stored in the memory 6020, such as implementing the method provided in the embodiment of the present application.

[0170] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the edge computing device 6000. In other embodiments of the present application, the edge computing device 6000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0171] In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0172] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0175] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for facial expression recognition based on multimodal data, characterized in that: The method comprises: Acquire an image to be identified and an EEG signal of a subject, wherein the image to be identified includes a facial image of the subject, and the EEG signal is acquired by an EEG signal acquisition device when the subject is in different emotional states; Preprocessing the image to be identified to obtain a facial feature map; Preprocessing the EEG signal to obtain an EEG characteristic graph; Performing data fusion on the facial feature map and the EEG feature map to obtain multimodal fusion data; Expression recognition is performed based on the facial feature map, the EEG feature map and the multimodal fusion data.

2. The method according to claim 1, characterized in that The preprocessing of the image to be identified to obtain a facial feature map includes: Adjusting the brightness of the image to be identified according to a preset brightness threshold range to obtain a first image; Detecting and cropping the face image of the subject in the first image; The face image of the subject is subjected to size normalization, denoising and contrast enhancement processing in sequence to obtain a facial feature map.

3. The method according to claim 2, characterized in that The preprocessing of the EEG signal to obtain an EEG characteristic graph includes: Performing filtering processing on the EEG signal; Extracting the first cycle, the second cycle and the third cycle with the top three amplitude values ​​in the target area based on the filtered EEG signal, wherein the amplitude values ​​are arranged from high to low; Dividing the filtered EEG signal into a plurality of first subsequences on average, wherein the length of the first subsequence is the length of the first period, and arranging the plurality of first subsequences into a first two-dimensional tensor, wherein each row in the first two-dimensional tensor corresponds to a different first subsequence, and each column in the first two-dimensional tensor corresponds to a different moment in the first subsequence; Dividing the filtered EEG signal into a plurality of second subsequences on average, wherein the length of the second subsequences is the length of the second period, and arranging the plurality of second subsequences into a second two-dimensional tensor, wherein each row in the second two-dimensional tensor corresponds to a different second subsequence, and each column in the second two-dimensional tensor corresponds to a different moment in the second subsequence; Dividing the filtered EEG signal into a plurality of third subsequences on average, wherein the length of the third subsequence is the length of the third period, and arranging the plurality of third subsequences into a third two-dimensional tensor, wherein each row in the third two-dimensional tensor corresponds to a different third subsequence, and each column in the third two-dimensional tensor corresponds to a different moment in the third subsequence; Feature extraction is performed on the first two-dimensional tensor, the second two-dimensional tensor, and the third two-dimensional tensor to obtain an electroencephalogram feature map.

4. The method according to claim 3, characterized in that The target area is an area adjacent to the α wave and the β wave, and the first cycle, the second cycle and the third cycle with the top three amplitude values ​​in the target area are extracted based on the EEG signal after filtering, including: Extracting frequency domain information of the filtered EEG signal using fast Fourier transform, wherein the frequency domain information includes amplitude values ​​corresponding to different frequency data points in the EEG signal; Based on the frequency domain information of the EEG signal, the first frequency data point, the second frequency data point and the third frequency data point with the top three amplitude values ​​are extracted in the area adjacent to the α wave and the β wave, the first period is the period corresponding to the first frequency data point, the second period is the period corresponding to the second frequency data point, and the third period is the period corresponding to the third frequency data point.

5. The method according to claim 3, characterized in that: The step of fusing the facial feature map and the EEG feature map to obtain multimodal fusion data includes: The facial feature map and the EEG feature map are channel-merged to obtain multimodal fusion data.

6. The method according to claim 5, characterized in that The performing expression recognition based on the facial feature map, the EEG feature map and the multimodal fusion data includes: Inputting the facial feature map, the EEG feature map and the multimodal fusion data into a preset expression recognition model for recognition, to obtain an expression recognition result of the subject; The structure of the preset expression recognition model includes a deep fusion extraction layer, a fully connected layer, an expression prediction layer and multiple feature fusion extraction layers. The multiple feature fusion extraction layers use deep separable convolution to extract feature information of different depths based on the facial feature map, the EEG feature map and the multimodal fusion data respectively; the deep fusion extraction layer is used to reduce the number of channels of the feature maps output by the multiple feature fusion extraction layers; the fully connected layer is used to perform feature fusion on the feature maps output by the deep fusion extraction layer; the expression prediction layer is used to map the output results of the fully connected layer to expression categories.

7. The method according to claim 6, characterized in that The multiple feature fusion extraction layers include an initial feature fusion extraction layer, a secondary feature fusion extraction layer, a tertiary feature fusion extraction layer, a quaternary feature fusion extraction layer, a quintuple feature fusion extraction layer, and a VI feature fusion extraction layer. The facial feature map, the EEG feature map, and the multimodal fusion data are input into a preset expression recognition model for recognition to obtain the expression recognition result of the subject, including: In the initial feature fusion extraction layer, using group convolution, the facial feature map and the EEG feature map are respectively subjected to a depth-separable convolution with an attention mechanism of compression and excitation, so as to obtain a P1 feature map and a P2 feature map with dimensions of N / 2×N / 2×12; 12 1×1 convolution kernels are used to extract fusion features from the multimodal fusion data, and convolution down-sampling is performed to obtain a P3 feature map with a dimension of N / 2×N / 2×12; the P1 feature map, the P2 feature map, and the P3 feature map are channel-merged to obtain a P4 feature map with a dimension of N / 2×N / 2×36; In the secondary feature fusion extraction layer, using grouped convolution, the P1 feature map and the P2 feature map are respectively subjected to a depth-separable convolution with an attention mechanism of compression and excitation, so as to obtain a P5 feature map and a P6 feature map with dimensions of N / 4×N / 4×24; 24 1×1 convolution kernels are used to perform fusion feature extraction on the P4 feature map, and convolution down-sampling is performed to obtain a P7 feature map with a dimension of N / 4×N / 4×24; the P5 feature map, the P6 feature map, and the P7 feature map are channel-merged to obtain a P8 feature map with a dimension of N / 4×N / 4×72; In the three-level feature fusion extraction layer, group convolution is used to perform depth-separable convolution of the P5 feature map and the P6 feature map respectively with the attention mechanism of compression and excitation, so as to obtain the P9 feature map and the P10 feature map with dimensions of N / 8×N / 8×48; 48 1×1 convolution kernels are used to extract fusion features of the P8 feature map, and convolution down-sampling is performed to obtain the P11 feature map with dimensions of N / 8×N / 8×48; the P9 feature map, the P10 feature map and the P11 feature map are channel-merged to obtain the P12 feature map with dimensions of N / 8×N / 8×144; In the four-level feature fusion extraction layer, group convolution is used to perform deep separable convolution of the P9 feature map and the P10 feature map respectively with the attention mechanism of compression and excitation, so as to obtain the P13 feature map and the P14 feature map with dimensions of N / 16×N / 16×96; 96 1×1 convolution kernels are used to perform fusion feature extraction on the P12 feature map, and convolution down-sampling is performed to obtain the P15 feature map with dimensions of N / 16×N / 16×96; the P13 feature map, the P14 feature map and the P15 feature map are channel-merged to obtain the P16 feature map with dimensions of N / 16×N / 16×288; In the five-level feature fusion extraction layer, group convolution is used to perform deep separable convolution of the P13 feature map and the P14 feature map with an attention mechanism of compression and excitation, respectively, to obtain a P17 feature map and a P18 feature map with dimensions of N / 32×N / 32×192; 192 1×1 convolution kernels are used to perform fusion feature extraction on the P16 feature map, and convolution downsampling is performed to obtain a P19 feature map with a dimension of N / 32×N / 32×192; the P17 feature map, the P18 feature map, and the P19 feature map are channel-merged to obtain a P20 feature map with a dimension of N / 32×N / 32×576; In the six-level feature fusion extraction layer, group convolution is used to perform depth-separable convolution of the P17 feature map and the P18 feature map respectively with the attention mechanism of compression and excitation, so as to obtain the P21 feature map and the P22 feature map with dimensions of N / 64×N / 64×384; 384 1×1 convolution kernels are used to perform fusion feature extraction on the P20 feature map, and convolution down-sampling is performed to obtain the P23 feature map with dimensions of N / 64×N / 64×384; the P21 feature map, the P22 feature map and the P23 feature map are channel-merged to obtain the P24 feature map with dimensions of N / 64×N / 64×1152; In the deep fusion extraction layer, using group convolution, 128 1×1 convolution kernels are used to perform feature extraction on the P21 feature map, the P22 feature map, and the P24 feature map, respectively, to obtain a P25 feature map, a P26 feature map, and a P27 feature map with dimensions of 1×1×128; the P25 feature map, the P26 feature map, and the P27 feature map are channel-merged to obtain a P28 feature map with a dimension of 1×1×384; In the fully connected layer, the P28 feature map is weighted connected to 64 neurons; In the expression prediction layer, the result after weight connection in the fully connected layer is weighted connected with M neurons to obtain the expression recognition result of the subject, where M represents the number of expression categories.

8. An expression recognition device based on multimodal data, characterized in that: The device comprises: An acquisition module, used to acquire an image to be recognized and an EEG signal of a subject, wherein the image to be recognized includes a facial image of the subject, and the EEG signal is acquired by an EEG signal acquisition device when the subject is in different emotional states; A first preprocessing module, used for preprocessing the image to be recognized to obtain a facial feature map; A second preprocessing module is used to preprocess the EEG signal to obtain an EEG characteristic graph; A data fusion module, used for fusing the facial feature map and the EEG feature map to obtain multimodal fusion data; An expression recognition module is used to perform expression recognition based on the facial feature map, the EEG feature map and the multimodal fusion data.

9. An edge computing device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program; the processor is used to run the computer program to implement the expression recognition method based on multimodal data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is run on a computer, the expression recognition method based on multimodal data as described in any one of claims 1-7 is implemented.

Images