Micro-expression recognition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to intelligent decision making, and relates to a micro-expression recognition method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining to-be-recognized original image data, original voice data and original brain wave data; performing preprocessing operation and feature extraction operation on the original image data, the original voice data and the original brain wave data to obtain image feature data, voice feature data and brain wave feature data; performing feature fusion operation on the image feature data, the voice feature data and the brain wave feature data to obtain three-mode fusion features; and performing micro expression classification operation on the three-mode fusion features according to a set classifier to obtain an expression classification result. According to the invention, through multi-modal data fusion, the advantages of each modal data are fully utilized, and the micro-expression recognition precision is improved; in addition, the influence of environmental factors can be effectively reduced by multi-modal data fusion, and the robustness and generalization ability are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent decision-making technology, and in particular to a micro-expression recognition method, device, computer equipment and storage medium. Background Art

[0002] Micro-expression recognition technology is an advanced method based on facial expression recognition technology. It infers an individual's emotional state and inner world by observing and analyzing tiny facial expression changes that last for a very short time.

[0003] Existing micro-expression recognition technology is mainly based on single-modality image data.

[0004] However, the applicant found that traditional micro-expression recognition technology based on single modal data has the following limitations: image data is greatly affected by factors such as lighting and shooting angle, and the recognition accuracy is not high; therefore, traditional micro-expression recognition technology has the problem of low recognition accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a micro-expression recognition method, apparatus, computer equipment and storage medium to solve the problem of low recognition accuracy in traditional micro-expression recognition technology.

[0006] In order to solve the above technical problems, the present application embodiment provides a micro-expression recognition method, which adopts the following technical solution:

[0007] Acquire original image data, original voice data and original brain wave data to be recognized;

[0008] Preprocessing the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data;

[0009] Performing feature extraction operations on the preprocessed image data, the preprocessed voice data, and the preprocessed brain wave data respectively to obtain image feature data, voice feature data, and brain wave feature data;

[0010] Performing a feature fusion operation on the image feature data, the speech feature data, and the brain wave feature data to obtain a trimodal fusion feature;

[0011] The micro-expression classification operation is performed on the three-modal fusion features according to the set classifier to obtain the expression classification result.

[0012] Furthermore, the steps of performing preprocessing operations on the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data specifically include the following steps:

[0013] Performing an image denoising operation on the original image data according to a Gaussian filtering method to obtain denoised image data;

[0014] Perform segmentation and cropping operations on the denoised image data according to the MASK-RCNN detection algorithm to obtain segmented image data;

[0015] Performing an image normalization operation on the segmented image data to obtain the preprocessed image data;

[0016] Performing a frame operation on the original voice data according to a preset frame length to obtain framed voice data;

[0017] Performing a speech denoising operation on the framed speech data according to a short-time energy threshold method to obtain denoised speech data;

[0018] Performing a speech normalization operation on the denoised speech data to obtain the preprocessed speech data;

[0019] Performing an EEG filtering operation on the original EEG data according to a high-pass filtering method to obtain filtered EEG data;

[0020] Performing brain wave denoising processing on the filtered brain wave data according to a filtering method to obtain denoised brain wave data;

[0021] Performing an EEG normalization operation on the denoised EEG data to obtain the pre-processed EEG data.

[0022] Furthermore, the step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data, and the preprocessed brain wave data to obtain image feature data, voice feature data, and brain wave feature data specifically includes the following steps:

[0023] Extracting color features of the preprocessed image data according to color moments;

[0024] Extracting texture features of the preprocessed image data according to wavelet transform;

[0025] Extracting shape features of the preprocessed image data according to the MASK-RCNN detection algorithm;

[0026] The color feature, the texture feature, and the shape feature are used as the image feature data.

[0027] Furthermore, the step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data, and the preprocessed brain wave data to obtain image feature data, voice feature data, and brain wave feature data specifically includes the following steps:

[0028] Extracting speech spectrum features of the preprocessed speech data according to Fourier transform;

[0029] Extracting instantaneous energy, average energy, and peak energy of the preprocessed speech data to obtain energy features;

[0030] Extracting speech time domain features of the preprocessed speech data according to an autocorrelation method;

[0031] The speech spectrum feature, the energy feature and the speech time domain feature are used as the speech feature data.

[0032] Furthermore, the step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data, and the preprocessed brain wave data to obtain image feature data, voice feature data, and brain wave feature data specifically includes the following steps:

[0033] Extracting brain wave spectrum characteristics of the pre-processed brain wave data according to Fourier transform;

[0034] Extracting the EEG time domain features of the pre-processed EEG data according to the variance analysis method;

[0035] Extracting event-related potential features of the preprocessed EEG data according to the evt-avg method;

[0036] The brain wave spectrum characteristics, the brain wave time domain characteristics and the event-related potential characteristics are used as the brain wave characteristic data.

[0037] Furthermore, the step of performing a feature fusion operation on the image feature data, the voice feature data and the brain wave feature data to obtain a trimodal fusion feature specifically includes the following steps:

[0038] The image feature data, the speech feature data and the brain wave feature data are mapped to a common semantic space according to a cross-modal mapping algorithm, wherein the cross-modal mapping algorithm is expressed as:

[0039]

[0040] in, They represent feature alignment, They represent the feature values ​​of language, image and brain wave respectively, and Z1 and q1 represent the learning parameters of the shared layer.

[0041] In order to solve the above technical problems, the present application also provides a micro-expression recognition device, which adopts the following technical solution:

[0042] The original data acquisition module is used to acquire the original image data, original voice data and original brain wave data to be recognized;

[0043] A preprocessing module, used to perform preprocessing operations on the original image data, the original voice data and the original brain wave data, respectively, to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data;

[0044] A feature extraction module, used to perform feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data, respectively, to obtain image feature data, voice feature data and brain wave feature data;

[0045] A feature fusion module, used for performing a feature fusion operation on the image feature data, the speech feature data and the brain wave feature data to obtain a trimodal fusion feature;

[0046] The micro-expression classification module is used to perform micro-expression classification operations on the three-modal fusion features according to a set classifier to obtain an expression classification result.

[0047] Furthermore, the preprocessing module includes:

[0048] An image denoising submodule, used for performing an image denoising operation on the original image data according to a Gaussian filtering method to obtain denoised image data;

[0049] A segmentation and cropping submodule is used to perform segmentation and cropping operations on the denoised image data according to the MASK-RCNN detection algorithm to obtain segmented image data;

[0050] An image normalization submodule, used for performing an image normalization operation on the segmented image data to obtain the preprocessed image data;

[0051] A framing submodule, used for performing a framing operation on the original voice data according to a preset frame length to obtain framed voice data;

[0052] A speech denoising submodule, used for performing a speech denoising operation on the framed speech data according to a short-time energy threshold method to obtain denoised speech data;

[0053] A speech normalization submodule, used for performing a speech normalization operation on the denoised speech data to obtain the pre-processed speech data;

[0054] The brain wave filtering submodule is used to perform a brain wave filtering operation on the original brain wave data according to a high-pass filtering method to obtain filtered brain wave data;

[0055] The EEG denoising submodule is used to perform EEG denoising processing on the filtered EEG data according to a filtering method to obtain denoised EEG data;

[0056] The brain wave normalization submodule is used to perform a brain wave normalization operation on the denoised brain wave data to obtain the pre-processed brain wave data.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0058] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the micro-expression recognition method as described above when executing the computer-readable instructions.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0060] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the micro-expression recognition method described above are implemented.

[0061] The present application provides a micro-expression recognition method, including: obtaining original image data, original voice data and original brain wave data to be recognized; performing preprocessing operations on the original image data, the original voice data and the original brain wave data respectively to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data; performing feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data respectively to obtain image feature data, voice feature data and brain wave feature data; performing feature fusion operations on the image feature data, the voice feature data and the brain wave feature data to obtain trimodal fusion features; performing micro-expression classification operations on the trimodal fusion features according to a set classifier to obtain expression classification results. Compared with the prior art, the present application improves the micro-expression recognition accuracy by fully utilizing the advantages of each modal data through multimodal data fusion; in addition, multimodal data fusion can effectively reduce the influence of environmental factors and improve robustness and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0064] Figure 2 is a flowchart of the implementation of the micro-expression recognition method provided in Example 1 of the present application;

[0065] Figure 3 is a schematic diagram of the structure of a micro-expression recognition device provided in Example 2 of the present application;

[0066] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0071] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc.

[0073] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0074] It should be noted that the micro-expression recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the micro-expression recognition device is generally set in the server / terminal device.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0076] Embodiment 1

[0077] Continue to refer Figure 2 , shows a flow chart of an embodiment of a micro-expression recognition method according to the present application. The micro-expression recognition method comprises: step S201, step S202, step S203, step S204 and step S205.

[0078] In step S201, original image data, original speech data and original brain wave data to be recognized are obtained.

[0079] In step S202, preprocessing operations are performed on the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data.

[0080] In step S203, feature extraction operations are performed on the preprocessed image data, the preprocessed speech data, and the preprocessed brain wave data to obtain image feature data, speech feature data, and brain wave feature data.

[0081] In step S204, a feature fusion operation is performed on the image feature data, the speech feature data and the brain wave feature data to obtain a tri-modal fusion feature.

[0082] In step S205, micro-expression classification operation is performed on the tri-modal fusion features according to the set classifier to obtain expression classification results.

[0083] In the embodiment of the present application, the fused multimodal features are classified to identify the category of micro-expressions. The classifier can use an existing classifier, such as SVM, KNN, etc., or a deep learning classifier, such as CNN, RNN, etc.

[0084] In an embodiment of the present application, a micro-expression recognition method is provided, including: obtaining original image data, original voice data and original brain wave data to be recognized; performing preprocessing operations on the original image data, original voice data and original brain wave data respectively to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data; performing feature extraction operations on the preprocessed image data, preprocessed voice data and preprocessed brain wave data respectively to obtain image feature data, voice feature data and brain wave feature data; performing feature fusion operations on the image feature data, voice feature data and brain wave feature data to obtain trimodal fusion features; performing micro-expression classification operations on the trimodal fusion features according to the set classifier to obtain expression classification results. Compared with the prior art, the present application improves the micro-expression recognition accuracy by fully utilizing the advantages of each modal data through multimodal data fusion; in addition, multimodal data fusion can effectively reduce the influence of environmental factors and improve robustness and generalization ability.

[0085] In some optional implementations of the embodiments of the present application, the above step S202 specifically includes the following steps:

[0086] Performing an image denoising operation on the original image data according to the Gaussian filtering method to obtain denoised image data;

[0087] According to the MASK-RCNN detection algorithm, the denoised image data is segmented and cropped to obtain segmented image data;

[0088] Performing image normalization operation on the segmented image data to obtain preprocessed image data;

[0089] Performing a frame operation on the original voice data according to a preset frame length to obtain framed voice data;

[0090] Performing a speech denoising operation on the framed speech data according to a short-time energy threshold method to obtain denoised speech data;

[0091] Performing a speech normalization operation on the denoised speech data to obtain preprocessed speech data;

[0092] Performing brain wave filtering operation on the original brain wave data according to the high-pass filtering method to obtain filtered brain wave data;

[0093] Performing brain wave denoising processing on the filtered brain wave data according to the filtering method to obtain denoised brain wave data;

[0094] Perform brain wave normalization operation on the denoised brain wave data to obtain preprocessed brain wave data.

[0095] In the embodiment of the present application, image data preprocessing mainly includes the following steps:

[0096] (1) Denoising: removing noise from the image, such as salt and pepper noise, Gaussian noise, etc. The present invention uses Gaussian filtering for denoising;

[0097] (2) Cropping: Cropping the image to the region of interest, such as the face region. The present invention uses MASK-RCNN to perform image segmentation and cropping;

[0098] (3) Normalization: Normalize the image pixel values ​​to make them have the same dimension. The present invention uses mean variance normalization for normalization.

[0099] In the embodiment of the present application, image data preprocessing mainly includes the following steps:

[0100] (1) Denoising: removing noise from the image, such as salt and pepper noise, Gaussian noise, etc. The present invention uses Gaussian filtering for denoising;

[0101] (2) Cropping: Cropping the image to the region of interest, such as the face region. The present invention uses MASK-RCNN to perform image segmentation and cropping;

[0102] (3) Normalization: Normalize the image pixel values ​​to make them have the same dimension. The present invention uses mean variance normalization for normalization.

[0103] In the embodiment of the present application, the EEG data preprocessing mainly includes the following steps:

[0104] (1) Filtering: remove noise in brain waves, such as electromyographic noise, power line noise, etc. The present invention uses high-pass filtering to remove noise;

[0105] (2) Denoising: removing sudden noise in brain waves, such as electrooculogram noise, electrocardiogram noise, etc. The present invention uses filtering method to remove noise;

[0106] (3) Standardization: Normalize the EEG signals to make them have the same dimension. The present invention uses mean-variance normalization for normalization.

[0107] In some optional implementations of the embodiments of the present application, the above step S203 specifically includes the following steps:

[0108] Extracting color features of preprocessed image data based on color moments;

[0109] Extracting texture features of preprocessed image data based on wavelet transform;

[0110] Extract shape features of preprocessed image data according to the MASK-RCNN detection algorithm;

[0111] Color features, texture features, and shape features are used as image feature data.

[0112] In the embodiments of the present application, the image-based feature extraction mainly includes the following:

[0113] (1) Color features: Color is an important feature of an image and can reflect changes in micro-expressions. Color features include brightness, chroma, and saturation. The present invention uses color moments to extract color features;

[0114] (2) Texture features: Texture is another important feature of an image, which can reflect subtle changes in micro-expressions. Texture features include gray-level co-occurrence matrix (GLCM) and wavelet transform (WT). The present invention uses wavelet transform to extract texture features;

[0115] (3) Shape features: Shape is the third important feature of an image, which can reflect the overall changes in micro-expressions. Shape features include contour features and geometric features. The present invention uses MASK-RCNN to extract shape features.

[0116] In some optional implementations of the embodiments of the present application, the above step S203 specifically includes the following steps:

[0117] Extracting speech spectrum features of preprocessed speech data according to Fourier transform;

[0118] Extract instantaneous energy, average energy, and peak energy of preprocessed speech data to obtain energy features;

[0119] Extracting speech time domain features of preprocessed speech data according to the autocorrelation method;

[0120] The speech spectrum features, energy features and speech time domain features are used as speech feature data.

[0121] In the embodiments of the present application, the feature extraction based on speech mainly includes the following:

[0122] (1) Spectral features: Spectrum is an important feature of speech and can reflect the frequency changes of micro-expressions. Spectral features include Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC). The present invention uses Fourier transform extraction;

[0123] (2) Energy features: Energy is an important feature of speech and can reflect the energy changes of micro-expressions. Energy features include instantaneous energy, average energy, and peak energy;

[0124] (3) Time domain features: Time domain features are important features of speech and can reflect the temporal changes of micro-expressions. Time domain features include time domain amplitude, time domain phase, and time domain envelope. The present invention uses the autocorrelation method to extract;

[0125] In some optional implementations of the embodiments of the present application, the above step S203 specifically includes the following steps:

[0126] Extracting the EEG spectrum characteristics of the preprocessed EEG data according to Fourier transform;

[0127] Extract the EEG time domain features of the preprocessed EEG data according to the variance analysis method;

[0128] The event-related potential features of the preprocessed EEG data were extracted according to the evt-avg method;

[0129] The EEG spectrum characteristics, EEG time domain characteristics and event-related potential characteristics are used as EEG feature data.

[0130] In the embodiments of the present application, the feature extraction methods based on brain waves are mainly the following:

[0131] (1) Spectral characteristics: Spectrum is an important feature of brain waves and can reflect the frequency changes of micro-expressions. Spectral characteristics include brain wave power spectrum and brain wave frequency band power. The present invention uses Fourier transform to extract spectral characteristics;

[0132] (2) Time domain features: Time domain features are important features of brain waves and can reflect the temporal changes of micro-expressions. Time domain features include brain wave waveform, brain wave amplitude, and brain wave phase. The present invention uses variance analysis to extract time domain features;

[0133] (3) Event-related potential (ERP): ERP is an instantaneous peak in brain waves that can reflect specific events of micro-expressions. ERP features include ERP amplitude, ERP waveform, and ERP duration. The present invention selects brain waves in a certain time interval before and after the stimulation onset time point, and uses the evt-avg method to extract ERP waveforms that reflect cognitive or sensory-related brain wave characteristics through multiple event-related averages.

[0134] In some optional implementations of the embodiments of the present application, the above step S204 specifically includes the following steps:

[0135] The image feature data, speech feature data and brain wave feature data are mapped to a common semantic space according to the cross-modal mapping algorithm, wherein the cross-modal mapping algorithm is expressed as:

[0136]

[0137] in, They represent feature alignment, They represent the feature values ​​of language, image and brain wave respectively, and Z1 and q1 represent the learning parameters of the shared layer.

[0138] In the embodiment of the present application, after the above feature value extraction, different spatial features are mapped to a common semantic space through a cross-modal alignment component, as follows:

[0139]

[0140] in, Representation of feature alignment, They represent the feature values ​​of language, image and brain wave respectively, and Z1 and q1 represent the learning parameters of the shared layer.

[0141] In some optional implementations of the embodiments of the present application, in order to adaptively fuse different modal features together, the present invention proposes a feature fusion component. n , then the posterior representation of each micro-expression sample is:

[0142] P(Y|X)=∑P(Y|X)(P(Z|X))

[0143] Among them, Z represents the potential micro-expression factor, and Y is the micro-expression label. P(Z|X) represents the joint distribution between multimodal features, so its fuzziness is defined as:

[0144] F=KL(p(zt|xt)||∏P(z t |x m ))

[0145] Among them, t indexes different micro-expression factors, and m indexes different modalities. Thus, the higher the F value, the weaker the correlation between different modal features, so more of these features need to be fused. Based on this fuzziness, the fusion weight of multimodal features is defined.

[0146] w m =softmax(-F)

[0147] That is, when the fuzziness F is higher, the corresponding mode is given a greater weight. Finally, the micro-expression features under different modes are weighted fused.

[0148] z=∑w m Z m

[0149] Among them, m represents data of different modes, Z m Represents the feature vector of the mth modal data through feature extraction.

[0150] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0151] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0153] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0154] Embodiment 2

[0155] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a micro-expression recognition device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] like Figure 3 As shown, the micro-expression recognition device 200 of the embodiment of the present application includes:

[0157] The original data acquisition module 210 is used to acquire the original image data, original voice data and original brain wave data to be recognized;

[0158] The preprocessing module 220 is used to perform preprocessing operations on the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data;

[0159] The feature extraction module 230 is used to perform feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data to obtain image feature data, voice feature data and brain wave feature data;

[0160] A feature fusion module 240 is used to perform a feature fusion operation on the image feature data, the speech feature data and the brain wave feature data to obtain a trimodal fusion feature;

[0161] The micro-expression classification module 250 is used to perform micro-expression classification operations on the tri-modal fusion features according to a set classifier to obtain an expression classification result.

[0162] In an embodiment of the present application, a micro-expression recognition device 200 is provided, including: an original data acquisition module 210, used to acquire original image data, original voice data and original brain wave data to be recognized; a preprocessing module 220, used to perform preprocessing operations on the original image data, original voice data and original brain wave data, respectively, to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data; a feature extraction module 230, used to perform feature extraction operations on the preprocessed image data, preprocessed voice data and preprocessed brain wave data, respectively, to obtain image feature data, voice feature data and brain wave feature data; a feature fusion module 240, used to perform feature fusion operations on image feature data, voice feature data and brain wave feature data, to obtain tri-modal fusion features; a micro-expression classification module 250, used to perform micro-expression classification operations on the tri-modal fusion features according to a set classifier, to obtain an expression classification result. Compared with the existing technology, the present application improves the accuracy of micro-expression recognition by fully utilizing the advantages of each modal data through multimodal data fusion; in addition, multimodal data fusion can effectively reduce the impact of environmental factors and improve robustness and generalization ability.

[0163] In some optional implementations of the embodiments of the present application, the preprocessing module includes:

[0164] An image denoising submodule is used to perform an image denoising operation on the original image data according to a Gaussian filtering method to obtain denoised image data;

[0165] The segmentation and cropping submodule is used to perform segmentation and cropping operations on the denoised image data according to the MASK-RCNN detection algorithm to obtain segmented image data;

[0166] An image normalization submodule is used to perform an image normalization operation on the segmented image data to obtain preprocessed image data;

[0167] A framing submodule, used to perform a framing operation on the original voice data according to a preset frame length to obtain framed voice data;

[0168] The speech denoising submodule is used to perform speech denoising operation on the framed speech data according to the short-time energy threshold method to obtain denoised speech data;

[0169] A speech normalization submodule is used to perform speech normalization operation on the denoised speech data to obtain preprocessed speech data;

[0170] The brain wave filtering submodule is used to perform brain wave filtering operation on the original brain wave data according to the high-pass filtering method to obtain filtered brain wave data;

[0171] The EEG denoising submodule is used to perform EEG denoising on the filtered EEG data according to the filtering method to obtain denoised EEG data;

[0172] The EEG normalization submodule is used to perform EEG normalization operation on the denoised EEG data to obtain preprocessed EEG data.

[0173] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0174] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0175] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0176] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the micro-expression recognition method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0177] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the micro-expression recognition method.

[0178] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0179] The computer device provided in the present application fully utilizes the advantages of each modal data through multimodal data fusion, thereby improving the accuracy of micro-expression recognition; in addition, multimodal data fusion can effectively reduce the impact of environmental factors and improve robustness and generalization capabilities.

[0180] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the micro-expression recognition method as described above.

[0181] The computer-readable storage medium provided in the present application fully utilizes the advantages of each modal data through multimodal data fusion, thereby improving the accuracy of micro-expression recognition; in addition, multimodal data fusion can effectively reduce the impact of environmental factors and improve robustness and generalization ability.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0183] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A micro-expression recognition method, characterized in that: The steps include: Acquire original image data, original voice data and original brain wave data to be recognized; Preprocessing the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data; Performing feature extraction operations on the preprocessed image data, the preprocessed voice data, and the preprocessed brain wave data respectively to obtain image feature data, voice feature data, and brain wave feature data; Performing a feature fusion operation on the image feature data, the speech feature data, and the brain wave feature data to obtain a trimodal fusion feature; The micro-expression classification operation is performed on the three-modal fusion features according to the set classifier to obtain the expression classification result.

2. The micro-expression recognition method according to claim 1, characterized in that: The step of performing preprocessing operations on the original image data, the original voice data and the original brain wave data to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data specifically includes the following steps: Performing an image denoising operation on the original image data according to a Gaussian filtering method to obtain denoised image data; Perform segmentation and cropping operations on the denoised image data according to the MASK-RCNN detection algorithm to obtain segmented image data; Performing an image normalization operation on the segmented image data to obtain the preprocessed image data; Performing a frame operation on the original voice data according to a preset frame length to obtain framed voice data; Performing a speech denoising operation on the framed speech data according to a short-time energy threshold method to obtain denoised speech data; Performing a speech normalization operation on the denoised speech data to obtain the preprocessed speech data; Performing an EEG filtering operation on the original EEG data according to a high-pass filtering method to obtain filtered EEG data; Performing brain wave denoising processing on the filtered brain wave data according to a filtering method to obtain denoised brain wave data; Performing an EEG normalization operation on the denoised EEG data to obtain the pre-processed EEG data.

3. The micro-expression recognition method according to claim 1, characterized in that: The step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data to obtain image feature data, voice feature data and brain wave feature data specifically includes the following steps: Extracting color features of the preprocessed image data according to color moments; Extracting texture features of the preprocessed image data according to wavelet transform; Extracting shape features of the preprocessed image data according to the MASK-RCNN detection algorithm; The color feature, the texture feature, and the shape feature are used as the image feature data.

4. The micro-expression recognition method according to claim 1, characterized in that: The step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data to obtain image feature data, voice feature data and brain wave feature data specifically includes the following steps: Extracting speech spectrum features of the preprocessed speech data according to Fourier transform; Extracting instantaneous energy, average energy, and peak energy of the preprocessed speech data to obtain energy features; Extracting speech time domain features of the preprocessed speech data according to an autocorrelation method; The speech spectrum feature, the energy feature and the speech time domain feature are used as the speech feature data.

5. The micro-expression recognition method according to claim 1, characterized in that: The step of performing feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data to obtain image feature data, voice feature data and brain wave feature data specifically includes the following steps: Extracting brain wave spectrum characteristics of the pre-processed brain wave data according to Fourier transform; Extracting the EEG time domain features of the pre-processed EEG data according to the variance analysis method; Extracting event-related potential features of the preprocessed EEG data according to the evt-avg method; The brain wave spectrum characteristics, the brain wave time domain characteristics and the event-related potential characteristics are used as the brain wave characteristic data.

6. The micro-expression recognition method according to claim 1, characterized in that: The step of performing a feature fusion operation on the image feature data, the voice feature data, and the brain wave feature data to obtain a trimodal fusion feature specifically includes the following steps: The image feature data, the speech feature data and the brain wave feature data are mapped to a common semantic space according to a cross-modal mapping algorithm, wherein the cross-modal mapping algorithm is expressed as: in, They represent feature alignment, They represent the feature values ​​of language, image and brain wave respectively, and Z1 and q1 represent the learning parameters of the shared layer.

7. A micro-expression recognition device, characterized in that: include: The original data acquisition module is used to acquire the original image data, original voice data and original brain wave data to be recognized; A preprocessing module, used to perform preprocessing operations on the original image data, the original voice data and the original brain wave data, respectively, to obtain preprocessed image data, preprocessed voice data and preprocessed brain wave data; A feature extraction module, used to perform feature extraction operations on the preprocessed image data, the preprocessed voice data and the preprocessed brain wave data, respectively, to obtain image feature data, voice feature data and brain wave feature data; A feature fusion module, used for performing a feature fusion operation on the image feature data, the speech feature data and the brain wave feature data to obtain a trimodal fusion feature; The micro-expression classification module is used to perform micro-expression classification operations on the three-modal fusion features according to a set classifier to obtain an expression classification result.

8. The micro-expression recognition device according to claim 7, characterized in that: The preprocessing module comprises: An image denoising submodule, used for performing an image denoising operation on the original image data according to a Gaussian filtering method to obtain denoised image data; A segmentation and cropping submodule is used to perform segmentation and cropping operations on the denoised image data according to the MASK-RCNN detection algorithm to obtain segmented image data; An image normalization submodule, used for performing an image normalization operation on the segmented image data to obtain the preprocessed image data; A framing submodule, used for performing a framing operation on the original voice data according to a preset frame length to obtain framed voice data; A speech denoising submodule, used for performing a speech denoising operation on the framed speech data according to a short-time energy threshold method to obtain denoised speech data; A speech normalization submodule, used for performing a speech normalization operation on the denoised speech data to obtain the preprocessed speech data; The brain wave filtering submodule is used to perform a brain wave filtering operation on the original brain wave data according to a high-pass filtering method to obtain filtered brain wave data; The EEG denoising submodule is used to perform EEG denoising processing on the filtered EEG data according to a filtering method to obtain denoised EEG data; The brain wave normalization submodule is used to perform a brain wave normalization operation on the denoised brain wave data to obtain the pre-processed brain wave data.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the micro-expression recognition method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the micro-expression recognition method according to any one of claims 1 to 6 are implemented.

Images