A micro-expression recognition method based on event data driving

Abstract

The application discloses a micro-expression recognition method based on event data driving, and the steps include: 1, preparing training data and corresponding events; 2, constructing an event-driven spatiotemporal motion enhancement network, and introducing an event-enhanced motion extractor module to utilize the high time resolution characteristics of event signals, enhance subtle motion details, and add an event-guided attention module to focus on subtle changes in specific areas and capture more accurate micro-expression spatial features; 3, fusing the features of the two branches to form the final feature representation, and training the constructed micro-expression neural network model; 4, using the trained model for prediction to achieve the goal of micro-expression recognition. The application improves the accuracy of micro-expression recognition by utilizing the ability of event data driving to capture subtle motion changes, so that more accurate classification results are obtained.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a micro-expression recognition method based on event data. Background Technology

[0002] As one of the most challenging topics in affective computing, micro-expression recognition reveals hidden emotions by analyzing involuntary facial movements. It has many potential applications, such as lie detection, sentiment analysis, and clinical diagnosis. However, micro-expressions are spontaneous, subtle, and rapid (1 / 25 to 1 / 3 of a second) facial movements. Traditional RGB cameras struggle to capture these movements completely. Therefore, micro-expression recognition methods using traditional RGB signals may suffer from low frame rates, resulting in lower recognition performance.

[0003] Event signals are captured by novel sensors called event cameras, which possess desirable properties such as high temporal resolution, low latency, and high dynamic range, making them suitable for micro-expression recognition. Recently, some work has focused on micro-expression recognition using events. Some researchers have provided emotion recognition datasets recorded with event cameras. However, their baseline models only use event signals, ignoring the static features provided by RGB images, which may limit the model's accuracy. Some researchers have proposed global-local event feature fusion networks. However, these proposed local event features are only an approximation and differ significantly from actual event signals. Therefore, the lack of temporal and polarity information hinders the network from achieving excellent performance. This invention argues that event signals are not yet fully utilized in micro-expression recognition. Summary of the Invention

[0004] The present invention addresses the shortcomings of the existing technology by proposing an event data-driven micro-expression recognition method. This method aims to improve the accuracy of micro-expression recognition by capturing subtle motion changes through event data, thereby obtaining more accurate micro-expression classification results.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The micro-expression recognition method based on event data of the present invention is characterized by including the following steps:

[0007] Step 1: Obtain the micro-expression image Y and its corresponding event sequence E. Y and E Y Convert to event frames V with multiple time resolutions;

[0008] Step 2: Construct an event-driven micro-expression recognition network, including: an event-enhanced motion extraction module, an event-guided attention module, and a recognition module;

[0009] Step 2.1: Construct an event-enhanced motion extraction module, including: two convolutional layers, a multilayer perceptron, a thinner, and a residual network;

[0010] Step 2.1.1: Take the first micro-expression image y1 of the micro-expression video image set Y as the starting frame, and take the micro-expression image with the highest expression intensity in the micro-expression video image set Y as the vertex frame. Subtract the vertex frame from the starting frame to obtain the original motion frame.

[0011] The original motion frame is processed by a convolutional layer with a kernel size of n1×n1 to obtain the initial motion state X0.

[0012] After the event frame V is processed by the convolutional layer with a kernel size of n2×n2 and the multilayer perceptron, the event representation M is obtained.

[0013] Step 2.1.2: After the refiner processes M and X0, it obtains the final motion state X. B ;

[0014] Step 2.1.3: Transfer the motion state X B The enhanced motion features are input into the residual network for feature extraction, resulting in deep motion features. B ;

[0015] Step 2.2: Construct an event-guided attention module to process V and y1, and obtain the event-guided output feature Output. er ;

[0016] Step 2.3: The recognition module uses equation (8) to obtain the final representation of the micro-expression sequence Y. f :

[0017] Output f =concat(Output) B Output er (8)

[0018] In equation (8), concat represents the concatenation operation;

[0019] The recognition module uses equation (9) to obtain the predicted micro-expression category of the micro-expression image sequence Y.

[0020] p pred =Linear(Output) f (9)

[0021] In equation (9), Linear(·) represents a linear classification function, and C represents the number of categories for micro-expression recognition;

[0022] Step 3: Construct the cross-entropy loss function L and train the micro-expression recognition network using the gradient descent algorithm until the cross-entropy loss function L converges, thereby obtaining a trained micro-expression classification model for recognizing input micro-expression images.

[0023] The micro-expression recognition method based on event data driven by the present invention is characterized in that step 1 includes the following steps:

[0024] Step 1.1: Obtain the micro-expression video image set Y = {y1, y2, ..., y...} i ,…,y N}, where y i Let Y represent the i-th micro-expression image, i = 1, 2, ..., N, where N is the total number of micro-expression images; let y be the set of true micro-expression categories corresponding to Y. label ;

[0025] Step 1.2, at a given timestamp t i Inside, when y i When the brightness change exceeds a preset threshold c, the i-th micro-expression image y is recorded. i The corresponding asynchronous event stream is E. i Thus, the event sequence E of the micro-expression video image set Y is obtained. Y ={E1,...,E i ,...,E N};

[0026] Step 1.3: Set the event sequence E Y Convert to event frames with multiple time resolutions Among them, V j Let B represent the j-th event frame, B represent the number of time intervals used in the bilinear temporal interpolation, and H and W represent the height and width of the event frame, respectively.

[0027] Step 2.12 includes the following steps:

[0028] Step a: Initialize j = 0;

[0029] Step b: The refiner uses equation (1) to obtain the (j+1)th update ratio z. j+1 and the (j+1)th intermediate output r j+1 :

[0030]

[0031] In equation (1), σ represents the sigmoid function, and Conv1 and Conv2 represent two convolutional layers in the thinner with kernels of n1×n1; W z W rThese represent the parameters of convolutional layers Conv1 and Conv2 in the thinner, respectively; h j This represents the j-th hidden state. When j = 0, h j =X0;

[0032] Step c: Use equation (2) to obtain the intermediate variables of the (j+1)th hidden state.

[0033]

[0034] In equation (2), tanh represents the tanh activation function, ⊙ represents element-wise multiplication, Conv3 represents the convolutional layer in the thinner with a kernel of n1×n1, and W m The parameters represent the convolutional layer Conv3;

[0035] Step d: Use equation (3) to obtain the (j+1)th hidden state h. j+1 :

[0036]

[0037] Step e: Use equation (4) to obtain the j-th motion direction Δ j :

[0038] Δ j =Conv4(relu(Conv4(h j+1 )))+h j+1 (4)

[0039] In equation (4), ReLU is the activation function, and Conv4 represents a convolutional layer with an n2×n2 kernel in the thinner;

[0040] Step f: Use equation (5) to obtain the (j+1)th motion state X. j+1 :

[0041] X j+1 =X j +Δ j (5)

[0042] Step g: After assigning j+1 to j, return to step b and execute sequentially until j = B-1, thus obtaining the motion state {X}. j} j∈[0:B-1] .

[0043] Step 2.2 includes the following steps:

[0044] The event-guided attention module includes: a visual converter, a fusion module, and a multi-head attention module;

[0045] Step 2.2.1: Transfer the event frame {V} j}j∈[0:B-1] As query vector Q e Key vector K e Sum vector V e The data is then input into a visual converter for processing, thereby obtaining the attention map Attn of the event using equation (4). e :

[0046]

[0047] In equation (4), σ represents the sigmoid function, T represents the transpose, and D represents the dimension of the key vector;

[0048] Step 2.2.2: Take the first micro-expression image y1 of the micro-expression image sequence Y as the corresponding query vector Q. r Key vector K r Value vector V r The image is then input into a visual converter for processing, thereby obtaining the attention map Attn of the micro-expression image using equation (5). r ;

[0049]

[0050] Step 2.2.3: The fusion module uses equation (6) to obtain the new event-guided attention map Attn. g :

[0051] Attn g =α·(Attn) e ⊙Attn r )+(1-α)·Attn r (6)

[0052] In equation (6), α represents the parameter set;

[0053] Step 2.2.4: The multi-head attention module uses equation (7) to calculate the output characteristics of the event-guided improved multi-head attention. er :

[0054] Output er =softmax(Attn) g V r (7)

[0055] In equation (7), softmax is the softmax activation function.

[0056] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the spatiotemporal motion-enhanced micro-expression recognition method, and the processor is configured to execute the program stored in the memory.

[0057] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the spatiotemporal motion-enhanced micro-expression recognition method.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1. This invention designs an event-driven spatiotemporal motion-enhanced micro-expression recognition network, which integrates event data into the micro-expression recognition task and effectively utilizes the internal relationship between two different data modes, thereby improving the accuracy and robustness of the micro-expression classification model.

[0060] 2. This invention proposes an event-enhanced motion extractor module to enhance the amplitude of micro-expression movements. By utilizing events with high temporal resolution, the original motion is gradually optimized and refined, enhancing the motion representation and thus enabling more effective modeling of subtle local changes occurring in micro-expressions.

[0061] 3. This invention also creates an event-guided attention module, which utilizes the unique sparsity that events mainly occur in areas of brightness change. This allows the spatially guided network to focus on areas of subtle change while learning global features, thereby improving the highlighting of subtle changes in micro-expressions and enhancing the accuracy and robustness of recognition.

[0062] 4. This invention integrates temporal local representations with minute movements with spatial global representations that cover the entire face but focus on areas of change, resulting in a more robust set of micro-expression representations. This enhances the model's ability to learn global and local information and improves accuracy.

[0063] 5. This invention introduces event data into RGB images and inputs it into a dual-branch network based on a visual converter and a convolutional network. It utilizes the rich spatial and static information in RGB frames, as well as the high dynamic range and temporal resolution of event streams, to comprehensively model the subtle changes in micro-expressions and achieve efficient and accurate classification of micro-expressions. Attached Figure Description

[0064] Figure 1 This is a schematic flowchart of the method of the present invention;

[0065] Figure 2 This is a structural diagram of the event-enhanced motion extractor module of the present invention. Detailed Implementation

[0066] In this embodiment, an event-data-driven micro-expression recognition method fuses and propagates data from both event and frame image modalities within a spatiotemporal motion-enhanced neural network. Figure 1 As shown, the method includes the following steps:

[0067] Step 1: Obtain micro-expression images and their corresponding event sequences:

[0068] Step 1.1: Obtain the micro-expression video image set Y = {y1, y2, ..., y i ,…,y N}, where y i Let Y represent the i-th micro-expression image, i = 1, 2, ..., N, where N is the total number of micro-expression images; let y be the set of true micro-expression categories corresponding to Y. label ;

[0069] Step 1.2, at a given timestamp t i Inside, when y i When the brightness change exceeds a preset threshold c, the i-th micro-expression image y is recorded. i The corresponding asynchronous event stream is Where, x ik ,t ik ,t ik ,p ik Let n represent the x-coordinate, y-coordinate, timestamp, and polarity of the k-th event in the i-th sample, respectively. i This represents the total number of events contained in the i-th sample; in this example, the event camera simulator ESIM is used to directly simulate the event data of the input video image set Y, thereby obtaining the event sequence E of the micro-expression video image set Y. Y ={E1,...,E i ,...,E N};

[0070] Step 1.3: Set the event sequence E Y Convert to event frames with multiple temporal resolutions Among them, V j Let B represent the j-th event frame, B represent the number of time intervals used in the bilinear temporal interpolation, and H and W represent the height and width of the event frame, respectively.

[0071] Step 2: Construct an event-driven micro-expression recognition network, including: an event-enhanced motion extraction module, an event-guided attention module, and a recognition module;

[0072] Step 2.1: Construct an event-enhanced motion extraction module, such as... Figure 2 As shown, it includes: two convolutional layers, a multilayer perceptron, a thinner, and a residual network;

[0073] Step 2.1.1: Take the first micro-expression image y1 of the micro-expression video image set Y as the starting frame, and take the micro-expression image with the highest expression intensity in the micro-expression video image set Y as the vertex frame. Subtract the vertex frame from the starting frame to obtain the original motion frame.

[0074] The original motion frame is processed by a convolutional layer with a kernel size of n1×n1 to obtain the initial motion state X0;

[0075] Event Frame {V j} j∈[0:B-1] After processing by a convolutional layer with an n2×n2 kernel and a multilayer perceptron, the event representation {M} is obtained. j} j∈[0:B-1] This adapts it to the refiner. M j This represents the j-th event.

[0076] Step 2.1.2: Initialize j = 0;

[0077] Step 2.1.3: The refiner uses equation (1) to obtain the (j+1)th update ratio z. j+1 and the (j+1)th intermediate output r j+1 :

[0078]

[0079] In equation (1), σ represents the sigmoid function, and Conv1 and Conv2 represent two convolutional layers in the thinner with kernels of n1×n1; W z W r These represent the parameters of convolutional layers Conv1 and Conv2 in the thinner, respectively; h j This represents the j-th hidden state. When j = 0, h j =X0. The z calculated in this example j+1 H represents the hidden state. j and the output M of the sensor j The degree of importance, with a value between 0 and 1;

[0080] Using equation (2), we obtain the intermediate variables of the (j+1)th hidden state.

[0081]

[0082] In equation (2), tanh represents the tanh activation function, ⊙ represents element-wise multiplication, Conv3 represents the convolutional layer in the thinner with a kernel of n1×n1, and W m The parameters represent the convolutional layer Conv3;

[0083] The (j+1)th hidden state h is obtained using equation (3).j+1 :

[0084]

[0085] The j-th motion direction Δ is obtained using equation (4). j :

[0086] Δ j =Conv4(relu(Conv4(h j+1 )))+h j+1 (4)

[0087] In equation (4), ReLU is the activation function, and Conv4 represents a convolutional layer with an n2×n2 kernel in the thinner.

[0088] The (j+1)th motion state X is obtained using equation (5). j+1 :

[0089] X j+1 =X j +Δ j (5)

[0090] Step 2.1.4: After assigning j+1 to j, return to step 2.1.3 and execute sequentially until j = B-1, thus obtaining the motion state {X}. j} j∈[0:B-1] ;

[0091] Step 2.1.5: Set the Bth motion state X B The enhanced motion features are input into the residual network for feature extraction, resulting in deep motion features. B .

[0092] Step 2.2: Construct an event-guided attention module, such as... Figure 1 As shown, it includes: a visual converter, a fusion module, and a multi-head attention module;

[0093] Step 2.2.1: Transfer the event frame {V} j} j∈[0:B-1] As query vector Q e Key vector K e Sum vector V e The data is then input into a visual converter for processing, thereby obtaining the attention map Attn of the event using equation (4). e :

[0094]

[0095] In equation (4), σ represents the sigmoid function, T represents the transpose, and D represents the dimension of the key vector;

[0096] Step 2.2.2: Take the first micro-expression image y1 of the micro-expression image sequence Y as the corresponding query vector Q. r Key vector K r Value vector V r The image is then input into a visual converter for processing, thereby obtaining the attention map Attn of the micro-expression image using equation (5). r ;

[0097]

[0098] Step 2.2.3: In this example, the sparsity of events is used to guide the attention of the RGB image to key regions. The fusion module uses equation (6) to obtain a new attention map Attn guided by the events. g :

[0099] Attn g =α·(Attn) e ⊙Attn r )+(1-α)·Attn r (6)

[0100] In equation (6), α represents the parameter set. In this example, α = 0.5.

[0101] Step 2.2.4: The multi-head attention module uses equation (7) to calculate the output characteristics of the event-guided improved multi-head attention. er :

[0102] Output er =softmax(Attn) g V r (7)

[0103] In equation (7), softmax is the softmax activation function.

[0104] Step 2.3: The recognition module uses equation (8) to obtain the final representation of the micro-expression sequence Y. f :

[0105] Output f =concat(Output) B Output er (8)

[0106] In equation (8), concat represents the concatenation operation;

[0107] The recognition module uses equation (9) to obtain the predicted micro-expression category of the micro-expression image sequence Y.

[0108] p pred=Linear(Output) f (9)

[0109] In equation (9), Linear(·) represents a linear classification function, and C represents the number of categories for micro-expression recognition.

[0110] Step 3: Construct the cross-entropy loss function L and train the micro-expression recognition network using the gradient descent algorithm until the cross-entropy loss function L converges, thereby obtaining a trained micro-expression classification model for recognizing input micro-expression images.

[0111] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0112] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A micro-expression recognition method based on event data, characterized in that, Includes the following steps: Step 1: Obtain the micro-expression image Y and its corresponding event sequence E. Y and E Y Convert to event frames V with multiple time resolutions; Step 2: Construct an event-driven micro-expression recognition network, including: an event-enhanced motion extraction module, an event-guided attention module, and a recognition module; Step 2.1: Construct an event-enhanced motion extraction module, including: two convolutional layers, a multilayer perceptron, a thinner, and a residual network; Step 2.1.1: Take the first micro-expression image y1 of the micro-expression video image set Y as the starting frame, and take the micro-expression image with the highest expression intensity in the micro-expression video image set Y as the vertex frame. Subtract the vertex frame from the starting frame to obtain the original motion frame. The original motion frame is processed by a convolutional layer with a kernel size of n1×n1 to obtain the initial motion state X0. After the event frame V is processed by the convolutional layer with a kernel size of n2×n2 and the multilayer perceptron, the event representation M is obtained. Step 2.1.2: After the refiner processes M and X0, it obtains the final motion state X. B ; Step 2.1.3: Transfer the motion state X B The enhanced motion features are input into the residual network for feature extraction, resulting in deep motion features. B ; Step 2.2: Construct an event-guided attention module to process V and y1, and obtain the event-guided output feature Output. er ; Step 2.3: The recognition module uses equation (8) to obtain the final representation of the micro-expression sequence Y. f : Output f =concat(Output B ,Output er ) (8) In equation (8), concat represents the concatenation operation; The recognition module uses equation (9) to obtain the predicted micro-expression category of the micro-expression image sequence Y. p pred =Linear(Output f ) (9) In equation (9), Linear(·) represents a linear classification function, and C represents the number of categories for micro-expression recognition; Step 3: Construct the cross-entropy loss function L and train the micro-expression recognition network using the gradient descent algorithm until the cross-entropy loss function L converges, thereby obtaining a trained micro-expression classification model for recognizing input micro-expression images.

2. The micro-expression recognition method based on event data as described in claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Obtain the micro-expression video image set Y = {y1, y2, ..., y...} i ,…,y N }, where y i Let Y represent the i-th micro-expression image, i = 1, 2, ..., N, where N is the total number of micro-expression images; let y be the set of true micro-expression categories corresponding to Y. label ; Step 1.2, at a given timestamp t i Inside, when y i When the brightness change exceeds a preset threshold c, the i-th micro-expression image y is recorded. i The corresponding asynchronous event stream is E. i Thus, the event sequence E of the micro-expression video image set Y is obtained. Y ={E1,...,E i ,...,E N }; Step 1.3: Set the event sequence E Y Convert to event frames with multiple time resolutions Among them, V j Let B represent the j-th event frame, B represent the number of time intervals used in the bilinear temporal interpolation, and H and W represent the height and width of the event frame, respectively.

3. The micro-expression recognition method based on event data as described in claim 2, characterized in that, Step 2.1.2 includes the following steps: Step a: Initialize j = 0; Step b: The refiner uses equation (1) to obtain the (j+1)th update ratio z. j+1 and the (j+1)th intermediate output r j+1 : In equation (1), σ represents the sigmoid function, and Conv1 and Conv2 represent two convolutional layers in the thinner with kernels of n1×n1; W z W r These represent the parameters of convolutional layers Conv1 and Conv2 in the thinner, respectively; h j This represents the j-th hidden state. When j = 0, h j =X0; Step c: Use equation (2) to obtain the intermediate variables of the (j+1)th hidden state. In equation (2), tanh represents the tanh activation function, ⊙ represents element-wise multiplication, Conv3 represents the convolutional layer in the thinner with a kernel of n1×n1, and W m The parameters represent the convolutional layer Conv3; Step d: Use equation (3) to obtain the (j+1)th hidden state h. j+1 : Step e: Use equation (4) to obtain the j-th motion direction Δ j : Δ j =Conv4(relu(Conv4(h j+1 )))+h j+1 (4) In equation (4), ReLU is the activation function, and Conv4 represents a convolutional layer with an n2×n2 kernel in the thinner; Step f: Use equation (5) to obtain the (j+1)th motion state X. j+1 : X j+1 =X j +Δ j (5) Step g: After assigning j+1 to j, return to step b and execute sequentially until j = B-1, thus obtaining the motion state {X}. j } j∈[0:B-1] .

4. The micro-expression recognition method based on event data as described in claim 3, characterized in that, Step 2.2 includes the following steps: The event-guided attention module includes: a visual converter, a fusion module, and a multi-head attention module; Step 2.2.1: Transfer the event frame {V} j } j∈[0:B-1] As query vector Q e Key vector K e Sum vector V e The data is then input into a visual converter for processing, thereby obtaining the attention map Attn of the event using equation (4). e : In equation (4), σ represents the sigmoid function, T represents the transpose, and D represents the dimension of the key vector; Step 2.2.2: Take the first micro-expression image y1 of the micro-expression image sequence Y as the corresponding query vector Q. r Key vector K r Value vector V r The image is then input into a visual converter for processing, thereby obtaining the attention map Attn of the micro-expression image using equation (5). r ; Step 2.2.3: The fusion module uses equation (6) to obtain the new event-guided attention map Attn. g : Attn g =α·(Attn e ⊙Attn r )+(1-α)·Attn r (6) In equation (6), α represents the parameter set; Step 2.2.4: The multi-head attention module uses equation (7) to calculate the output characteristics of the event-guided improved multi-head attention. er : Output er =softmax(Attn g )V r (7) In equation (7), softmax is the softmax activation function.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the micro-expression recognition methods of claims 1-4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the micro-expression recognition methods described in claims 1-4.

Images