A micro-expression recognition method, system, device and storage medium based on multivariate optical flow feature fusion

Abstract

The present invention discloses a micro-expression recognition method, system, device and storage medium based on multivariate optical flow feature fusion. The method includes obtaining original video, generating optical flow images respectively through multiple optical flow algorithms, extracting horizontal optical flow vectors and vertical optical flow vectors therein for superposition, obtaining horizontal images, vertical images and overall images, and inputting them into the MRKCBN network model for feature extraction, outputting corresponding image output features to obtain local features; inputting the optical flow image generated by one of the optical flow algorithms into the VIT network model for feature extraction and outputting global features, splicing local features and global features to obtain splicing features; fusing the splicing features through the APSB module to obtain fused features; and classifying the fused features through the fully connected layer to obtain micro-expression recognition results. The present invention uses multiple optical flow algorithms to fuse optical flow information to improve the accuracy of recognition; and takes into account both local features and global features through MRKCBN and APSB.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer micro-expression recognition, and specifically relates to a micro-expression recognition method, system, device and storage medium based on multivariate optical flow feature fusion. Background Art

[0002] Micro-expression recognition has been a research area that has garnered considerable attention in recent years. It involves a deep understanding of human emotion and cognition and has applications in many fields, such as psychology, medicine, and security monitoring. In this field, researchers have proposed various micro-expression recognition methods based on optical flow, which is derived from brightness differences between consecutive frames and can estimate subtle facial movements.

[0003] Existing micro-expression recognition methods have the following defects:

[0004] (1) Although optical flow is used to obtain facial motion differences, the optical flow algorithm is not fully utilized. When the face is affected by problems such as lighting changes, facial occlusion, or inconsistent expression intensity, its recognition accuracy is greatly reduced.

[0005] (2) The channel attention mechanism in the CBAM module used in existing micro-expression recognition methods uses MLP to compress channels. However, MLP is prone to falling into local optimal solutions rather than global optimal solutions, and MLP cannot achieve the approximation ability, accuracy, and interpretability of KAN. At the same time, the spatial attention mechanism uses ordinary 2D convolution to extract features from the spliced ​​features. The limitation of 2D convolution is that it only captures information within the local domain, which makes it difficult to understand global features.

[0006] (3) The convolutional model used in existing micro-expression recognition methods is good at capturing the spatial hierarchical structure of images, such as low-level features such as edges, textures, and shapes, and is also good at extracting higher-level semantic information, but its ability to obtain global information is not ideal; the VIT model outputs global information that captures the image, but it lacks a spatial hierarchical structure, and when the data scale is limited, the VIT model is prone to overfitting. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a micro-expression recognition method, system, device and storage medium based on multivariate optical flow feature fusion, which improves the accuracy of recognition by fusing optical flow information through multiple optical flow algorithms; and takes into account both local and global features through MRKCBN and APSB.

[0008] The present invention provides the following technical solutions:

[0009] In a first aspect, a micro-expression recognition method based on multivariate optical flow feature fusion is provided, comprising: obtaining an original video, pre-processing the original video, generating optical flow images respectively through a plurality of optical flow algorithms, extracting the horizontal optical flow vector and the vertical optical flow vector in the optical flow image respectively, superimposing the horizontal optical flow vector, the vertical optical flow vector and the optical flow image of each optical flow algorithm to obtain a horizontal image, a vertical image and an overall image; inputting the horizontal image, the vertical image and the overall image into a pre-built MRKCBN network model for feature extraction, outputting corresponding image output features, processing the image output features to obtain local features; inputting the optical flow image generated by one of the plurality of optical flow algorithms into a pre-acquired VIT network model for feature extraction, outputting global features, and splicing the local features and the global features to obtain spliced ​​features; fusing the spliced ​​features through an APSB module to obtain fused features; passing the fused features through a layer normalization module and a multi-layer perceptron in sequence, and then classifying through a fully connected layer to obtain a micro-expression recognition result.

[0010] As a preferred technical solution of the present invention, the original video is obtained, and optical flow images are generated respectively through multiple optical flow algorithms after preprocessing the original video, including: cutting the original video into multiple video sequences, performing Euler video amplification on each video sequence, and cutting it into frame images; for the amplified frame images, taking the initial frame of the video sequence as the starting frame image, calculating the optical flow difference between adjacent frame images, and taking the frame with the most drastic optical flow change as the peak frame image; based on the starting frame image and the peak frame image, optical flow images are generated respectively using optical flow algorithms.

[0011] As a preferred technical solution of the present invention, the multiple optical flow algorithms include TVL1, DIS and PCA.

[0012] As an optimal technical solution of the present invention, the construction process of the MRKCBN network model includes: based on the original CBAM module including the spatial attention module and the channel attention module, the neural network structure KAN is integrated into the CBAM module to obtain a KCBAM module; the KCBAM module is combined with the residual block to obtain the RKCB module; the RKCB module is combined with the original MobileNet to obtain the MRKCBN network model.

[0013] As a preferred technical solution of the present invention, the KCBAM module includes a channel attention module KCAM and a spatial attention module KSAM;

[0014] The linear module in the original channel attention module is replaced by the KAN Linear module to obtain a new channel attention module KCAM, which is expressed as:

[0015]

[0016] in, represents the input sequence of KCAM, represents the average pooling operation, represents the maximum pooling operation, represents the activation function, KAN represents the KAN network structure, represents the output sequence of KCAM;

[0017] The convolution module in the original spatial attention module is replaced by the KAN_Conv2d convolution module to obtain a new spatial attention module KSAM, which is expressed as:

[0018] M s ( F 2 ) = σ ( K A N C o n v 7 × 7 ( [ A v g P o o l ( F 2 ) ; M a x P o o l ( F 2 ) ] ) )

[0019] in, represents the input sequence of KSAM, represents the convolution operation, Represents the output sequence of KSAM.

[0020] As a preferred technical solution of the present invention, the horizontal image, the vertical image and the overall image are respectively input into a pre-built MRKCBN network model for feature extraction, the corresponding image output features are output, and the image output features are processed to obtain local features, including:

[0021] The image output features corresponding to the horizontal image, the vertical image, and the overall image are spliced, and convolution and maximum pooling operations are performed on the spliced ​​features to obtain the local features.

[0022] As a preferred technical solution of the present invention, the step of fusing the spliced ​​features through the APSB module to obtain the fused features includes:

[0023] Initialize a linear layer to get the initial attention score, expressed as:

[0024]

[0025] in, represents the i-th element in the concatenated feature sequence, W and b represent the weight and bias of the linear layer respectively, represents the initial attention score of the i-th element;

[0026] Normalize the initial attention score and express it as:

[0027]

[0028] in, represents the activation function, represents the normalized attention score of the i-th element, I represents the total number of elements, and exp represents the exponential function;

[0029] The key vector K is aggregated through the attention pooling operation to obtain K′, which is expressed as:

[0030]

[0031] in, represents the attention pooling operation, X represents the concatenated feature sequence;

[0032] Feature fusion is performed based on the value vector V, query vector Q and K′, which is expressed as:

[0033]

[0034] in, Indicates fusion features.

[0035] In a second aspect, a micro-expression recognition system based on multivariate optical flow feature fusion is provided, comprising: an optical flow extraction module for acquiring an original video, pre-processing the original video, generating optical flow images using a plurality of optical flow algorithms, extracting horizontal optical flow vectors and vertical optical flow vectors from the optical flow images, and superimposing the horizontal optical flow vectors, vertical optical flow vectors, and optical flow images of each optical flow algorithm to obtain a horizontal image, a vertical image, and an overall image;

[0036] A local feature extraction module is used to input the horizontal image, vertical image and overall image into a pre-built MRKCBN network model for feature extraction, output corresponding image output features, and process the image output features to obtain local features;

[0037] A global feature extraction module is used to input an optical flow image generated by one of the multiple optical flow algorithms into a pre-acquired VIT network model for feature extraction, output a global feature, and splice the local feature and the global feature to obtain a spliced ​​feature;

[0038] A feature fusion module, configured to fuse the splicing features through an APSB module to obtain fused features;

[0039] The classification module is used to pass the fusion features through the layer normalization module and the multi-layer perceptron in sequence, and then classify them through the fully connected layer to obtain the micro-expression recognition results.

[0040] In a third aspect, a micro-expression recognition device based on multivariate optical flow feature fusion is provided, comprising a processor and a storage medium; the storage medium is used to store instructions; and the processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention provides a micro-expression recognition method based on multivariate optical flow feature fusion, which uses multiple optical flow algorithms to fuse optical flow information, thereby enhancing the robustness of the model to environmental changes or expression changes. At the same time, horizontal optical flow vectors and vertical optical flow vectors are extracted separately, further improving the recognition accuracy. In complex facial expressions, horizontal optical flow vectors and vertical optical flow vectors can provide additional clues to help distinguish real expressions from disguised expressions, thereby reducing recognition errors.

[0044] 2. The MRKCBN network model adopted in the micro-expression recognition method based on multivariate optical flow feature fusion provided by the present invention integrates the KAN module into the original CBAM module, which not only improves the accuracy of the model in data fitting and solving partial differential equations, but also enhances the interpretability and visualization capabilities of the model, and improves the model's ability to capture global features.

[0045] 3. The present invention provides a micro-expression recognition method based on multivariate optical flow feature fusion. The APSB module finally adopts the features extracted by the convolution model and the VIT model to fuse the features, taking into account both local features and global features, thereby improving the accuracy of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the structure of the KCBAM module in an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of the structure of the KCAM module in an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the structure of the KSAM module in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the structure of the RKCB module in an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the structure of the MRKCBN model in an embodiment of the present invention;

[0051] Figure 6 3 is a structural diagram of a micro-expression recognition method based on multivariate optical flow feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0053] Example 1

[0054] This embodiment provides a micro-expression recognition method based on multivariate optical flow feature fusion. The method specifically includes the following steps:

[0055] Step 1: Obtain the original video, preprocess the original video, generate optical flow images using multiple optical flow algorithms, and extract the horizontal optical flow vector and vertical optical flow vector in the optical flow image respectively. Superimpose the horizontal optical flow vector, vertical optical flow vector and optical flow image of each optical flow algorithm to obtain the horizontal image, vertical image and overall image.

[0056] Step 1.1: Cut the original video into multiple video sequences, perform Euler video upscaling on each video sequence, and cut it into frame images.

[0057] Step 1.2: For the amplified frame image, take the initial frame of the video sequence as the starting frame image, calculate the optical flow difference between adjacent frame images, and take the frame with the most drastic optical flow change as the peak frame image.

[0058] Step 1.3: Based on the starting frame image and the peak frame image, generate an optical flow image using an optical flow algorithm. In this embodiment, the multiple optical flow algorithms include TVL1, DIS, and PCA.

[0059] Step 1.4: Extract the horizontal and vertical optical flow vectors from the optical flow image, and superimpose the horizontal and vertical optical flow vectors and the optical flow image for each optical flow algorithm to obtain a horizontal image, a vertical image, and an overall image. In this embodiment, the horizontal image, the vertical image, and the overall image are denoted as TVL1_DIS_PCA(u), TVL1_DIS_PCA(v), and TVL1_DIS_PCA, respectively.

[0060] Step 2: Input the horizontal image, vertical image and overall image into the pre-built MRKCBN network model for feature extraction, output the corresponding image output features, and process the image output features to obtain local features.

[0061] The construction process of the MRKCBN network model specifically includes:

[0062] Step 1: Based on the original CBAM (Convolutional Block Attention Module) module consisting of the spatial attention module and the channel attention module, the neural network structure KAN (Kolmogorov-Arnold Networks) is integrated into the CBAM module to obtain the KCBAM (KAN Convolutional Block Attention Module) module.

[0063] like Figure 1 As shown, the KCBAM module includes a channel attention module (KAN Channel Attention Module) KCAM and a spatial attention module (KAN Spatial Attention Module) KSAM.

[0064] like Figure 2 As shown in Figure 1, the linear module in the original channel attention module is replaced by the KAN Linear (KAN Networks Linear, KAN linear module) module to obtain a new channel attention module KCAM, which is expressed as:

[0065]

[0066] in, represents the input sequence of KCAM, represents the average pooling operation, represents the maximum pooling operation, represents the activation function, KAN represents the KAN network structure, Represents the output sequence of KCAM.

[0067] In this embodiment, the KAN Linear module uses the B-spline Function and Functions are used to approximate traditional activation functions, providing an adaptive linear layer that can dynamically adjust its behavior according to the input data and control the complexity of the model through regularization loss.

[0068] like Figure 3 As shown in Figure 1, the convolution module in the original spatial attention module is replaced by the KAN_Conv2d convolution module to obtain a new spatial attention module KSAM, which is expressed as:

[0069] M s ( F 2 ) = σ ( K A N C o n v 7 × 7 ( [ A v g P o o l ( F 2 ) ; M a x P o o l ( F 2 ) ] ) )

[0070] in, represents the input sequence of KSAM, represents the convolution operation, Represents the output sequence of KSAM.

[0071] In this embodiment, KAN_Conv2d uses B-splines to approximate the traditional activation function, and implements a more flexible and powerful convolution layer through features such as grouped convolution and normalization, so that KCBAM can better adapt to the characteristics of the data and improve the expressiveness and generalization performance of the model.

[0072] Step 2: Combine the KCBAM module with the residual block to obtain the RKCB (Residual KAN CBAMBlock, Residual KAN Convolutional Neural Network Attention Module) module.

[0073] like Figure 4 As shown in the figure, the improved KCBAM module is combined with the residual block, so that it can be used as a lightweight module in MobileNet, which is already a lightweight network, combining channel and spatial information, thereby greatly improving the performance of the model.

[0074] Step 3: Combine the RKCB module with the original MobileNet to obtain the MRKCBN (Mobile Residual KANCBAM Block Net, movable residual KAN convolutional neural network attention module network) network model.

[0075] like Figure 5 As shown, in this embodiment, the above-mentioned RKCB module is used on the basis of MobileNet V3 to combine channel and spatial information to improve model performance.

[0076] like Figure 6 As shown, step 2 is based on the above-constructed MRKCBN network model, and the image output features corresponding to each of the horizontal image, the vertical image, and the overall image are spliced, and the spliced ​​features are subjected to convolution and maximum pooling operations to obtain the local features. In this embodiment, TVL1_DIS_PCA(u), TVL1_DIS_PCA(v), and TVL1_DIS_PCA are respectively input into the above-mentioned MRKCBN as three channels, and the image output features extracted from the three channels are concat spliced, and the spliced ​​features are subjected to convolution and maximum pooling operations to obtain the local features, which can reduce the number of parameters and the amount of calculation while extracting features.

[0077] Step 3: Input the optical flow image generated by one of the multiple optical flow algorithms into the pre-acquired VIT network model for feature extraction, output the global features, and splice the local features and the global features to obtain spliced ​​features.

[0078] like Figure 6 As shown, in this embodiment, the optical flow image generated by the TVL1 optical flow algorithm, which has been well-received in the traditional field of micro-expression detection, is used as the data input for the VIT (vision transformer) network model. The optical flow image generated by this method is denoted as TVL1.

[0079] Step 4: The spliced ​​features are fused through the APSB (Attention Pooling Self-attention Block) module to obtain fused features.

[0080] Compared with the original multi-head self-attention mechanism, APSB has a better understanding of Q and K. T Directly perform dot product operations and propose an attention pooling-based algorithm to calculate the attention score. This module can reduce the dimension of the key vector to reduce computational complexity; extract key features from the key vector to enhance the model's ability to capture important information; and filter out some unimportant information to highlight key information. The specific steps are as follows:

[0081] Step 4.1. Initialize a linear layer to get the initial attention score, expressed as:

[0082]

[0083] in, represents the i-th element in the concatenated feature sequence, W and b represent the weight and bias of the linear layer respectively, represents the initial attention score of the i-th element.

[0084] Step 4.2: Normalize the initial attention score and express it as:

[0085]

[0086] in, represents the activation function, represents the normalized attention score of the i-th element, I represents the total number of elements, and exp represents the exponential function.

[0087] Step 4.3: Aggregate the key vector K through the attention pooling operation to obtain K′, which is expressed as:

[0088]

[0089] in, represents the attention pooling operation, and X represents the concatenated feature sequence.

[0090] Step 4.4: Perform feature fusion based on the value vector V, query vector Q, and K′, expressed as:

[0091]

[0092] in, Indicates fusion features.

[0093] Step 5: The fused features are sequentially passed through the layer normalization (LayerNorm) module and the multi-layer perceptron (MLP), and then classified through the fully connected layer (FC) to obtain the micro-expression recognition results.

[0094] The layer normalization module improves model stability and generalization capabilities, while reducing the model's reliance on initialization weights and simplifying hyperparameter selection. Multilayer perceptrons, by introducing nonlinear activation functions (such as ReLU), empower the model with powerful capabilities to learn and simulate complex function mappings. They also extract useful features from normalized activation values ​​for further feature extraction. The layer normalization module reduces instability and the risk of overfitting during early training, while the multilayer perceptron effectively learns data features. The combination of these two improves efficiency.

[0095] Example 2

[0096] This embodiment provides a micro-expression recognition system based on multivariate optical flow feature fusion, including:

[0097] The optical flow extraction module is used to obtain the original video, generate optical flow images through multiple optical flow algorithms after preprocessing the original video, and extract the horizontal optical flow vectors and vertical optical flow vectors in the optical flow images respectively. The horizontal optical flow vectors, vertical optical flow vectors and optical flow images of each optical flow algorithm are superimposed to obtain the horizontal image, vertical image and overall image.

[0098] The local feature extraction module is used to input the horizontal image, vertical image and overall image into the pre-built MRKCBN network model for feature extraction, output corresponding image output features, and process the image output features to obtain local features.

[0099] The global feature extraction module is used to input the optical flow image generated by one of the multiple optical flow algorithms into the pre-acquired VIT network model for feature extraction, output global features, and splice the local features and global features to obtain spliced ​​features.

[0100] The feature fusion module is used to fuse the splicing features through the APSB module to obtain fused features.

[0101] The classification module is used to pass the fusion features through the layer normalization module and the multi-layer perceptron in sequence, and then classify them through the fully connected layer to obtain the micro-expression recognition results.

[0102] Example 3

[0103] This embodiment provides a micro-expression recognition device based on multivariate optical flow feature fusion, including a processor and a storage medium. The storage medium is configured to store instructions. The processor is configured to operate according to the instructions to execute the steps of the micro-expression recognition method based on multivariate optical flow feature fusion described in Example 1.

[0104] Example 4

[0105] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the micro-expression recognition method based on multivariate optical flow feature fusion described in Example 1 are implemented.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A micro-expression recognition method based on multivariate optical flow feature fusion, characterized in that: include: Obtain the original video, generate optical flow images respectively through multiple optical flow algorithms after preprocessing the original video, extract the horizontal optical flow vector and vertical optical flow vector in the optical flow image respectively, superimpose the horizontal optical flow vector, vertical optical flow vector and optical flow image of each optical flow algorithm to obtain the horizontal image, vertical image and overall image; The horizontal image, the vertical image and the overall image are respectively input into the pre-built MRKCBN network model for feature extraction, the corresponding image output features are output, and the image output features are processed to obtain local features; Inputting an optical flow image generated by one of the multiple optical flow algorithms into a pre-acquired VIT network model for feature extraction, outputting a global feature, and splicing the local feature with the global feature to obtain a spliced ​​feature; The splicing features are subjected to feature fusion through the APSB module to obtain fusion features; The fused features are sequentially passed through a layer normalization module and a multi-layer perceptron, and then classified through a fully connected layer to obtain a micro-expression recognition result; The construction process of the MRKCBN network model includes: Based on the original CBAM module including the spatial attention module and the channel attention module, the neural network structure KAN is integrated into the CBAM module to obtain the KCBAM module; Combining the KCBAM module with the residual block to obtain the RKCB module; Combine the RKCB module with the original MobileNet to obtain the MRKCBN network model; The concatenated features are fused through the APSB module to obtain fused features, including: initializing a linear layer to obtain the initial attention score, expressed as: in, represents the i-th element in the concatenated feature sequence, W and b represent the weight and bias of the linear layer respectively, represents the initial attention score of the i-th element, represents the activation function; The initial attention score is normalized and expressed as: in, represents the activation function, represents the normalized attention score of the i-th element, I represents the total number of elements, and exp represents the exponential function; The key vector K is aggregated through the attention pooling operation to obtain K′, which is expressed as: in, represents the attention pooling operation, and X represents the concatenated feature sequence; Feature fusion is performed based on the value vector V, query vector Q and K′, which is expressed as: in, Represents fusion features.

2. The micro-expression recognition method based on multivariate optical flow feature fusion according to claim 1, characterized in that: The original video is obtained, and after preprocessing the original video, optical flow images are generated respectively by using a variety of optical flow algorithms, including: The original video is cut into multiple video sequences, each video sequence is enlarged by Euler video, and cut into frame images; For the amplified frame image, taking the initial frame of the video sequence as the starting frame image, calculating the optical flow difference between adjacent frame images, and taking the frame with the most dramatic optical flow change as the peak frame image; Based on the starting frame image and the peak frame image, optical flow images are generated using optical flow algorithms respectively.

3. The micro-expression recognition method based on multivariate optical flow feature fusion according to claim 2, characterized in that: The multiple optical flow algorithms include TVL1, DIS and PCA.

4. The micro-expression recognition method based on multivariate optical flow feature fusion according to claim 1, characterized in that: The KCBAM module includes a channel attention module KCAM and a spatial attention module KSAM; The KAN Linear module is used to replace the linear module in the original channel attention module to obtain a new channel attention module KCAM, which is expressed as: in, represents the input sequence of KCAM, represents the average pooling operation, represents the maximum pooling operation, represents the activation function, KAN represents the KAN network structure, represents the output sequence of KCAM; The convolution module in the original spatial attention module is replaced by the KAN_Conv2d convolution module to obtain a new spatial attention module KSAM, which is expressed as: in, represents the input sequence of KSAM, represents the convolution operation, Represents the output sequence of KSAM.

5. The micro-expression recognition method based on multivariate optical flow feature fusion according to claim 1, characterized in that: The horizontal image, the vertical image and the overall image are respectively input into the pre-built MRKCBN network model for feature extraction, the corresponding image output features are output, and the image output features are processed to obtain local features, including: The image output features corresponding to the horizontal image, the vertical image and the overall image are spliced, and the spliced ​​features are subjected to convolution and maximum pooling operations to obtain the local features.

6. A micro-expression recognition system based on multivariate optical flow feature fusion, characterized in that: include: The optical flow extraction module is used to obtain the original video, generate optical flow images respectively through a variety of optical flow algorithms after preprocessing the original video, and extract the horizontal optical flow vector and vertical optical flow vector in the optical flow image respectively, and superimpose the horizontal optical flow vector, vertical optical flow vector and optical flow image of each optical flow algorithm to obtain the horizontal image, vertical image and overall image; A local feature extraction module is used to input the horizontal image, the vertical image and the overall image into a pre-built MRKCBN network model for feature extraction, output corresponding image output features, and process the image output features to obtain local features; A global feature extraction module, used for inputting an optical flow image generated by one of the multiple optical flow algorithms into a pre-acquired VIT network model for feature extraction, outputting global features, and splicing the local features with the global features to obtain spliced ​​features; A feature fusion module, used for fusing the spliced ​​features through an APSB module to obtain fused features; A classification module is used to pass the fusion features through a layer normalization module and a multi-layer perceptron in sequence, and then classify them through a fully connected layer to obtain a micro-expression recognition result; The construction process of the MRKCBN network model includes: Based on the original CBAM module including the spatial attention module and the channel attention module, the neural network structure KAN is integrated into the CBAM module to obtain the KCBAM module; Combining the KCBAM module with the residual block to obtain the RKCB module; Combine the RKCB module with the original MobileNet to obtain the MRKCBN network model; The concatenated features are fused through the APSB module to obtain fused features, including: initializing a linear layer to obtain the initial attention score, expressed as: in, represents the i-th element in the concatenated feature sequence, W and b represent the weight and bias of the linear layer respectively, represents the initial attention score of the i-th element, represents the activation function; The initial attention score is normalized and expressed as: in, represents the activation function, represents the normalized attention score of the i-th element, I represents the total number of elements, and exp represents the exponential function; The key vector K is aggregated through the attention pooling operation to obtain K′, which is expressed as: in, represents the attention pooling operation, and X represents the concatenated feature sequence; Feature fusion is performed based on the value vector V, query vector Q and K′, which is expressed as: in, Represents fusion features.

7. A micro-expression recognition device based on multivariate optical flow feature fusion, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the micro-expression recognition method based on multivariate optical flow feature fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the micro-expression recognition method based on multivariate optical flow feature fusion as described in any one of claims 1 to 5 are implemented.

Images