CoTNet-based micro-expression recognition method driven by space-time features and application of CoTNet-based micro-expression recognition method

Abstract

The invention relates to a CoTNet-based micro-expression recognition method under space-time feature driving and application thereof. The method comprises the following steps: extracting an optical flow diagram of a micro-expression sample by using an optical flow method; extracting global features and local features by adopting CoTNet based on a context attention mechanism; based on CASMEII, SAMM and SMIC data sets, a micro-expression recognition model is trained and verified for the preprocessed data through a leaving-one cross validation method, and the micro-expression recognition model is used for recognizing medium-high-risk passengers in civil aviation security. According to the method, the CoTNet network and the optical flow method are combined, the micro-expressions are recognized by analyzing facial movement, key information in a micro-expression sequence is captured more comprehensively, the accuracy and robustness of micro-expression recognition are improved, meanwhile, micro-expression recognition is achieved based on the CoTNet micro-expression recognition method under space-time feature driving, and the method has the advantages of being high in recognition efficiency, high in accuracy and high in robustness. And the real emotional state of the passenger is analyzed, the risk of carrying dangerous goods and making dangerous behaviors of the passenger is judged, and related departments are assisted in identifying medium-and-high-risk passengers.

Description

Technical Field

[0001] The present invention relates to the technical field of micro-expression recognition, and in particular to a micro-expression recognition method based on CoTNet driven by space-time features and application thereof. Background Art

[0002] Micro-expressions are unconscious facial movements under emotional stimulation, which are difficult or even impossible to fake or control, and can reveal people's hidden emotions in specific scenarios. Therefore, as a branch of affective computing, micro-expression recognition has potential applications in different fields such as civil aviation security inspection, criminal interrogation and clinical medicine.

[0003] As an important transportation industry, civil aviation not only carries people's travel needs, but is also an important pillar of national and regional economic development. Safety is the lifeline of civil aviation, and strengthening security is crucial to ensuring transportation safety. With the development of artificial intelligence technology, integrating micro-expression-based emotion computing methods into civil aviation security can effectively improve the level of civil aviation safety. The "Smart Civil Aviation Construction Roadmap" officially promulgated on January 21, 2022 states: In 2025, we must achieve phased results in digital transformation, and one important aspect is to realize one face for travel and one platform for supervision.

[0004] With the development of image processing technology and computer technology, more and more experts and scholars have begun to invest in the research of micro-expression analysis, and the micro-expression recognition method has also evolved from the initial manual judgment to intelligent technical means. The development of computer vision and image processing technology has provided strong support for micro-expression recognition. Researchers use deep learning, neural networks and other technologies to efficiently and accurately identify micro-expressions, which also provides the possibility for a real-time civil aviation passenger safety assessment system. However, the current micro-expression recognition research does not pre-process the samples well, and ignores the feature fusion of the two dimensions of space and time. Due to the characteristics of short duration, low intensity, and sparse facial action units of micro-expressions, the difficulty of micro-expression recognition is greatly increased, so it needs further improvement. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a micro-expression recognition method based on CoTNet driven by spatiotemporal features and its use.

[0006] The present invention is achieved through the following technical solutions:

[0007] A micro-expression recognition method based on CoTNet driven by spatiotemporal features comprises the following steps:

[0008] S1. Extract the optical flow map of micro-expression samples using the optical flow method;

[0009] S2. Use CoTNet based on contextual attention mechanism to extract global and local features;

[0010] S3. Based on the CASMEII, SAMM and SMIC datasets, the micro-expression recognition model is trained and verified using the leave-one-out cross-validation method on the preprocessed data.

[0011] According to the above technical solution, preferably, in step S1, the TLV1 method is used to extract the optical flow map of the micro-expression sample.

[0012] According to the above technical solution, preferably, step S1 includes: extracting an optical flow map from a start frame to a vertex frame in a micro-expression sample; and fusing the optical flow map with the original image using a feature fusion method.

[0013] According to the above technical solution, preferably, step S2 includes:

[0014] A convolution layer and a pooling layer, used for extracting features from the optical flow map;

[0015] The CoTNet network includes four CoT modules, which are used to capture both global and local feature information;

[0016] The fully connected layer converts the global features and local features into one-dimensional vectors, and outputs them after linear combination.

[0017] According to the above technical solution, preferably, in step S3, the CASMEII, SAMM and SMIC data sets are preprocessed, including the following steps: reclassifying and fusing the samples in the CASMEII, SAMM and SMIC data sets, and marking the starting frame and vertex frame in the samples; performing face cropping on the fused samples; performing image enhancement processing on the cropped samples; and adjusting the brightness, contrast, saturation and hue of the sample images.

[0018] According to the above technical solution, preferably, in step S3, face cropping is performed using an MTCNN model.

[0019] According to the above technical solution, preferably, in step S3, the image enhancement processing includes deflecting the sample image at a small angle within 8 degrees.

[0020] The present application also discloses a method for micro-expression recognition based on CoTNet driven by space-time features. Based on the above-mentioned method for micro-expression recognition based on CoTNet driven by space-time features, the micro-expression recognition model is used to identify medium- and high-risk passengers in civil aviation security, and specifically includes the following steps:

[0021] S21. Deploy high frame rate cameras at airports to capture facial videos of passengers;

[0022] S22. Under the effect of the stimulation method, stimulate the passenger to produce micro-expressions and collect them;

[0023] S23. The collected micro-expression information is identified by the micro-expression recognition model, and then combined with the passenger's background information, checked baggage information, carry-on baggage information and personal security inspection information to identify medium and high-risk passengers.

[0024] The beneficial effects of the present invention are:

[0025] The CoTNet network in the present invention combines the advantages of Transformer and CNN, has strong global and local feature extraction capabilities, can perform multi-scale feature learning on micro-expressions, and better capture the subtle changes of micro-expressions; the optical flow method can effectively capture the motion information between consecutive frames, which is helpful to extract the time series features of micro-expressions and improve the accuracy and stability of micro-expression recognition;

[0026] At the same time, the CoTNet network is combined with the optical flow method to identify micro-expressions by analyzing facial movements, which more comprehensively captures the key information in the micro-expression sequence and improves the accuracy and robustness of micro-expression recognition. Compared with other networks, it has superior computing power and accurate recognition results, and has high application promotion value in practical applications.

[0027] In addition, the CoTNet-based micro-expression recognition method driven by the space-time features proposed in the present invention is used to realize micro-expression recognition, analyze the passenger's real emotional state, and then combine the passenger's background information, checked baggage information, carry-on baggage information and personal security inspection information to judge the risk of passengers carrying dangerous goods and engaging in dangerous behaviors, and assist relevant departments in identifying medium and high-risk passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall network structure design of the present invention.

[0029] Figure 2 It is a network diagram of the CoT module in the present invention.

[0030] Figure 3 It is a schematic diagram of the CoTNet network structure in the present invention.

[0031] Figure 4 It is the micro-expression recognition dataset in the present invention.

[0032] Figure 5 It is a schematic diagram of the distribution of the number of samples of the fusion data in the present invention.

[0033] Figure 6It is a loss value image showing the change in loss value when the model in the present invention is trained on the fusion data set.

[0034] Figure 7 It is a schematic diagram of face AU motion unit encoding in the present invention.

[0035] Figure 8 It is the specific data of the optical flow ablation experiment in the present invention.

[0036] Fig. 9 It is the specific data of the CoTNet ablation experiment in this invention.

[0037] Fig.10 is the confusion matrix of the fused data set in the present invention.

[0038] Fig.11 It is a schematic diagram of the application system framework of the micro-expression-based emotion computing in civil aviation security in the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and the best embodiment. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the invention.

[0041] Example 1: Figure 1 As shown, the present invention comprises the following steps:

[0042] S1. Extract the optical flow map of the micro-expression sample using the optical flow method. Specifically, first extract the optical flow map from the start frame to the vertex frame in the micro-expression sample. In this example, the TLV1 method is preferably used to extract the optical flow map of the micro-expression sample; and the optical flow map is fused with the original image using the feature fusion method.

[0043] In motion analysis and recognition, target tracking, and autonomous driving, the optical flow method is used to extract motion information, achieving more accurate motion recognition, target tracking, and environmental perception. At the same time, the optical flow method is sensitive to tiny movements. Micro-expressions are short-lived and low-intensity, and are relatively small facial movements. Therefore, the optical flow method is used in this example to extract dynamic features of facial movements. At the same time, the optical flow method is robust to common factors such as lighting changes, making the network more capable of dealing with various interference factors in actual scenes.

[0044] Optical flow is based on three assumptions: (1) the illumination remains unchanged between two frames; (2) the motion of the same pixel between two frames is small; and (3) the motion of adjacent pixels is similar. Micro-expressions occur for a short time and the motion is small, so the three assumptions of optical flow are that the illumination remains unchanged, the motion of the same pixel is small, and the motion of adjacent pixels is similar. Micro-expressions occur for a short time and the motion is small, so the three assumptions of optical flow are that the illumination remains unchanged, the motion of the same pixel is small, and the motion of adjacent pixels is similar. Let I(x,y,t) be the brightness value at position (x,y) at time t, and the distance moved by the pixel in time dt be (dx,dt). Since the brightness value between the two frames does not change, we can get the following formula:

[0045] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0046] Perform Taylor series expansion on the above formula, remove the general terms and divide by dt to obtain the following formula:

[0047]

[0048] If p and q are the horizontal and vertical directions of the pixel, the velocity components are:

[0049]

[0050] Substituting p and q into the above formula can obtain the optical flow change of each pixel in the image.

[0051] S2. Use CoTNet based on contextual attention mechanism to extract global features and local features. Specifically, it includes: convolution layer and pooling layer, which are used to extract features from the optical flow map; CoTNet network, including 4 CoT modules, which are used to capture the information of global features and local features at the same time; fully connected layer, which converts the global features and local features into one-dimensional vectors, and outputs them after linear combination.

[0052] To solve the problem of sparse facial action units when micro-expressions occur, CoTNet based on contextual attention mechanism is used to extract effective spatial and temporal features, and feature extraction is achieved through different convolutions to enhance the ability to extract facial action-related features. The original self-attention mechanism directly uses the image to obtain the attention matrix of each spatial position, without considering the relationship between the images themselves, and ignoring the rich information of the local part of the image. In this example, the CoT module is used to extract local features of the image, combining the advantages of the self-attention mechanism and CNN to improve the feature extraction ability of the network. The CoTNet network consists of four CoT modules, which can capture global and local information at the same time. Global information considers all regions as a whole, while local information focuses on the neighborhood around each region. By combining global and local information, the CoTNet network can better understand the semantics and details in the image. The fully connected layer converts the feature map into a one-dimensional vector and linearly combines it to produce a new output. The fully connected layer is usually located at the end of the network. After feature extraction by the convolutional layer and the pooling layer, the extracted features are integrated to provide input for tasks such as classification or regression.

[0053] The self-attention mechanism has achieved good results in many computer vision tasks. The self-attention mechanism hopes that the machine will notice the correlation between different positions in the input sample. At the beginning, three matrices, Query, Key, and Value, are required. For a given input sequence, three new representations of Query, Key, and Value are obtained through linear transformation. However, the traditional self-attention mechanism obtains the self-attention matrix through independent Query and Key at each spatial position, without considering the local features of the image.

[0054] like Figure 2 As shown in the figure, the CoT module makes full use of local information to obtain the attention matrix and enhance the model's feature extraction capability. First, the model obtains the Key value (also called static context representation) through 3×3 convolution. The larger the Key value, the larger the receptive field, and the wider the context information range associated with the dynamic attention matrix obtained later; then, the input feature value is directly used as the Query value, concatenated with the Key value on the channel, and the dynamic self-attention matrix is ​​learned through two consecutive 1×1 convolutions; then, the Value value is obtained through 1×1 convolution, and the learned dynamic multi-head attention matrix is ​​multiplied by the Value value to obtain the dynamic context representation of the input. Finally, the static context representation is short-circuited to the module output part, fused with the dynamic context representation, and the final result is used as the module output.

[0055] like Figure 3As shown in the figure, the CoTNet network combines the network structure of ResNet and improves the self-attention mechanism to form a CoT module to replace the 3×3 module in the ResNet network. This network combines the advantages of CNN and Transformer, fully considers local and global information and performs self-attention calculations, which improves the network's ability to recognize micro-expressions.

[0056] S3. Based on the CASMEII, SAMM and SMIC datasets, the micro-expression recognition model is trained and verified using the leave-one-out cross-validation method on the preprocessed data.

[0057] The micro-expression dataset is a basic resource for micro-expression recognition research, which contains micro-expression samples from different cultures, ages, genders, etc. At present, there are 8 commonly used micro-expression datasets (see Figure 4 ). These datasets provide valuable resources for researchers and contribute to the development and application of micro-expression recognition technology. In the field of micro-expression recognition, researchers mostly use the spontaneously generated SMIC, CASMEII, and SAMM datasets. The SMIC dataset was released by Pfister et al., which includes three sub-datasets: HS, VIS, and NIR, which were taken by a high frame rate camera, a normal camera, and a near-infrared camera, respectively. The CASMEII dataset was released by Yan et al. in 2014. The face size in this dataset is larger, and the number of samples in each category is relatively even. SAMM is a spontaneous micro-expression dataset proposed by the Manchester Metropolitan University team in 2016, and it is encoded using the facial action coding system. In the field of micro-expression recognition, SMIC, CASMEII, and SAMM datasets are mostly used for experiments.

[0058] The CASMEII, SAMM and SMIC data sets are preprocessed, including the following steps:

[0059] First, the samples in the CASMEII, SAMM and SMIC datasets are reclassified and fused. The SMIC dataset is divided into three categories: positive, negative and surprise; the CASMEII dataset is divided into five categories: happiness, depression, surprise, disgust and others; the SAMM dataset is divided into eight categories: happiness, fear, surprise, anger, disgust, sadness, contempt and others. Since the three datasets classify micro-expressions differently and the samples are unbalanced, the dataset samples need to be divided and re-fused. Happiness is marked as positive, depression, disgust, fear, anger, sadness and anger are all classified as negative, and surprise is still classified as surprise. The classification of the dataset after fusion is not balanced, and the number of negative samples is the largest. The number of samples in specific categories is as follows: Figure 5In the case of unbalanced samples, the recognition accuracy of the samples with a small number will be low, and the information of the few sample categories cannot be fully utilized. Therefore, the fused data is resampled to balance the number of samples of each category, and the sample with the least number of samples is used as the basis for expansion. The sample is expanded according to the number of three categories to achieve the purpose of balancing the number of samples of different categories in the data set.

[0060] Among the three data sets, the SMIC-HS data set only annotates the start frame and offset frame, but does not annotate the vertex frame. However, the vertex frame is an indispensable annotation in micro-expression recognition, so the sample vertex frame must be obtained by calculation. The number of frames of each sample is different, and the vertex frame is not a fixed frame or occurs in the middle of the entire sample, so the sample vertex frame is determined by calculating the change between each frame and the start frame and offset frame. First, obtain the start frame and offset frame in the sample of the data set, and calculate the pixel difference between each frame in the sample and the start frame and offset frame. The frame with the largest difference is the frame with the largest facial expression change, which is the vertex frame of this sample.

[0061] Secondly, the images in the dataset used to train and test the neural network should contain as little background information as possible that is irrelevant to micro-expression movements, but the micro-expression images provided by the dataset contain more irrelevant background information, which will affect the neural network's extraction of feature information. The MTCNN (Multi-task Cascaded Convolutional Networks) model is used for face cropping.

[0062] At the same time, the cropped samples are enhanced and the sample images are deflected at a small angle within 8 degrees. In the real world, the angle and direction of facial expressions may change slightly. By deflecting the micro-expression dataset at a small angle within 8 degrees, the changes in different observation angles or head postures in real situations are simulated, which enriches the representation ability of micro-expression data, increases the diversity of data samples, and makes the trained model more robust and better able to cope with micro-expressions at different facial angles and directions.

[0063] In addition, adjusting the brightness, contrast, saturation, and hue of sample images can increase the diversity of the data, allowing the model to better learn the characteristics of micro-expressions, enabling it to perform better when faced with different styles, lighting conditions, or shooting devices, thereby improving the performance and generalization ability of the micro-expression recognition model.

[0064] After that, the leave-one-out cross-validation method is used for model training, leaving only one sample as the test set each time, and the other samples are the training set. This method is suitable for small sample data sets, making the most of all samples for model training and improving the generalization ability of the model. Each sample has the opportunity to be used as a test sample, which can more accurately evaluate the performance of the model on the entire data set.

[0065] To make the results more referenceable, unweighted F1 (Unweighted F1-score, UF1) and unweighted average recall (Unweighted Average Rrcall, UAR) are used as evaluation indicators. UF1 is the harmonic mean of precision and recall. Precision is the proportion of all samples identified as positive that are truly positive, and recall is the ratio of correctly predicted positive samples to actual positive samples. The specific formula is as follows:

[0066]

[0067] TP is the number of positive samples correctly identified, FP is the number of negative samples identified as positive samples, and FN is the number of positive samples identified as negative samples. UAR is the average recall rate on each category, that is, the average recall rate of each micro-expression category. The specific formula is as follows:

[0068]

[0069] Where N is the number of samples in category k.

[0070] UF1 and UAR consider the performance of each category without being affected by category imbalance. In image recognition tasks, the number of samples in different categories may vary greatly. Both UF1 and UAR are indicators calculated based on macro-average, which are not affected by the number of samples and have good robustness in dealing with category imbalance problems.

[0071] The model is verified and characterized through loss values, feature area visualization, ablation experiments, confusion matrix, etc.

[0072] The loss value measures the performance of the model by measuring the difference between the predicted value and the true value. The smaller the loss value, the better the network performance. The cross entropy loss function is more sensitive to classification problems. Since the cross entropy loss function has a higher degree of penalty for classification errors, the model will pay more attention to misclassified samples during the optimization process, thereby improving classification performance. The cross entropy loss function is as follows:

[0073]

[0074] Figure 6 The loss value changes when the model is trained on the fusion dataset. As the number of iterations increases, the loss value of the model continues to decrease and converge, which means that the model can reach a convergence state.

[0075] Feature Map Visualization is a technique for visualizing feature maps in neural networks. Through feature map visualization, we can intuitively observe and understand the areas that the neural network focuses on.

[0076] AU refers to different action units in the face, each of which represents a specific muscle or muscle group on the face. By observing the intensity of changes in different areas, a person's emotional state can be analyzed. Figure 7 It is the face AU motion unit encoding.

[0077] According to the coding rules, when AU6, AU12 and AU28 appear, they are judged as positive, that is, the actions of lifting cheeks, raising mouth corners and sucking lips to cover teeth are positive categories. When identifying positive samples, the network should pay more attention to cheeks, mouth corners and lips.

[0078] Ablation experiment is a commonly used experimental design method in scientific research. Its main function is to help researchers understand the functions and effects of each component in a complex system and their contribution to the overall performance of the system. The analysis is performed by gradually reducing certain components or factors in the system and then observing the changes in the overall performance of the system.

[0079] Optical flow is a very important part of the whole model. The optical flow method can extract the temporal motion information of objects from continuous image sequences, thereby helping to capture the dynamic changes in micro-expressions and providing important dynamic features for micro-expression recognition. By using the motion information extracted by the optical flow method, the interference of illumination on micro-expression recognition results can be reduced, improving the accuracy and robustness of recognition.

[0080] To verify the importance of the optical flow method in the model, the following ablation experiment was performed. Flow-CoTNet with optical flow features was compared with CoTNet without optical flow features. In the fusion dataset, Flow-CoTNet is 0.1255 higher than CoTNet in UF1 and 0.1258 higher in UAR. The Flow-CoTNet algorithm results are superior on both the SMIC and CASME II datasets. Fig. 9 It is the specific data of the optical flow ablation experiment.

[0081] Similarly, CoTNet also has an advantage in micro-expression recognition. To verify the advantages of the CoTNet algorithm, CoTNet is compared with the ResNet algorithm in the case of integrating optical flow. Micro-expressions have complex nonlinear characteristics, and ResNet, as a deep neural network, can effectively learn and express these nonlinear characteristics. Through the stacking and training of multiple layers of residual blocks, ResNet can gradually extract and combine nonlinear features in micro-expressions, thereby achieving accurate recognition of micro-expressions. And it has been proven to achieve good performance in many computer vision tasks, but the ablation experiment shows that CoTNet is better than ResNet in micro-expression recognition. On the fusion dataset, Flow-CoTNet is 0.0956 higher than Flow-ResNet in UF1 and 0.1134 higher in UAR. Fig.10 It is the specific data of the ablation experiment.

[0082] The confusion matrix is ​​also an important method to measure the recognition results of the algorithm. The confusion matrix shows the prediction results for each category, including true positives, false negatives, false positives, and true negatives. The diagonal line of the picture shows the prediction probability of different categories, which can more comprehensively evaluate the performance of different categories. Fig.11 is the confusion matrix of the fused dataset.

[0083] By comparing the ablation matrices of the Flow-CoTNet and CoTNet algorithms, we can conclude that the accuracy of Flow-CoTNet in identifying negative samples is 0.07 higher than that of CoTNet, the accuracy of positive samples is 0.40 higher, and the accuracy of surprise samples is 0.30 higher. This shows that after adding the optical flow algorithm, the overall micro-expression recognition rate and the recognition rate of different categories have been improved.

[0084] In the ablation matrix comparison between Flow-CoTNet and Flow-ResNet algorithms, it can be concluded that the accuracy of Flow-CoTNet's negative samples is 0.03 higher than that of Flow-ResNet, the accuracy of positive samples is 0.30 higher, and the accuracy of surprising samples is 0.08 higher. This shows that in the presence of optical flow, CoTNet is better than ResNet.

[0085] Overall, the confusion matrix shows that both the optical flow method and CoTNet have made great contributions to improving the accuracy of micro-expression recognition, and combining the two can achieve better results. It is the integration of spatial features, as well as local and global features, that improves the overall accuracy of micro-expression recognition.

[0086] The CoTNet network in the present invention combines the advantages of Transformer and CNN, has strong global and local feature extraction capabilities, can perform multi-scale feature learning on micro-expressions, and better capture the subtle changes of micro-expressions; the optical flow method can effectively capture the motion information between consecutive frames, which is helpful to extract the time series features of micro-expressions and improve the accuracy and stability of micro-expression recognition; at the same time, the CoTNet network is combined with the optical flow method to recognize micro-expressions by analyzing facial movements, which more comprehensively captures the key information in the micro-expression sequence, improves the accuracy and robustness of micro-expression recognition, and has superiority over other networks. At the same time, the efficient computing power and accurate recognition results also have high application promotion value in real-time applications.

[0087] Example 2: Fig.11 As shown, the present application also discloses a use of a micro-expression recognition method based on CoTNet driven by space-time features. Based on the above-mentioned micro-expression recognition method based on CoTNet driven by space-time features, the micro-expression recognition model is used to identify medium- and high-risk passengers in civil aviation security, and specifically includes the following steps:

[0088] S21. Deploy high-frame rate cameras at airports (such as security checkpoints, boarding, and customs clearance) to capture facial videos of passengers.

[0089] S22. Under the influence of stimulation methods (such as language stimulation, video stimulation, warning sign stimulation, etc.), stimulate passengers to produce micro-expressions and collect them, and determine the location where the micro-expressions occur (starting frame, peak frame and end frame).

[0090] S23. The micro-expression recognition model is used to identify the collected micro-expression information, analyze the passenger's true emotional state, and then combine the passenger's background information, checked baggage information, carry-on baggage information and personal security inspection information to identify medium and high-risk passengers.

[0091] The application system of micro-expression-based emotional computing in civil aviation security only requires the deployment of cameras, without any contact equipment or other special equipment. Under the action of external stimulation methods, it can quickly capture the emotional fluctuations of risky passengers and analyze the emotional state of passengers. This system integrates passenger emotional information with passenger background information, checked baggage information, carry-on baggage information and personal security inspection information to achieve a comprehensive analysis of passenger risks, and can provide early warning for risky passengers, thereby improving civil aviation security capabilities.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A micro-expression recognition method based on CoTNet driven by spatiotemporal features, characterized in that: The steps include: S1. Extract the optical flow map of micro-expression samples using the optical flow method; S2. Use CoTNet based on contextual attention mechanism to extract global and local features; S3. Based on the CASMEII, SAMM and SMIC datasets, the micro-expression recognition model is trained and verified using the leave-one-out cross-validation method on the preprocessed data.

2. According to the micro-expression recognition method based on CoTNet driven by space-time features as described in claim 1, it is characterized in that: In step S1, the optical flow map of the micro-expression sample is extracted using the TLV1 method.

3. A micro-expression recognition method based on CoTNet driven by space-time features according to claim 1 or 2, characterized in that: Step S1 includes: extracting an optical flow map from a start frame to a vertex frame in a micro-expression sample; and fusing the optical flow map with an original image using a feature fusion method.

4. According to the micro-expression recognition method based on CoTNet driven by space-time features as described in claim 3, it is characterized in that: Step S2 includes: A convolution layer and a pooling layer, used for extracting features from the optical flow map; The CoTNet network includes four CoT modules, which are used to capture both global and local feature information; The fully connected layer converts the global features and local features into one-dimensional vectors, and outputs them after linear combination.

5. According to the micro-expression recognition method based on CoTNet driven by space-time features as described in claim 4, it is characterized in that: In step S3, the CASMEII, SAMM and SMIC data sets are preprocessed, including the following steps: reclassifying and fusing the samples in the CASMEII, SAMM and SMIC data sets, and marking the starting frame and vertex frame in the samples; Perform face cropping on the fused samples; perform image enhancement processing on the cropped samples; Adjust the brightness, contrast, saturation, and hue of the sample image.

6. According to the micro-expression recognition method based on CoTNet driven by space-time features as described in claim 5, it is characterized in that: In step S3, face cropping is performed using the MTCNN model.

7. According to the micro-expression recognition method based on CoTNet driven by space-time features as described in claim 5, it is characterized in that: In step S3, the image enhancement process includes deflecting the sample image at a small angle within 8 degrees.

8. A method for micro-expression recognition based on CoTNet driven by space-time features, based on the method for micro-expression recognition based on CoTNet driven by space-time features according to any one of claims 1 to 7, characterized in that: The micro-expression recognition model is used to identify medium- and high-risk passengers in civil aviation security.

9. The use of a micro-expression recognition method based on CoTNet driven by space-time features according to claim 8, characterized in that: The steps include: S21. Deploy high frame rate cameras at airports to capture facial videos of passengers; S22. Under the effect of the stimulation method, stimulate the passenger to produce micro-expressions and collect them; S23. Recognize the collected micro-expression information through the micro-expression recognition model.

10. The use of a micro-expression recognition method based on CoTNet driven by space-time features according to claim 9, characterized in that: In step S23, the background information, checked baggage information, carry-on baggage information and personal security information of the passengers are combined to identify medium and high risk passengers.

Images