Micro-expression recognition method and system based on graph convolution and Transform cascade model

Abstract

The invention relates to a micro-expression recognition method and system based on graph convolution and a Transform cascade model, and belongs to the technical field of computer vision and image processing. A deep modeling framework oriented to micro-expression spatio-temporal dynamics is constructed by combining facial structure prior, deformation amplification features and optical flow motion features; the problem of difficulty in detection caused by weak micro-expression amplitude, short duration time and insignificant local change is solved; designing a cascade model formed by alternately stacking a drawing convolutional network and a Transform, and realizing joint characterization of a facial local structure relationship and global dynamic dependence; a learnable graph structure and a multi-source feature adaptive fusion mechanism are introduced, and the feature discrimination capability is improved while the prior reasonability of the structure is ensured, so that the accuracy and robustness of micro-expression recognition are improved, and technical support is provided for automatic micro-expression recognition in application scenes such as psychological analysis, security monitoring and human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a micro-expression recognition method and system based on graph convolution and Transformer cascade model, and particularly to the design and implementation of a deep learning model for extracting subtle facial action features from facial image sequences and realizing automatic micro-expression recognition. Background Technology

[0002] Microexpressions are involuntary, subtle changes in human facial expression that typically last between 0.04 and 0.5 seconds, occurring when emotions are suppressed or rapidly changing. Due to their short duration, small amplitude, and difficulty in detection, microexpressions are widely used in fields such as psychological analysis, security monitoring, medical diagnosis, and human-computer interaction. Accurate identification of microexpressions has significant research and application value.

[0003] However, automatic micro-expression recognition still faces many challenges. First, micro-expressions involve extremely small movements, typically manifesting only as slight changes in local muscles, making it difficult for traditional methods based on texture or optical flow features to fully capture these subtle differences. Second, micro-expression sample data is scarce; publicly available datasets are limited in size and imbalanced in categories, severely restricting the training performance of deep learning models. Third, complex structural relationships and correlations exist between different facial regions; relying solely on local convolution or global modeling easily overlooks the topological connections between key motion regions, leading to insufficient recognition accuracy. In recent years, micro-expression recognition methods based on optical flow estimation, spatiotemporal texture description, and convolutional neural networks (CNNs) have been extensively studied, but due to limitations in their modeling capabilities, they still fall short in handling weak movements, long-distance dependencies, and fine-grained feature fusion.

[0004] With the development of Graph Neural Networks (GNNs) and Visual Transformers, dividing facial regions into nodes and modeling their spatial topological relationships using graph structures has become a new research direction. Graph Convolutional Networks (GCNs) can characterize the connection structure between local regions, which is beneficial for modeling the associations of subtle muscle movements. Visual Transformers, relying on self-attention mechanisms, can capture global dynamic relationships across regions, giving them an advantage in modeling the spatiotemporal features of subtle facial expressions. However, existing methods often rely on a single model structure, failing to adequately model the joint structure and global dynamics; some methods use static graph structures or fixed adjacency relationships, lacking adaptive representation of the associations of real facial movements; furthermore, the deformation features of facial expressions and optical flow motion features are often modeled separately, and an effective multi-source dynamic fusion mechanism has not yet been formed, resulting in limited recognition performance.

[0005] Therefore, there is an urgent need for a new model architecture that combines the advantages of graph convolution and visual Transformer to achieve joint modeling of local facial structural relationships, global dynamic dependencies, and subtle motion features, thereby improving the accuracy and robustness of micro-expression recognition. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a micro-expression recognition method and system based on a cascaded model of graph convolution and Transformer. By combining facial structure priors, deformation amplification features, and optical flow motion features, a deep modeling framework for the spatiotemporal dynamics of micro-expressions is constructed, solving the problem of difficulty in detection caused by the weak amplitude, short duration, and insignificant local changes of micro-expressions. A cascaded model of alternating stacked graph convolutional networks and Transformers is designed to achieve joint representation of local facial structural relationships and global dynamic dependencies. A learnable graph structure and a multi-source feature adaptive fusion mechanism are introduced to improve feature discrimination ability while ensuring the rationality of structural priors, thereby improving the accuracy and robustness of micro-expression recognition and providing technical support for automatic micro-expression recognition in application scenarios such as psychological analysis, security monitoring, and human-computer interaction.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a micro-expression recognition method based on a graph convolution and Transformer cascade model, including... The starting frame and peak frame of the micro-expression sequence are obtained. Face detection, key point localization, alignment and cropping are performed on the starting frame and peak frame to obtain a standardized face image. The starting frame and peak frame pair are selected based on preset rules. A pre-trained deformation amplification network is used to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions, which are then used to highlight subtle muscle movements. The TV-L1 optical flow algorithm is used to calculate the optical flow image based on the start frame and the peak frame to obtain pixel-level motion information. The pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation. Based on facial key points, standardized face images are divided into regions, and a static adjacency matrix representing facial topological relationships is constructed. By utilizing the amplified deformation features to calculate the similarity between regions, and by weighting the static adjacency matrix, an initial graph structure with both structural priors and data-driven characteristics is obtained. The optical flow image is divided into several image blocks and mapped into a sequence of feature vectors. These blocks are then input into a visual Transformer for global dynamic modeling to obtain global spatiotemporal features across regions. The amplified deformation features are weighted and fused with the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The multimodal fusion features and the initial graph structure are input into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers. The graph convolutional network aggregates local neighborhood structure information, the Transformer captures long-distance dependencies, and the graph structure is updated in a learnable manner during training to obtain micro-expression discrimination features. The micro-expression discrimination features are classified, and the corresponding micro-expression category results are output.

[0008] As a further technical solution, a pre-trained deformation amplification network is used to perform deformation enhancement processing on the starting frame and the peak frame; The deformation amplification network models the residual structural changes of the input starting frame and peak frame pairs, and outputs deformation feature vectors for multiple local regions. Each deformation feature vector corresponds to the degree of deformation of a local facial region.

[0009] In a further preferred embodiment, firstly, the deformation amplification network performs feature encoding on the starting frame and the peak frame, and extracts intermediate feature representations that characterize the geometric structure of the face respectively; then, by performing difference or residual calculation on the starting frame and the peak frame, residual information reflecting the structural changes of local facial regions is obtained, and the residual information is amplified. After amplification, the deformation amplification network outputs a feature map containing amplified deformation information. As a further technical solution, standardized facial images are region-divided based on facial key points; including: Based on a pre-defined set of facial landmarks, the face is divided into several sub-regions, each of which serves as a node in a graph structure. A static adjacency matrix is ​​constructed using the spatial proximity and anatomical correlation between nodes to represent the topological connections between the facial regions.

[0010] As a further technical solution, the similarity between regions is calculated using amplified deformation features, and the static adjacency matrix is ​​weighted to obtain an initial graph structure that combines structural priors and data-driven characteristics; including: For each pair of regions with static connectivity, which refers to facial region nodes defined as having direct connectivity in the static adjacency matrix, the corresponding magnified deformation feature vectors are extracted, and the cosine similarity between the two is calculated. The obtained cosine similarity is used as a weight coefficient and multiplied element-wise with the static adjacency matrix to obtain the initial weighted adjacency matrix, i.e., the initial graph structure.

[0011] Further preferred, for regions with static connectivity, the corresponding amplified deformation feature vectors are extracted, and cosine similarity is calculated, as shown below: ; in, Indicates the first i The facial region and the first j Similarity of deformation features between facial regions; This represents the magnified deformation feature vector corresponding to the i-th facial region; This represents the magnified deformation feature vector corresponding to the j-th facial region.

[0012] As a further technical solution, the optical flow image is divided into several image patches and mapped into a sequence of feature vectors, which are then input into a visual Transformer for global dynamic modeling to obtain cross-regional global spatiotemporal features; including: The optical flow image is divided into several image patches of a preset size. Each image patch is flattened and linearly mapped to obtain a sequence of feature vectors. These sequences are then superimposed with learnable positional codes and input into a visual Transformer. A self-attention mechanism is used to establish dependencies between different image patches to capture cross-regional motion correlations and overall dynamic patterns. Specifically, this includes: The above feature vector sequence is input into the visual Transformer. The correlation weights between different image patches are calculated through a multi-head self-attention mechanism. After iterative processing by a multi-layer self-attention and feedforward network, the feature vector output by the visual Transformer not only contains local optical flow information, but also integrates dynamic correlation information from different facial regions, ultimately forming a global spatiotemporal feature representation that can reflect the overall facial motion pattern and the coordinated changes between regions.

[0013] As a further technical solution, the amplified deformation features are weighted and fused with the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; including: The amplified deformation features and global spatiotemporal features are mapped to the same dimensional space, and then linearly weighted and fused using a learnable fusion coefficient α, as shown below: ; Where α is the weight automatically learned through gradient descent during training; This indicates that by fusing weights α, the pair and The feature representation obtained after linear weighted fusion is the multimodal fusion feature representation; This represents the deformation features extracted by the deformation amplification network and then aggregated by region, used to characterize the geometric deformation information of each local region of the face between the starting frame and the peak frame; This represents the global spatiotemporal features obtained by performing global dynamic modeling of optical flow images using a visual Transformer. It is used to characterize the cross-regional motion correlations and overall dynamic patterns between different facial regions.

[0014] As a further technical solution, the cascaded model includes multiple cascaded graph Transformer modules, each graph Transformer module including a graph convolution submodule, i.e., a graph convolutional network, a self-attention submodule, and residual connection and normalization units that cooperate with it. The graph convolution submodule is used to: aggregate local neighborhood information of node features under the constraints of the current graph structure. The input is the node feature matrix and the corresponding adjacency matrix. By weighted aggregation of the features of neighboring nodes that are connected to the current node, a node feature representation that integrates neighborhood structure information is obtained. The self-attention submodule is used to: perform global dependency modeling on the node features output by graph convolution. It calculates the correlation weights between different nodes through the self-attention mechanism, so that each node comprehensively considers the feature information of other nodes during the feature update process, thereby establishing long-distance dependencies across regions. Residual connections and normalization operations are introduced between adjacent graph Transformer modules. This method aggregates local neighborhood structure information through graph convolutional networks, captures long-range dependencies through Transformers, and performs learnable updates to the graph structure during training to obtain micro-expression discrimination features; including: The multimodal fusion features and the initial graph structure are input together into the first graph Transformer module. First, the graph convolution submodule performs local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint, so that the node features are fused with the structural and deformation information of the adjacent regions, thereby obtaining an intermediate feature representation containing local topological relationships. Subsequently, the self-attention submodule performs global modeling of the intermediate feature representation. By calculating the correlation weight between any two region nodes, the node features perceive the dynamic relationship between distant regions. At the same time, during the training process, the initial graph structure is adaptively updated by introducing a learnable adjacency matrix, so that the graph structure dynamically adjusts the connection weights according to the facial motion patterns of different samples. After being processed by multiple cascaded graph Transformer modules, the final node features or their convergence results constitute the micro-expression discrimination features, which include both local facial structural relationships and cross-regional dynamic dependency information.

[0015] A further preferred embodiment involves the graph convolution submodule performing local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint. This allows the node features to be fused with the structural and deformation information of adjacent regions, thereby obtaining an intermediate feature representation that includes local topological relationships. The specific implementation process is as follows: Initial input: The input to the Graph Convolutional Network (GCN) is the node features of each facial region and the corresponding adjacency matrix. The adjacency matrix defines the connection relationship between nodes in the graph. Aggregation process: The graph convolution submodule aggregates the features of each node using an adjacency matrix; in each graph convolution layer, the features of a node are weighted and averaged with the features of its neighboring nodes to obtain a new feature representation; the specific formula is as follows: ; in, It is the first The node feature matrix of the layer; It is a standardized adjacency matrix, which includes structural information between nodes; It is the first The learnable weight matrix of the layer; It is an activation function; It is the first +1 layer node feature matrix; Aggregated features: The new feature representations are generated by aggregating graph convolutions, which integrate the local topological relationships and deformation information of nodes to produce intermediate feature representations.

[0016] As a further technical solution, the initial graph structure is updated during training using a learnable adjacency matrix; a learnable weight matrix with the same number of nodes is introduced on the basis of the initial weighted adjacency matrix to adjust the connection strength of each edge, and the updated adjacency matrix is ​​normalized using the Sigmoid function.

[0017] As a further technical solution, the classification module includes a fully connected layer and an activation function layer. By performing nonlinear mapping on the micro-expression discrimination features, it outputs the probability distribution of each target micro-expression category and determines the final micro-expression recognition result based on preset decision rules. During the training of the cascaded model, the cross-entropy loss function is used to optimize the network parameters in a supervised manner.

[0018] A second aspect of the present invention provides a micro-expression recognition system based on a graph convolution and Transformer cascade model, comprising: The data acquisition and preprocessing module is configured to: acquire the start frame and peak frame of the micro-expression sequence, perform face detection, key point localization, alignment and cropping on the start frame and peak frame to obtain a standardized face image, and select the start frame and peak frame pair based on preset rules; The deformation feature extraction module is configured to: use a pre-trained deformation amplification network to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions in order to highlight subtle muscle movements; The optical flow feature extraction module is configured to: calculate the optical flow image based on the start frame and the peak frame using the TV-L1 optical flow algorithm to obtain pixel-level motion information; the pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation; The graph structure construction module is configured to: divide a standardized face image into regions based on facial key points, construct a static adjacency matrix representing the topological relationships of the face; calculate the similarity between regions using magnified deformation features, weight the static adjacency matrix, and obtain an initial graph structure that combines structural priors and data-driven characteristics. The global dynamic modeling module is configured to divide the optical flow image into several image blocks and map them into a sequence of feature vectors, input them into the visual Transformer for global dynamic modeling, and obtain global spatiotemporal features across regions. The feature fusion module is configured to: perform weighted fusion of the amplified deformation features and the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The Graph-Transformer Cascaded Modeling Module is configured to: input multimodal fusion features and initial graph structure into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers; aggregate local neighborhood structure information through graph convolutional networks; capture long-distance dependencies through Transformers; and perform learnable updates to the graph structure during training to obtain micro-expression discrimination features. The classification module is configured to output micro-expression category recognition results based on micro-expression discrimination features.

[0019] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps in the micro-expression recognition method based on graph convolution and Transformer cascade model described in the first aspect of the present invention.

[0020] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the micro-expression recognition method based on graph convolution and Transformer cascade model described in the first aspect of the present invention.

[0021] The beneficial effects of this invention are as follows: This invention employs a dual-branch feature extraction method combining deformation amplification networks and optical flow estimation. This allows for the acquisition of sufficiently discriminative geometric deformation and pixel-level motion information even with minimal micro-expression amplitude. By combining a structural prior map constructed from facial key points with a similarity weighting mechanism based on deformation features, the graph structure conforms to facial anatomical topology while highlighting connections in regions more relevant to micro-expressions. Alternating graph convolution and Transformer operations on the learnable graph structure comprehensively models local facial structural relationships and global dynamic dependencies, enhancing the completeness and discriminative power of micro-expression feature representation. Furthermore, the adaptive weighted fusion of multimodal features and an end-to-end training framework improve the model's robustness and generalization ability to micro-expressions from different individuals, expression categories, and under varying acquisition conditions, resulting in high recognition accuracy and stability in practical applications. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart illustrating a micro-expression recognition method based on a graph convolution and Transformer cascade model.

[0024] Figure 2 This is a structural diagram of a micro-expression recognition system based on a graph convolution and Transformer cascade model. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that, without conflict, the embodiments and features described herein can be combined with each other. The description of these embodiments is merely illustrative and not intended to limit the scope of protection of the present invention.

[0028] Terminology Explanation: 1. Keypoints refer to the set of feature points used to describe the geometric structure of a face, automatically located in a standardized face image using a facial keypoint detection algorithm. These keypoints are typically distributed in important anatomical locations of the face, including but not limited to the eyebrow contour, eye contour, nose wing, corner of the mouth, and jawline. In specific implementations, keypoints can be obtained from existing facial keypoint detection models, whose output is the two-dimensional coordinate information of each keypoint in the image coordinate system. This invention does not limit the specific implementation method of the keypoint detection algorithm or the number of keypoints; as long as the keypoints can stably reflect the spatial structural relationships of various facial regions, they can be used in this invention for region division and graph structure construction of face images. Keypoints are mainly used to guide the division of facial regions and, based on this, to construct a static adjacency matrix representing facial topological relationships, providing structural prior information for subsequent graph convolution modeling.

[0029] 2. A deformation amplification network is a deep learning network model used to enhance subtle geometric deformations between the start and peak frames in a micro-expression sequence. This network models the structural differences between input frame pairs, extracting and amplifying deformation features reflecting minute movements in local facial regions. In a preferred embodiment of this invention, the deformation amplification network can employ a learning-based video motion amplification network structure, such as MagNet (MotionMagnification Network), to extract amplified shape features from the start and peak frames. This network can significantly enhance subtle and difficult-to-observe geometric changes such as eyebrow raising, eyelid changes, and mouth corner contraction, making local deformations related to micro-expressions more prominent. The deformation amplification network outputs multiple deformation feature vectors corresponding to local regions, each representing the degree of geometric deformation of the corresponding facial region during the micro-expression process. This invention does not limit the specific number of network layers, parameter scale, or training method of the deformation amplification network; as long as it can enhance subtle facial deformations and output region-level deformation features, it is applicable to this invention.

[0030] 3. The TV-L1 optical flow algorithm is an optical flow calculation method used to estimate pixel-level motion information between two image frames. This algorithm optimizes the pixel motion between adjacent frames by constructing an energy function containing L1 norm data terms and a Total Variation (TV) regularization term. The TV-L1 optical flow algorithm has strong noise resistance and can stably estimate pixel motion vectors even with small motion amplitudes or large illumination changes, making it particularly suitable for applications such as micro-expressions where the movement amplitude is extremely small and the duration is short. The optical flow image calculated by the TV-L1 optical flow algorithm is used to describe pixel-level motion changes, typically including horizontal motion components, vertical motion components, and corresponding motion amplitude information. In this invention, the optical flow image, as an important representation of facial dynamic information, is further input into a visual Transformer for global dynamic modeling.

[0031] 4. The Visual Transformer is a visual feature modeling network based on a self-attention mechanism, used to model the global relationships between input visual features. This model divides the input image into several image patches, maps each patch to a sequence of feature vectors, introduces positional encoding, and then uses a self-attention mechanism to model the correlations between different image patches. Unlike traditional convolutional neural networks, which mainly focus on local neighborhood features, the Visual Transformer can effectively capture long-distance dependencies across regions, making it suitable for modeling the overall dynamic collaborative relationships between different facial regions. In this invention, the Visual Transformer is mainly used to perform global dynamic encoding on the optical flow image obtained by the TV-L1 optical flow algorithm to extract global spatiotemporal features reflecting the cross-regional motion correlations of the face. This invention does not limit the specific network structure, number of layers, or parameter scale of the Visual Transformer; it can be a pre-trained model or trained or fine-tuned during the task of this invention.

[0032] Example 1 A micro-expression recognition method based on a cascaded model of graph convolution and Transformer is proposed. Micro-expressions typically exhibit characteristics such as small amplitude, short duration, and weak local changes. Therefore, the processing requires comprehensive modeling from multiple dimensions, including structure, dynamics, spatial topology, and temporal dependencies. This embodiment constructs a recognition framework that combines deformation amplification, optical flow estimation, structural prior maps, and cascaded deep networks to achieve high-precision representation of fine-grained dynamic features of micro-expressions. Figure 1 As shown, including

[0033] The starting frame and peak frame of the micro-expression sequence are obtained. Face detection, key point localization, alignment and cropping are performed on the starting frame and peak frame to obtain a standardized face image. The starting frame and peak frame pair are selected based on preset rules. In this invention, preset rules are used to determine the start frame and peak frame from the micro-expression image sequence to characterize the state changes before and after the micro-expression occurs. Specifically, the start frame is selected as the image frame before the micro-expression action occurs, when the facial expression is relatively stable and the overall movement amplitude is small, to represent the neutral state of the face; the peak frame is selected as the image frame where the micro-expression action is most significant and the local or overall facial movement amplitude reaches its maximum value, to represent the peak state of the micro-expression.

[0034] In a preferred embodiment, the starting frame and the peak frame can be directly determined based on the time-labeled information of the micro-expression sequence. Alternatively, by analyzing the degree of facial motion change or optical flow amplitude between consecutive frames, the image frame with the smallest motion change can be selected as the starting frame, and the image frame with the largest motion change can be selected as the peak frame. Finally, the starting frame and the peak frame are paired in chronological order to form a starting frame-peak frame pair, which serves as the input for subsequent deformation amplification feature extraction and optical flow calculation.

[0035] Face detection can be achieved using deep learning models, such as MTCNN, RetinaFace, and DSFD, to ensure high detection accuracy under different lighting conditions, facial poses, and occlusions. Affine transformations are applied to the detected facial regions to align key features such as the eyes and corners of the mouth, reducing the impact of pose differences.

[0036] In this step, preprocessing operations such as brightness equalization, noise removal, and color normalization can be performed on the original image to improve the robustness of subsequent feature extraction. Finally, a uniformly sized face image (e.g., 224×224) is obtained, and the starting frame and peak frame are paired as input for subsequent feature extraction based on the time annotation of the micro-expression sequence.

[0037] A pre-trained deformation amplification network is used to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions, which are then used to highlight subtle muscle movements. The TV-L1 optical flow algorithm is used to calculate the optical flow image based on the start frame and the peak frame to obtain pixel-level motion information. The pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation. The TV-L1 optical flow algorithm optimizes the solution of pixel grayscale changes between two image frames to estimate the motion vector of the corresponding pixel in the time dimension. The optical flow image obtained by the TV-L1 optical flow algorithm is used to describe pixel-level motion information, which refers to the motion changes of each pixel in the image between the starting frame and the peak frame. Specifically, it includes the horizontal motion component, the vertical motion component, and the motion amplitude information calculated from the components. This pixel-level motion information can finely characterize the subtle motion changes of local facial regions during micro-expression processes, providing input features for subsequent global dynamic modeling based on visual Transformer, thereby enhancing the expressive power of micro-expression dynamic features.

[0038] Based on facial key points, standardized face images are divided into regions, and a static adjacency matrix representing facial topological relationships is constructed. By utilizing the amplified deformation features to calculate the similarity between regions, and by weighting the static adjacency matrix, an initial graph structure with both structural priors and data-driven characteristics is obtained. The optical flow image is divided into several image blocks and mapped into a sequence of feature vectors. These blocks are then input into a visual Transformer for global dynamic modeling to obtain global spatiotemporal features across regions. The amplified deformation features are weighted and fused with global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The multimodal fusion features and the initial graph structure are input into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers. The graph convolutional network aggregates local neighborhood structure information, the Transformer captures long-distance dependencies, and the graph structure is updated in a learnable manner during training to obtain micro-expression discrimination features. The micro-expression discrimination features are classified, and the corresponding micro-expression category results are output.

[0039] Example 2 The micro-expression recognition method based on graph convolution and Transformer cascade model described in Example 1 differs in that: A pre-trained deformation amplification network is used to perform deformation enhancement processing on the starting frame and the peak frame. The deformation amplification network is used to amplify the small geometric changes in the facial region that are difficult to observe, making it easier for the model to capture the subtle facial dynamics of key areas such as the eyebrows, corners of the eyes, and corners of the mouth.

[0040] In this invention, the deformation amplification network employs an existing network architecture to enhance subtle facial deformations between the starting frame and the peak frame. In a preferred embodiment, the deformation amplification network can utilize a learning-based video motion amplification network, such as the MagNet network structure. This type of network models the differences between input frame pairs, extracting and amplifying deformation information reflecting changes in the geometric structure of local facial regions. This invention does not improve the specific network structure or internal parameters of the deformation amplification network; instead, it leverages its excellent amplification capability for minute deformations and applies it to micro-expression recognition scenarios. By enhancing the deformation of the starting and peak frames, subtle facial expression changes that are difficult to observe directly in areas such as eyebrows, corners of the eyes, and corners of the mouth can be significantly highlighted, thus providing clearer and more discriminative input features for subsequent region similarity calculation, graph structure construction, and micro-expression discrimination feature learning.

[0041] The deformation amplification network models the residual structural changes of the input starting frame and peak frame pairs, outputting deformation feature vectors for multiple local regions. Each deformation feature vector corresponds to the degree of deformation of a local facial region. This feature can reflect the dynamic geometric structure of different regions during the micro-expression process, which is of great significance for determining whether a region participates in micro-expression changes.

[0042] Residual structural change modeling: In deformation amplification networks, "residual structural change modeling" refers to enhancing subtle geometric changes by comparing the differences between the starting frame and the peak frame using residual learning methods. This is typically done by calculating the pixel differences in corresponding regions of the two frames (or using feature differences extracted using a convolutional neural network). The network focuses on these differences, learning patterns of subtle changes between frames, and then amplifies these differences through residual connections, thereby amplifying micro-expression features that are difficult to detect under normal visual conditions.

[0043] Deformation feature vectors for local regions: The deformation feature vector for each local region is obtained after processing by the deformation amplification network, and it represents the degree of geometric deformation in that region. Specifically, the deformation amplification network extracts features from facial regions (such as eyes, mouth, etc.), and then generates the feature vector for each local region by aggregating these features (such as pooling or weighted pooling). This feature vector reflects the deformation that occurs in that region during micro-expression processing, thus providing valuable information for subsequent micro-expression recognition.

[0044] The Deformation Amplification Network models the residual structural changes of the input start frame and peak frame pair. Its core lies in explicitly modeling and enhancing the geometric structural differences between the two frames. Specifically, first, the Deformation Amplification Network encodes the features of the start frame and peak frame, extracting intermediate feature representations that characterize the facial geometry. Then, by performing difference or residual calculation on the start frame and peak frame, residual information reflecting structural changes in local facial regions is obtained. This residual information is then amplified to highlight the subtle geometric deformations generated during micro-expressions. After amplification, the Deformation Amplification Network outputs a feature map containing amplified deformation information.

[0045] The onset and apex frames represent the initial and most salient states of micro-expressions. The onset frame typically represents a stable facial expression without significant changes, while the apex frame represents the state where the micro-expression is most prominent. In the deformation amplification network, these two frames are used for difference modeling to extract deformation features reflecting micro-expressions.

[0046] Residual Structure Variation Modeling Process: In deformation amplification networks, the purpose of residual structure variation modeling is to enhance subtle geometric deformations. Specifically, the network models the feature differences between the starting frame and the peak frame, typically by calculating the differences between the two frames and processing them through a residual network. The residual network can capture subtle spatial and temporal differences between these two images, especially subtle movements in local facial regions, thereby enhancing these micro-expression features. This process typically uses convolutional layers to extract image features, calculate pixel differences between the two frames, and amplify these differences.

[0047] How to calculate the deformation feature vector: The deformation feature vector is obtained by extracting features from local regions in an image. These local regions are typically determined by detecting facial landmarks. The deformation feature vector of each local region represents the geometric deformation of that region during micro-expression processing. For example, deformation features of local regions are extracted by convolutional calculation of the difference between the starting frame and the peak frame, and then the feature vector of that region is obtained through methods such as pooling or weighted convergence. This vector reflects the degree of geometric deformation of the region and supports subsequent micro-expression recognition.

[0048] Amplifying Deformation Information: Amplifying deformation information refers to the result of the network amplifying the minute geometric differences between the starting frame and the peak frame after processing by a deformation amplification network. Specifically:

[0049] After residual calculation, the subtle differences are processed through a series of steps (such as convolution operations and the application of amplification factors) to enhance these minute deformations, enabling the network to more sensitively capture changes in micro-expressions.

[0050] The output of amplified deformation information is an image in which subtle facial deformations are enhanced, making previously imperceptible micro-expression dynamics more apparent, thus providing clearer input for subsequent feature representation and micro-expression recognition.

[0051] By comparing the differences in facial regions between the starting frame and the peak frame, the deformation amplification network not only enhances the representation of these differences, but also amplifies the changes in micro-expressions between different facial regions, making these subtle changes more obvious.

[0052] Based on the region segmentation results of facial key points in a face image, the feature maps are grouped according to the corresponding local facial regions. Multiple deformation feature vectors for each local region are obtained through regional feature aggregation operations (e.g., average pooling or weighted convergence). Each deformation feature vector characterizes the degree of geometric deformation of the corresponding local facial region between the starting frame and the peak frame, thus forming a region-level deformation feature representation. This provides a basis for subsequent region similarity calculation, graph structure weighting, and micro-expression discrimination.

[0053] The TV-L1 optical flow algorithm is used to calculate the optical flow image by analyzing the starting frame and the peak frame. The TV-L1 optical flow algorithm combines the data terms of the L1 norm with the TV regularization term, which has strong noise resistance and is suitable for scenarios with extremely small movements, such as micro-expressions.

[0054] Optical flow images can describe pixel-level motion changes, including horizontal and vertical motion components and their amplitude information, and are used to characterize fine-grained dynamics during micro-expression processes. Optical flow images are formatted into a three-channel format so that they can be processed as visual images in subsequent Transformer encoding.

[0055] Optical flow images can be normalized and abnormal motion vectors can be smoothed to ensure the feature stability of the input visual Transformer.

[0056] The standardized face image is divided into regions based on facial key points; including: According to a preset set of facial landmarks, which refers to the set of feature points used to characterize the geometric structure of a face obtained from a standardized face image through a facial keypoint detection algorithm, facial landmarks are distributed in important anatomical locations of the face, including but not limited to the eyebrow contour, eye contour, nose wing, corner of mouth, and facial contour. In a specific implementation, the face can be divided into several local regions with clear semantic meanings, such as the eye region, eyebrow region, nose region, and mouth region, based on the spatial distribution of the facial landmarks. Each local region covers a set of spatially close and semantically related facial landmarks. In this way, the set of facial landmarks is used to guide the division of face regions, thereby providing a stable facial structure foundation for subsequent graph structure modeling. The face is divided into several sub-regions corresponding to the eye, eyebrow, nose wing, and corner of mouth regions, and each sub-region serves as a node in the graph structure. A static adjacency matrix is ​​constructed through the spatial proximity and anatomical relevance between nodes to represent the topological connection relationships between the facial regions. The static adjacency matrix is ​​used to represent the topological connection relationships between the various sub-regions obtained by the face region division. Specifically, each facial sub-region is treated as a node in a graph structure. Whether to establish a connection between nodes is determined based on the spatial proximity of corresponding key points and the correlation in facial anatomy. When two regions are spatially adjacent or have a direct anatomical relationship, a connection is established between their corresponding nodes; when two regions are spatially far apart and have weak anatomical correlation, no connection is established. This method allows the construction of a static adjacency matrix containing only reasonable connections. This matrix conforms to the prior knowledge of natural facial structure while avoiding redundant connections between irrelevant regions, thus providing stable topological constraints for subsequent aggregation of structural information based on graph convolution. For example, 24 or 36 local regions can be represented. Each region, as a node in the graph structure, describes its independent movement behavior in micro-expressions.

[0057] By utilizing amplified deformation features to calculate the similarity between regions, and weighting the static adjacency matrix, an initial graph structure with both structural priors and data-driven characteristics is obtained; including: For each pair of regions with static connections (defined as facial region nodes with direct connections in the static adjacency matrix), the corresponding amplified deformation feature vectors are extracted, and the cosine similarity between them is calculated. This cosine similarity is then used as a weighting coefficient and element-wise multiplied with the static adjacency matrix to obtain the initial weighted adjacency matrix, i.e., the initial graph structure. This allows the graph structure to highlight connections between regions with higher deformation correlations while preserving the facial topological prior.

[0058] The term "regions with static connectivity" refers to facial region nodes defined as having direct connections in the static adjacency matrix. Specifically, during the construction of the static adjacency matrix based on facial key points, connections between region nodes are pre-determined based on spatial proximity and anatomical relevance. Region pairs with corresponding non-zero values ​​in the static adjacency matrix are considered to have static connectivity; while region pairs without connections are excluded from subsequent similarity calculations and weighted processing. Through this definition, "regions with static connectivity" are explicitly limited to pairs of region nodes with direct topological associations under prior constraints of facial structure.

[0059] The amplified deformation feature vector is obtained from the output of the aforementioned deformation amplification network. Specifically, the deformation amplification network models the residual structural changes of the starting frame and the peak frame and generates a feature map containing amplified deformation information. Subsequently, based on the region segmentation results determined by facial key points, the feature map is divided according to the corresponding local facial regions. For each facial region, the deformation features within that region are aggregated, for example, using average pooling or weighted pooling, to obtain the amplified deformation feature vector for that region. The amplified deformation feature vector is used to characterize the degree of geometric deformation of the facial region between the starting frame and the peak frame, and serves as the input feature for subsequent region similarity calculations.

[0060] By calculating the cosine similarity between amplified deformation feature vectors of regions with static connectivity, and then using the resulting similarity as a weighting coefficient for element-wise multiplication with the static adjacency matrix, an initial weighted adjacency matrix is ​​generated. This initial weighted adjacency matrix, based on the static adjacency matrix and incorporating region deformation similarity information, represents the weighted connectivity relationships and simultaneously reflects both facial structure priors and the degree of correlation in region deformation. In this invention, the initial weighted adjacency matrix constitutes the initial graph structure used for subsequent graph convolution and graph Transformer modeling, and it can be further adaptively updated during training using a learnable adjacency matrix.

[0061] For regions with static connectivity, the corresponding amplified deformation feature vectors are extracted, and the cosine similarity is calculated, as shown below: ; in, Indicates the first i The facial region and the first j Similarity of deformation features between facial regions; used to measure the consistency of deformation behavior between two regions during micro-expression processes. , These represent the index numbers of the two facial region nodes involved in the similarity calculation. Facial region nodes are regions that are partitioned based on facial key points and have connections in the static adjacency matrix. This represents the magnified deformation feature vector corresponding to the i-th facial region. The deformation feature vector is output by the deformation magnification network and obtained by region feature aggregation, and is used to characterize the degree of geometric deformation of the region between the starting frame and the peak frame. This represents the magnified deformation feature vector corresponding to the j-th facial region.

[0062] By multiplying the similarity matrix element-wise with the static adjacency matrix, an initial weighted adjacency matrix is ​​obtained, thus enabling the graph structure to both preserve the prior facial structure and reflect the consistency of deformation behavior between different regions.

[0063] For regions with significant deformation, their adjacency weights will be significantly increased, thus allowing for greater focus on information propagation between these regions during subsequent graph convolution processes.

[0064] The optical flow image is divided into several image patches and mapped to a sequence of feature vectors. These patches are then input into a visual Transformer for global dynamic modeling to obtain cross-regional global spatiotemporal features, including: The optical flow image is divided into several image patches of a preset size (e.g., 16×16). Each image patch is flattened and linearly mapped to obtain a sequence of feature vectors. These are then superimposed with learnable positional codes and input into a visual Transformer. A self-attention mechanism is used to establish dependencies between different image patches to capture cross-regional motion correlations and overall dynamic patterns. Specifically, this includes: After inputting the optical flow image into the visual Transformer, a self-attention mechanism is used to achieve global spatiotemporal feature modeling across regions. Specifically, firstly, the optical flow image is divided into several image blocks of a preset size, and each image block is flattened and linearly mapped to obtain a corresponding feature vector representation. Simultaneously, positional encoding is introduced to preserve the spatial location information of each image block in the original image. Then, the above feature vector sequence is input into the visual Transformer, and a multi-head self-attention mechanism is used to calculate the correlation weights between different image blocks. This allows each image block to comprehensively consider the motion information of other regions during feature update, thereby establishing cross-regional dependencies. After iterative processing through multiple layers of self-attention and feedforward networks, the feature vector output by the visual Transformer not only contains local optical flow information but also integrates dynamic correlation information from different facial regions, ultimately forming a global spatiotemporal feature representation that reflects the overall facial motion pattern and inter-regional collaborative changes.

[0065] In the visual Transformer, the relevance weights are calculated using a self-attention mechanism. The specific process is as follows:

[0066] By mapping the feature representation of each image patch, we obtain the query, key, and value representations of each image patch.

[0067] The core of the self-attention mechanism is to obtain the relevance weights between image patches by calculating the similarity between the query and the key. This similarity is usually calculated using a dot product, for example:

[0068] ; in, It is a query matrix. It is a key matrix. It is a value matrix. This is the dimension of the key vector. The purpose of this formula is to measure the strength of the relationship between image patches by calculating the similarity between the query and the key, and then apply these weights to the corresponding values ​​to update the feature representation of the image patches.

[0069] In this way, the model can learn to take into account both spatial and dynamic information between image patches when updating each layer.

[0070] Iterative processing refers to processing using a multi-layered self-attention mechanism and a feedforward network. The specific steps are as follows:

[0071] First, a multi-head self-attention mechanism is used to calculate the correlation weights between image patches and apply these weights to the features of each image patch. Through this mechanism, the network can simultaneously focus on different parts of an image patch and the relationships between them, enabling each patch to learn motion information from different regions.

[0072] Next, the image patch features updated through self-attention are input into the feedforward network. The feedforward network typically includes a series of nonlinear transformations to further enhance the feature representation capability.

[0073] This process is carried out iteratively at each layer. Through multi-layer processing, the model can gradually capture more complex cross-regional and spatiotemporal dynamic relationships.

[0074] The self-attention mechanism and feedforward network in each layer update the feature representation of the image patch, enabling the image patch to take into account more information from other regions in subsequent layers of the network.

[0075] Global spatiotemporal features are feature representations processed through multiple layers of self-attention and feedforward networks, capable of comprehensively reflecting the overall facial motion patterns and coordinated changes between regions. Specifically, they include the following aspects:

[0076] Local optical flow information: that is, the motion information within the region represented by each image block, which is usually calculated by the TV-L1 optical flow algorithm.

[0077] Dynamic cross-regional associations: Through self-attention mechanisms, the model can learn motion relationships between different facial regions. For example, eyebrow movements may be related to the contraction of the corners of the mouth, and the model can establish dependencies between these regions by calculating their relationships.

[0078] Overall facial motion patterns: By modeling the entire sequence of image patches, the final output features can reflect the overall dynamics of facial expressions, such as subtle facial movement patterns.

[0079] The resulting global spatiotemporal features are high-dimensional vectors that contain the relationships between image patches and dynamic information across regions. These features can be used for subsequent micro-expression recognition tasks.

[0080] The feature acquisition method is as follows: Local optical flow features: Optical flow image patches calculated by the TV-L1 algorithm, each image patch contains motion information of the region.

[0081] Cross-regional dynamic features: Through self-attention and multi-head mechanisms, the network can automatically learn the dependencies between each image patch and other regions and calculate their correlation weights.

[0082] Global spatiotemporal features: After processing through multiple layers of self-attention and feedforward networks, the resulting feature representations integrate dynamic information from different regions and are further optimized through the network.

[0083] Global spatiotemporal features are used to characterize the overall dynamic relationship between different facial regions during micro-expression events, and serve as an important input for subsequent multimodal feature fusion and graph structure modeling.

[0084] The acquired optical flow image is divided into several fixed-size image blocks (e.g., 16×16). Each image block is flattened and linearly projected to generate a visual token, and positional encoding is added to preserve spatial information. The above token sequence is input into the visual Transformer model.

[0085] The Transformer employs a multi-head self-attention mechanism to compute correlations between different image patches, capturing long-range dynamic dependencies, such as the long-range dependency between eyebrow twitches and mouth corner compression. This process extracts overall facial dynamic patterns, providing crucial input for subsequent feature fusion.

[0086] Pre-trained visual Transformers can be used to improve feature representation capabilities.

[0087] The amplified deformation features are weighted and fused with the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; including: After mapping the amplified deformation features and global spatiotemporal features to the same feature space, they are linearly weighted according to preset initial weights, and the weights are adaptively updated during model training to balance the contributions of structural deformation information and pixel-level motion information in the overall feature representation.

[0088] The amplified deformation features and global spatiotemporal features are mapped to the same dimensional space, and then linearly weighted and fused using a learnable fusion coefficient α, as shown below: ; Where α represents the weights automatically learned through gradient descent during training; αf is a learnable weight parameter used for feature fusion, initialized as one of the network parameters during model training and automatically learned through backpropagation and gradient descent optimization. Specifically, during the training phase, the prediction results of the fused features are calculated based on the classification loss function. This error is applied to the fusion weights α through backpropagation, thus continuously updating α during training iterations to adaptively balance the contributions of deformation features and dynamic features in the overall feature representation. This is used to balance the contributions of deformation and dynamic information in the overall feature representation. This indicates that by fusing weights α, the pair and The feature representation obtained after linear weighted fusion is the multimodal fusion feature representation; through the above method, It contains both structural deformation information and dynamic motion information, and serves as the input features for subsequent graph convolution and Transformer cascade modeling. This represents the deformation features extracted by the deformation amplification network and then aggregated by region, used to characterize the geometric deformation information of each local region of the face between the starting frame and the peak frame; This represents the global spatiotemporal features obtained after global dynamic modeling of the optical flow image by a visual Transformer. These features are used to characterize cross-regional motion relationships and overall dynamic patterns between different facial regions. To prevent inconsistencies in feature scale, the input features can be normalized to ensure a more stable representation of the fused features.

[0089] The cascaded model consists of multiple cascaded graph Transformer modules. Each graph Transformer module includes a graph convolution submodule, i.e., a graph convolutional network, a self-attention submodule, and residual connection and normalization units that work together with it. The graph convolution submodule is used to: aggregate local neighborhood information of node features under the constraints of the current graph structure. The input is the node feature matrix and the corresponding adjacency matrix. By weighted aggregation of the features of neighboring nodes that are connected to the current node, a node feature representation that integrates neighborhood structure information is obtained. The self-attention submodule is used to: perform global dependency modeling on the node features output by graph convolution. It calculates the correlation weights between different nodes through the self-attention mechanism, so that each node comprehensively considers the feature information of other nodes during the feature update process, thereby establishing long-distance dependencies across regions. Residual connections and normalization operations are introduced between adjacent graph Transformer modules. The output of the current module is superimposed on its input features and normalized to alleviate the gradient vanishing problem in deep network training and improve the stability and convergence of feature representation.

[0090] This method aggregates local neighborhood structure information through graph convolutional networks, captures long-range dependencies through Transformers, and performs learnable updates to the graph structure during training to obtain micro-expression discrimination features; including: The multimodal fusion features and the initial graph structure are input together into the first graph Transformer module. First, the graph convolution submodule performs local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint, so that the node features are fused with the structural and deformation information of the adjacent regions, thereby obtaining an intermediate feature representation containing local topological relationships. Subsequently, the self-attention submodule performs global modeling of the intermediate feature representation. By calculating the correlation weights between any two region nodes, the node features perceive the dynamic relationships between distant regions. This process is repeated layer by layer in multiple graph Transformer modules. As the network depth increases, the node features gradually evolve from local structural information to high-level semantic features that integrate global dynamic relationships. Simultaneously, during training, a learnable adjacency matrix is ​​introduced to adaptively update the initial graph structure, allowing the graph structure to dynamically adjust connection weights based on the facial motion patterns of different samples.

[0091] The self-attention submodule performs global modeling of intermediate feature representations: The self-attention submodule is primarily responsible for handling global dependencies between image regions. Its implementation process is as follows:

[0092] Calculating Relevance Weights: In the self-attention mechanism, the relevance weights between each node and other nodes are calculated. Specifically, first, the features of each node are mapped to a query, key, and value vector space. Next, the similarity between the query vector and the key vector is calculated (usually via a dot product), yielding the relevance weights between each pair of nodes.

[0093] ; in, It is a query vector. It is a key vector. It is a value vector. This is the dimension of the key vector. The calculated attention weights are used to weight the value vector of each node, ultimately updating the node's features.

[0094] Cross-regional dynamic association: By calculating similarity weights between image patches, the network can perceive the motion relationships between different facial regions, especially dynamic changes over long distances. This process helps the model capture global dynamic information about facial movements.

[0095] Adaptive update of initial graph structure: The initial graph structure is defined by a static adjacency matrix, initialized based on the anatomical structure of the face and the spatial relationships between keypoints. However, during training, the graph structure is adaptively updated based on input micro-expression samples. Specifically:

[0096] Learnable adjacency matrix: During training, a learnable adjacency matrix is ​​introduced to dynamically adjust the connection weights of the graph structure. This matrix is ​​updated based on the facial motion patterns of the current sample during each forward propagation.

[0097] Update formula: By introducing a learnable adjacency matrix The connection strength between each pair of nodes can be updated using the following formula: ; in, It is a static adjacency matrix based on facial anatomy. It is an adjustable matrix learned during training. It is an activation function.

[0098] Graph structure update: During training, the network adjusts the connection weights between each node according to the facial movement patterns of different samples, making the graph structure more flexible and better able to adapt to different micro-expression data.

[0099] After being processed by multiple cascaded graph Transformer modules, the final node features or their converged results constitute the micro-expression discrimination features. These features include both local facial structural relationships and cross-regional dynamic dependency information. They are then input into subsequent classification modules for micro-expression category determination.

[0100] After processing by multiple graph Transformer modules, the final node features or their converged results constitute the micro-expression discrimination features. These features include:

[0101] Local facial structural relationships: Through graph convolution and self-attention mechanisms, the network can learn the structural relationships between various regions of the face, such as the relative positions and movements of the eyes, mouth and eyebrows.

[0102] Cross-regional dynamic dependency information: Through the self-attention mechanism, the network can capture the dynamic dependencies between different facial regions, especially the cross-regional collaborative movements during micro-expression changes.

[0103] These features are then fed into the subsequent classification module for micro-expression category determination, helping the model to classify facial expressions based on their overall dynamic patterns.

[0104] The graph convolutional submodule performs local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint; this process is based on Graph Convolutional Networks (GCN). By performing local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint, the node features are fused with the structural and deformation information of adjacent regions, thereby obtaining an intermediate feature representation containing local topological relationships; the specific implementation process is as follows:

[0105] Initial input: The input to the Graph Convolutional Network (GCN) is the node features of each facial region and the corresponding adjacency matrix. The adjacency matrix defines the connection relationship between nodes in the graph; for example, whether there is a connection between facial regions.

[0106] Aggregation process: The graph convolution submodule aggregates the features of each node using an adjacency matrix; in each graph convolution layer, the features of a node are weighted and averaged with the features of its neighboring nodes to obtain a new feature representation; the specific formula is as follows: ; in, It is the first The node feature matrix of the layer; It is a standardized adjacency matrix, which includes structural information between nodes; It is the first The learnable weight matrix of the layer; It is an activation function (such as ReLU); It is the first +1 layer node feature matrix; Aggregated features: The new feature representations are generated by aggregating graph convolutions, which integrate the local topological relationships and deformation information of nodes to produce intermediate feature representations.

[0107] The local topological relationship refers to the node connection structure formed by the region division guided by facial key points and its adjacency matrix constraints. The adjacency matrix determines the set of adjacent nodes and limits the range of local information propagation based on the spatial proximity relationship and anatomical relevance between regions. The deformation information refers to the region-level amplified deformation feature vector extracted from the starting frame and peak frame pairs and aggregated by region by the deformation amplification network, which is used to characterize the degree of geometric deformation of the corresponding facial region. Furthermore, the adjacency matrix can be weighted based on the similarity of the deformation feature vectors of adjacent regions, so that adjacent regions with higher deformation synergy contribute more in graph convolution aggregation, thereby realizing the joint fusion of structural topological constraints and deformation features in the local neighborhood aggregation process to generate intermediate feature representations.

[0108] The initial graph structure is updated during training using a learnable adjacency matrix. Based on the initial weighted adjacency matrix, a learnable weight matrix equal to the number of nodes is introduced to adjust the connection strength of each edge. The updated adjacency matrix is ​​then normalized using the sigmoid function. This ensures the numerical stability of the graph structure and its adaptability to different facial motion patterns in various samples.

[0109] The classification module includes a fully connected layer and an activation function layer. It outputs the probability distribution of each target micro-expression category by performing nonlinear mapping on the micro-expression discrimination features, and determines the final micro-expression recognition result based on preset decision rules. The cascaded model uses the cross-entropy loss function during training to optimize the network parameters in a supervised manner.

[0110] Example 3 A micro-expression recognition system based on a graph convolution and Transformer cascade model, such as Figure 2 As shown, it includes: The data acquisition and preprocessing module is configured to: acquire the start frame and peak frame of the micro-expression sequence, perform face detection, key point localization, alignment and cropping on the start frame and peak frame to obtain a standardized face image, and select the start frame and peak frame pair based on preset rules; The deformation feature extraction module is configured to: use a pre-trained deformation amplification network to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions in order to highlight subtle muscle movements; The optical flow feature extraction module is configured to: calculate the optical flow image based on the start frame and the peak frame using the TV-L1 optical flow algorithm to obtain pixel-level motion information; the pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation; The graph structure construction module is configured to: divide a standardized face image into regions based on facial key points, construct a static adjacency matrix representing the topological relationships of the face; calculate the similarity between regions using magnified deformation features, weight the static adjacency matrix, and obtain an initial graph structure that combines structural priors and data-driven characteristics. The global dynamic modeling module is configured to divide the optical flow image into several image blocks and map them into a sequence of feature vectors, input them into the visual Transformer for global dynamic modeling, and obtain global spatiotemporal features across regions. The feature fusion module is configured to: perform weighted fusion of the amplified deformation features and the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The Graph-Transformer Cascaded Modeling Module is configured to: input multimodal fusion features and initial graph structure into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers; aggregate local neighborhood structure information through graph convolutional networks; capture long-distance dependencies through Transformers; and perform learnable updates to the graph structure during training to obtain micro-expression discrimination features. The classification module is configured to output micro-expression category recognition results based on micro-expression discrimination features.

[0111] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the micro-expression recognition method based on graph convolution and Transformer cascade model as described in any of Embodiments 1-3.

[0112] In this embodiment, the computer-readable storage medium may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, flash memory, USB flash drive, solid-state drive (SSD), memory card, or any medium capable of storing instructions that can be read by a processor. Those skilled in the art will understand that any storage medium capable of electronically storing computer programs can be implemented using the method described in this embodiment.

[0113] The program stored on a computer-readable storage medium includes a set of instructions for performing all the steps of Embodiment 1, specifically including but not limited to: When the processor executes the program, it performs the following operations: (1) Obtain the micro-expression start frame and peak frame, perform image detection, alignment and cropping, and complete data preprocessing; (2) Use a deformation amplification network to extract the amplified deformation features of the starting frame and the peak frame; (3) The TV-L1 optical flow algorithm is used to calculate the optical flow image and extract pixel-level dynamic information; (4) Divide the region based on facial key points and construct a static structural prior map; (5) Calculate the similarity between regions using deformation features, and perform weighted updates on the static adjacency matrix to form an initial weighted graph structure; (6) Perform image block division and Transformer encoding on the optical flow image to obtain global dynamic features; (7) Learnable weighted fusion of deformation features and dynamic features to form a multimodal feature representation; (8) A cascaded model of graph convolution and Transformer stacking is used for deep modeling to output micro-expression discrimination features; (9) Input the discriminant features into the classification module and output the micro-expression recognition results.

[0114] The above-described program execution steps are completely consistent with the method steps in Embodiment 1. Those skilled in the art should understand that as long as the program logic corresponds to Embodiment 1, it can be regarded as achieving the technical effect of the present invention.

[0115] The computer program in this embodiment can be stored in the computer-readable storage medium as source code, object code, script file, or any form of executable instructions; the program can be stored separately or packaged as part of an application or as part of an operating system.

[0116] The program can be written to the computer-readable storage medium via network download, copying to removable media, remote push, or other methods. This invention does not limit the method of program distribution; as long as the program is stored in the storage medium and executed by a processor, the technical solution of this invention can be achieved.

[0117] The micro-expression recognition method of the present invention can be ported to any hardware device with processing capabilities through the computer-readable storage medium of this embodiment, enabling it to automatically recognize micro-expressions, thereby achieving flexible deployment across platforms and devices.

[0118] Example 5 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the micro-expression recognition method based on graph convolution and Transformer cascade model as described in any of Embodiments 1-3.

[0119] The processor can be a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or any other processor capable of executing program instructions. This invention does not limit the type of processor, as long as it can execute the program logic in this embodiment.

[0120] The memory may include a cache, read-only memory (ROM), random access memory (RAM), hard disk, flash memory, solid-state drive (SSD), memory card, or other computer-readable storage medium capable of storing program instructions and data. The memory is used to store program instructions for micro-expression recognition, deformation feature data, optical flow feature data, graph structure data, and intermediate features during the model inference process.

[0121] In this embodiment, the electronic device may further include a communication interface, a display device, and an input device. The communication interface is used for data exchange between devices, including a wired transmission interface and a wireless network interface; the display device is used to display the micro-expression recognition results; and the input device is used to provide operation instructions to the electronic device, such as gesture input, touch input, or button input.

[0122] When the processor executes a computer program stored in memory, it performs the following operations: (1) Obtain the start frame and peak frame of the micro-expression sequence through the data acquisition unit; (2) Use the deformation amplification network to extract local geometric deformation features; (3) Calculate optical flow images based on the TV-L1 model and extract dynamic motion features; (4) Divide the region based on facial key points and construct a static adjacency matrix; (5) Calculate the region similarity using deformation features and update the graph structure using weighted average; (6) Perform Transformer encoding on the optical flow image to obtain global dynamic features; (7) The deformation amplification features and optical flow dynamic features are weighted and fused to obtain a multimodal feature representation; (8) Perform deep modeling on the fused feature input graph – Transformer cascade network and update the graph structure; (9) Output the corresponding micro-expression category results based on the classification module.

[0123] Optionally, the electronic device may be a server, edge computing device, smart camera, mobile terminal, or embedded device. By deploying the micro-expression recognition program of the present invention on different types of electronic devices, automatic micro-expression recognition can be achieved in monitoring systems, psychological assessment systems, intelligent human-computer interaction devices, and behavior recognition platforms.

[0124] Furthermore, when the electronic device is a smart mobile terminal, the processor can be a mobile SOC (System-on-Chip), the memory is onboard flash memory, and the program can be installed via OTA or an app store. When the electronic device is a smart surveillance camera, the processor can be an embedded AI chip, and the program can be updated via network push. This invention does not limit the hardware form of the electronic device; as long as the device can execute the method steps of this invention, it is considered to fall within the protection scope of this invention.

[0125] In summary, the electronic device provided in this embodiment can implement a micro-expression recognition method based on graph convolution and Transformer cascade model by executing the corresponding program, thereby completing the automatic detection and recognition of micro-expressions. It has the characteristics of high efficiency, accuracy and flexible deployment.

Claims

1. A micro-expression recognition method based on a graph convolution and Transformer cascade model, characterized in that, include The starting frame and peak frame of the micro-expression sequence are obtained. Face detection, key point localization, alignment and cropping are performed on the starting frame and peak frame to obtain a standardized face image. The starting frame and peak frame pair are selected based on preset rules. A pre-trained deformation amplification network is used to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions, which are then used to highlight subtle muscle movements. The TV-L1 optical flow algorithm is used to calculate the optical flow image based on the start frame and the peak frame to obtain pixel-level motion information. The pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation. Based on facial key points, standardized face images are divided into regions, and a static adjacency matrix representing facial topological relationships is constructed. By utilizing the amplified deformation features to calculate the similarity between regions, and by weighting the static adjacency matrix, an initial graph structure with both structural priors and data-driven characteristics is obtained. The optical flow image is divided into several image patches and mapped into a sequence of feature vectors. These patches are then input into a visual Transformer for global dynamic modeling to obtain global spatiotemporal features across regions. The amplified deformation features are weighted and fused with the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The multimodal fusion features and the initial graph structure are input into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers. The graph convolutional network aggregates local neighborhood structure information, the Transformer captures long-distance dependencies, and the graph structure is updated in a learnable manner during training to obtain micro-expression discrimination features. The micro-expression discrimination features are classified, and the corresponding micro-expression category results are output.

2. The micro-expression recognition method based on graph convolution and Transformer cascade model according to claim 1, characterized in that, A pre-trained deformation amplification network is used to perform deformation enhancement processing on the starting frame and the peak frame; The deformation amplification network models the residual structure change of the input starting frame and peak frame pair, and outputs deformation feature vectors of multiple local regions. Each deformation feature vector corresponds to the degree of deformation of a local facial region. First, the deformation amplification network encodes the features of the starting frame and the peak frame, extracting intermediate feature representations that characterize the geometric structure of the face. Then, by performing difference or residual calculation on the starting frame and the peak frame, residual information reflecting the structural changes of local facial regions is obtained, and the residual information is amplified. After amplification, the deformation amplification network outputs a feature map containing amplified deformation information.

3. The micro-expression recognition method based on graph convolution and Transformer cascade model according to claim 1, characterized in that, The standardized face image is divided into regions based on facial key points; including: Based on a pre-defined set of facial landmarks, the face is divided into several sub-regions, each of which serves as a node in a graph structure. A static adjacency matrix is ​​constructed using the spatial proximity and anatomical correlation between nodes to represent the topological connections between the facial regions.

4. The micro-expression recognition method based on graph convolution and Transformer cascade model according to claim 1, characterized in that, By utilizing amplified deformation features to calculate the similarity between regions, and weighting the static adjacency matrix, an initial graph structure with both structural priors and data-driven characteristics is obtained; including: For each pair of regions with static connection relationships, which are defined as facial region nodes with direct connection relationships in the static adjacency matrix, the corresponding magnified deformation feature vectors are extracted, the cosine similarity between the two is calculated, and the obtained cosine similarity is used as a weight coefficient and multiplied element-wise with the static adjacency matrix to obtain the initial weighted adjacency matrix, i.e. the initial graph structure. For regions with static connectivity, the corresponding amplified deformation feature vectors are extracted, and the cosine similarity is calculated, as shown below: ； in, Indicates the first i The facial region and the first j Similarity of deformation features between facial regions; This represents the magnified deformation feature vector corresponding to the i-th facial region; This represents the magnified deformation feature vector corresponding to the j-th facial region.

5. The micro-expression recognition method based on graph convolution and Transformer cascade model according to claim 1, characterized in that, The optical flow image is divided into several image patches and mapped to a sequence of feature vectors. These patches are then input into a visual Transformer for global dynamic modeling to obtain cross-regional global spatiotemporal features, including: The optical flow image is divided into several image patches of a preset size. Each image patch is flattened and linearly mapped to obtain a sequence of feature vectors. These sequences are then superimposed with learnable positional codes and input into a visual Transformer. A self-attention mechanism is used to establish dependencies between different image patches to capture cross-regional motion correlations and overall dynamic patterns. Specifically, this includes: The feature vector sequence is input into the visual Transformer, and the correlation weights between different image patches are calculated through a multi-head self-attention mechanism. After iterative processing by multi-layer self-attention and feedforward networks, the feature vector output by the visual Transformer not only contains local optical flow information, but also integrates dynamic correlation information from different facial regions, and finally forms a global spatiotemporal feature representation that can reflect the overall facial motion pattern and the coordinated changes between regions.

6. The micro-expression recognition method based on a graph convolution and Transformer cascade model according to claim 1, characterized in that, The amplified deformation features are weighted and fused with the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; including: The amplified deformation features and global spatiotemporal features are mapped to the same dimensional space, and then linearly weighted and fused using a learnable fusion coefficient α, as shown below: ； Where α is the weight automatically learned through gradient descent during training; This indicates that by fusing weights α, the pair and The feature representation obtained after linear weighted fusion is the multimodal fusion feature representation; This represents the deformation features extracted by the deformation amplification network and then aggregated by region, used to characterize the geometric deformation information of each local region of the face between the starting frame and the peak frame; This represents the global spatiotemporal features obtained by performing global dynamic modeling of optical flow images using a visual Transformer, used to characterize cross-regional motion associations and overall dynamic patterns between different facial regions; The cascaded model includes multiple cascaded graph Transformer modules. Each graph Transformer module includes a graph convolution submodule, i.e., a graph convolutional network, a self-attention submodule, and residual connection and normalization units that work together with it. The graph convolution submodule is used to: aggregate local neighborhood information of node features under the constraints of the current graph structure. The input is the node feature matrix and the corresponding adjacency matrix. By weighted aggregation of the features of neighboring nodes that are connected to the current node, a node feature representation that integrates neighborhood structure information is obtained. The self-attention submodule is used to: perform global dependency modeling on the node features output by graph convolution. It calculates the correlation weights between different nodes through the self-attention mechanism, so that each node comprehensively considers the feature information of other nodes during the feature update process, thereby establishing long-distance dependencies across regions. Residual connections and normalization operations are introduced between adjacent graph Transformer modules. This method aggregates local neighborhood structure information through graph convolutional networks, captures long-range dependencies through Transformers, and performs learnable updates to the graph structure during training to obtain micro-expression discrimination features; including: The multimodal fusion features and the initial graph structure are input together into the first graph Transformer module. First, the graph convolution submodule performs local neighborhood aggregation on the features of each facial region node under the adjacency matrix constraint, so that the node features are fused with the structural and deformation information of the adjacent regions, thereby obtaining an intermediate feature representation containing local topological relationships. Subsequently, the self-attention submodule performs global modeling of the intermediate feature representation. By calculating the correlation weight between any two region nodes, the node features perceive the dynamic relationship between distant regions. At the same time, during the training process, the initial graph structure is adaptively updated by introducing a learnable adjacency matrix, so that the graph structure dynamically adjusts the connection weights according to the facial motion patterns of different samples. After being processed by multiple cascaded graph Transformer modules, the final node features or their convergence results constitute the micro-expression discrimination features, which include both local facial structural relationships and cross-regional dynamic dependency information.

7. A micro-expression recognition method based on a graph convolution and Transformer cascade model according to any one of claims 1-6, characterized in that, The graph convolution submodule performs local neighborhood aggregation on the features of each facial region node under the constraint of the adjacency matrix, enabling the node features to fuse with the structural and deformation information of adjacent regions, thereby obtaining an intermediate feature representation containing local topological relationships; the specific implementation process is as follows: Initial input: The input to the Graph Convolutional Network (GCN) is the node features of each facial region and the corresponding adjacency matrix. The adjacency matrix defines the connection relationship between nodes in the graph. Aggregation process: The graph convolution submodule aggregates the features of each node using an adjacency matrix; in each graph convolution layer, the features of a node are weighted and averaged with the features of its neighboring nodes to obtain a new feature representation; the specific formula is as follows: ； in, It is the first The node feature matrix of the layer; It is a standardized adjacency matrix, which includes structural information between nodes; It is the first The learnable weight matrix of the layer; It is an activation function; It is the first +1 layer node feature matrix; Aggregated features: The new feature representations after graph convolution are aggregated, and the local topological relationships and deformation information of the nodes are fused to generate intermediate feature representations; The initial graph structure is updated during training using a learnable adjacency matrix. Based on the initial weighted adjacency matrix, a learnable weight matrix with the same number of nodes is introduced to adjust the connection strength of each edge, and the updated adjacency matrix is ​​normalized using the Sigmoid function. The classification module includes a fully connected layer and an activation function layer. It outputs the probability distribution of each target micro-expression category by performing nonlinear mapping on the micro-expression discrimination features, and determines the final micro-expression recognition result based on preset decision rules. The cascaded model training process uses the cross-entropy loss function to optimize the network parameters in a supervised manner.

8. A micro-expression recognition system based on a graph convolution and Transformer cascade model, characterized in that, include: The data acquisition and preprocessing module is configured to: acquire the start frame and peak frame of the micro-expression sequence, perform face detection, key point localization, alignment and cropping on the start frame and peak frame to obtain a standardized face image, and select the start frame and peak frame pair based on preset rules; The deformation feature extraction module is configured to: use a pre-trained deformation amplification network to process the starting frame and peak frame pairs to extract amplified deformation features of local facial regions in order to highlight subtle muscle movements; The optical flow feature extraction module is configured to: calculate the optical flow image based on the start frame and the peak frame using the TV-L1 optical flow algorithm to obtain pixel-level motion information; the pixel-level motion information includes the horizontal motion component of the pixel, the vertical motion component of the pixel, and the motion amplitude information obtained by calculation; The graph structure construction module is configured to: divide a standardized face image into regions based on facial key points, construct a static adjacency matrix representing the topological relationships of the face; calculate the similarity between regions using magnified deformation features, weight the static adjacency matrix, and obtain an initial graph structure that combines structural priors and data-driven characteristics. The global dynamic modeling module is configured to divide the optical flow image into several image blocks and map them into a sequence of feature vectors, input them into the visual Transformer for global dynamic modeling, and obtain global spatiotemporal features across regions. The feature fusion module is configured to: perform weighted fusion of the amplified deformation features and the global spatiotemporal features to form a multimodal fusion feature representation of micro-expressions; The Graph-Transformer Cascaded Modeling Module is configured to: input multimodal fusion features and initial graph structure into a cascaded model consisting of alternating stacks of graph convolutional networks and Transformers; aggregate local neighborhood structure information through graph convolutional networks; capture long-distance dependencies through Transformers; and perform learnable updates to the graph structure during training to obtain micro-expression discrimination features. The classification module is configured to output micro-expression category recognition results based on micro-expression discrimination features.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it is used to implement the steps in the micro-expression recognition method based on graph convolution and Transformer cascade model as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps in the micro-expression recognition method based on graph convolution and Transformer cascade model as described in any one of claims 1-7.

Images