Facial area prior guidance-based expression recognition method and system

Abstract

The invention provides an expression recognition method based on facial region prior guidance. The method comprises the following steps: acquiring a face image data set containing expression category labels; the method comprises the following steps: constructing an expression recognition model based on facial region prior guidance, wherein the model comprises a data processing layer, a two-way token generation module, a sequence fusion and position coding layer, a hierarchical prior guidance Transform encoder module, a global feature fusion layer and a classification layer; using samples in the facial expression data set to train an expression recognition model based on facial area prior guidance; and performing facial expression recognition on a newly input test picture by using the trained model, and outputting an expression category. According to the invention, through a layered learnable attention guiding mechanism, the prior depth of the face area is fused into the model reasoning process, and the recognition accuracy and robustness in a complex scene are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a method and system for facial expression recognition based on prior guidance of facial regions. Background Technology

[0002] In recent years, the rapid development of modern information technology, represented by artificial intelligence, has profoundly changed the mode of human-computer interaction. Enabling machines to recognize and understand human emotions is a key link in achieving natural, efficient, and intelligent human-computer interaction. Facial Expression Recognition (FER), as a core technology of affective computing and human-computer interaction, aims to automatically identify human emotional states by analyzing facial images or video sequences. It has broad application prospects in fields such as intelligent healthcare, fatigue driving monitoring, and intelligent robotics.

[0003] Facial expression recognition methods have mainly gone through the development stages from traditional handmade features, to shallow machine learning models, and finally to deep learning models. The first two types of methods rely heavily on manually designed feature extractors. Although they can achieve certain results in controlled laboratory environments, their feature representation capabilities are limited, and they generally suffer from insufficient generalization ability and poor robustness, making it difficult to cope with the complex changes in real-world application scenarios.

[0004] With the development of large-scale data and computing hardware, deep learning methods have made groundbreaking progress in image understanding. Deep learning-based methods can automatically learn the mapping relationship from face images to expression categories in an "end-to-end" manner, effectively avoiding the limitations of manual feature design in traditional methods. Currently, methods based on convolutional neural networks (CNN) and visual transformers (ViT) have become the mainstream in the field of expression recognition. However, they each have their inherent shortcomings in expression recognition tasks: CNN models are good at extracting local features, but their global modeling ability is limited, making it difficult to capture complex collaborative change patterns between multiple facial regions. Their performance also drops significantly when faced with complex scenes such as occlusion and pose changes. Although ViT models can effectively suppress occlusion interference by globally modeling image sequence relationships through self-attention, their standard block processing method treats the face as a series of undifferentiated image blocks, lacking local inductive bias for facial regional structures. This causes the model to fail to prioritize regions highly related to expressions (such as eyes and mouth) and may be interfered with by irrelevant information such as background and hair, thus affecting learning efficiency and accuracy.

[0005] While existing research attempts to combine the advantages of CNNs and ViT, or to enhance model performance by introducing prior information such as facial landmarks and semantic segmentation, these methods are still largely limited to relatively shallow and static approaches such as early feature splicing. These methods fail to transform the inherent spatial relationships and semantic dependencies between facial regions into a mechanism capable of dynamic learning and hierarchical guidance at a deeper network level. This makes it difficult for the model to achieve progressive reasoning from local detail perception to global semantic understanding, resulting in significant challenges to recognition accuracy and robustness in complex scenarios such as occlusion, lighting changes, and pose diversity. Therefore, how to deeply integrate facial region priors into the model architecture and construct a dynamically adjustable, hierarchically learnable attention guidance mechanism has become crucial for improving facial expression recognition performance. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing facial expression recognition technologies, particularly addressing the difficulties CNNs face in modeling long-range dependencies and the lack of local inductive bias and insufficient utilization of prior knowledge in ViT. This invention proposes a facial expression recognition method and system based on prior knowledge guidance of facial regions. The aim is to introduce prior knowledge of facial semantic regions into the self-attention calculation of the Transformer encoder through a learnable, hierarchical attention bias matrix, thereby achieving more accurate and robust facial expression recognition. The technical solution provided by this invention is as follows:

[0007] Firstly, an expression recognition method based on facial region prior guidance includes the following steps:

[0008] Step 1: Obtain a dataset of face images containing facial expression category labels;

[0009] Step 2: Construct an expression recognition model based on facial region prior guidance. The model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer.

[0010] The data processing layer preprocesses the input face image, including resizing, data augmentation, and normalization.

[0011] The dual-path token generation module includes a region token generation unit, a spatial feature token generation unit, and a common convolutional augmentation embedding module, which generates a region token sequence that encodes local semantic information and a spatial feature token sequence that encodes global context information. The region token sequence is obtained based on a pre-trained face parsing model.

[0012] The sequence fusion and location encoding layer concatenates the region token sequence and the spatial feature token sequence and adds them to the fused location encoding element by element. After passing through a normalization layer and a random deactivation layer, the encoder input sequence is obtained.

[0013] The hierarchical prior-guided Transformer encoder module introduces learnable region prior attention bias matrices at different layers of the encoder to implement facial region prior guidance. The facial region priors include facial semantic priors implemented through a pre-trained face parsing model, and facial region collaborative priors implemented through hierarchical initialization of bias matrices.

[0014] The global feature fusion layer extracts and enhances the output sequence of the Transformer encoder module, performs weighted fusion on the enhanced region token features, and outputs an image-level global feature vector.

[0015] The classification layer classifies the global feature vector and outputs the facial expression category;

[0016] Step 3: Train the expression recognition model based on facial region prior using a face image dataset, adopt the Adam method as the optimization strategy, and select cross-entropy as the loss function for gradient backpropagation.

[0017] Step 4: Use the trained model to perform facial expression recognition on the newly input test image and output the expression category.

[0018] Preferably, the specific structure of the dual-path token generation module is as follows:

[0019] The region token generation unit is configured to: call a pre-trained face parsing model to obtain a pixel-level semantic segmentation map of the input face image; extract M facial semantic regions according to predefined mapping rules; for each semantic region, generate its binary mask and calculate the minimum bounding rectangle; crop out the region image patch and scale it to a fixed size; independently input each scaled region image patch into the convolutional enhancement embedding module to obtain the corresponding region feature map; input each region feature map into a global average pooling layer to generate a feature vector encoding local semantic information as a region token; and combine all M region tokens into a region token sequence.

[0020] The spatial feature token generation unit is configured to: input the entire input face image into the convolutional enhancement embedding module for processing to obtain a feature map, and then input the feature map into an adaptive average pooling layer and a single flattening layer in sequence to generate a feature vector sequence that encodes global context information, which serves as the spatial feature token sequence, with the number of tokens being N.

[0021] The convolutional enhancement embedding module, serving as a shared feature extractor for both the region token generation unit and the spatial feature token generation unit, is a three-level convolutional structure. The first-level convolutional component includes a convolutional layer, a batch normalization layer, and an activation layer. Its convolutional layers use 7×7 kernels with a stride of 2 and padding of 3, expanding the number of channels to one-quarter of the hidden dimension. The second-level convolutional component includes a convolutional layer, a batch normalization layer, and an activation layer. Its convolutional layers use 3×3 kernels with a stride of 1 and padding of 1, expanding the number of channels to one-half of the hidden dimension. The third-level convolutional component includes a convolutional layer and a batch normalization layer. Its convolutional layers use 3×3 kernels with a stride of 1 and padding of 1, expanding the number of channels to the full hidden dimension.

[0022] Preferably, the fusion position coding in the sequence fusion and position coding layer is obtained by concatenating two independent learnable position coding matrices, including a regional position coding for representing the semantic region order relationship and a spatial position coding based on a two-dimensional image grid.

[0023] Preferably, the learnable region prior attention bias matrix in the hierarchical prior-guided Transformer encoder module The structure is (M+N)×(M+N), where the top-left M×M submatrix is ​​initialized according to the hierarchy to achieve collaborative priors for facial regions, and the rest is initialized as a zero matrix.

[0024] Preferably, the method of initializing submatrices according to hierarchy to achieve collaborative prior knowledge of facial regions specifically includes:

[0025] The initial value of the bias matrix of the shallow encoder is based on the region spatial proximity matrix, which is obtained by calculating the Euclidean distance between the center points of the region bounding box and transforming it using the Gaussian kernel function;

[0026] The initial value of the bias matrix of the middle layer encoder is based on a dual attention dynamic fusion mechanism, specifically: extracting region token features from the current layer input sequence; projecting the region token features into a query matrix and a key matrix, and calculating self-attention weights; projecting the region token features into a query matrix, using the facial prototype feature library as the key matrix, and calculating prototype attention weights; generating a gating signal based on the region token features; using the gating signal to fuse the self-attention weight matrix and the prototype attention weight matrix to obtain a matrix, which serves as the prior bias source for the middle layer.

[0027] The bias matrix of the deep encoder is initialized to zero, and no explicit prior is applied.

[0028] Preferably, the facial prototype feature library is constructed using the following method:

[0029] Obtain a large-scale face dataset;

[0030] For each image in the large-scale face dataset, the processing flow of the region token generation unit in step 2 is used to extract the region token of each semantic region;

[0031] For each semantic region, K-Means clustering is performed on all region tokens extracted from the large-scale face dataset, with the number of clusters ranging from 5 to 10, and each cluster center serving as a prototype;

[0032] Combine the prototypes of all regions to form a facial prototype feature library.

[0033] Preferably, the global feature fusion layer is configured as follows:

[0034] The first M vectors are extracted from the output sequence of the hierarchical prior-guided Transformer encoder module to obtain the enhanced region token feature matrix;

[0035] The enhanced region token feature matrix is ​​input into an adaptive weighted network to generate importance weights for each region token, and then the weighted sums are performed to obtain an image-level global feature vector.

[0036] Preferably, the adaptive weighted network includes a weight generation unit and a weighted summation unit;

[0037] The weight generation unit consists of a linear projection layer and a softmax layer connected in sequence. The enhanced region token feature matrix is ​​input into the linear projection layer and mapped to a scalar. Then, it is normalized by the softmax layer to generate a corresponding adaptive importance weight for each region token.

[0038] The weighted summation unit is configured to use a normalized weight vector to perform a weighted summation on the region token feature matrix, outputting an image-level global feature vector.

[0039] Secondly, an expression recognition system based on prior knowledge guidance of facial regions includes:

[0040] The sample library module is used to acquire and store a dataset of face images containing facial expression category labels;

[0041] The facial expression recognition model building module is used to build an facial expression recognition model based on facial region prior guidance. The model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer.

[0042] The model training module is used to train an expression recognition model based on facial region prior guidance using samples from the face image dataset, and save the trained expression recognition model.

[0043] The facial expression recognition module is used to perform facial expression recognition on newly input test images using a trained facial expression recognition model and output the expression category.

[0044] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0045] (1) By constructing a learnable, hierarchical attention bias matrix, the facial region prior is injected into the model in a “soft constraint” manner, which not only provides effective initialization guidance, but also allows the model to adaptively adjust according to the data during training, avoiding the limitations brought by hard priors, and achieving an effective balance between prior guidance and adaptive learning.

[0046] (2) The layered guidance strategy of shallow (spatial proximity), middle (statistical association) and deep (autonomous semantics) simulates the cognitive reasoning process of humans from local perception to global understanding, enabling the network to perform hierarchical reasoning and significantly improving the ability to recognize complex and subtle expressions.

[0047] (3) The facial prototype feature library provides typical prototype references across data, enabling the model to learn and build reliable inter-regional attention associations in the mid-level attention guidance process, which further enhances the model's ability to distinguish subtle expressions.

[0048] (4) By using the dual-path token generation module, the local semantic details of the face and the global spatial context information are captured at the same time, forming a feature representation with richer information and stronger discriminative power. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is a structural diagram of an expression recognition model based on facial region prior guidance;

[0051] Figure 2 This is a structural diagram of the convolutional enhancement embedding module;

[0052] Figure 3 This is a structural diagram of the dual-path token generation module;

[0053] Figure 4 This is a structural diagram of the sequence fusion and position coding layer;

[0054] Figure 5This is a structural diagram of the layered prior-guided Transformer encoder module; Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Example 1: A facial expression recognition method based on facial region prior guidance, comprising the following steps:

[0058] Step 1: Obtain a face image dataset containing expression category labels. The method of this invention uses the publicly available face image dataset FER2013 for model training and validation. Alternatively, other publicly available face image data can be used, or image data can be collected independently to create a face image dataset containing expression category labels.

[0059] Step 2: Construct an expression recognition model guided by prior knowledge of facial regions, such as... Figure 1 As shown, the model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer.

[0060] Step 3: Use samples from the face image dataset to train the expression recognition model based on facial region priors. During training, the Adam method is used as the optimization strategy, and cross-entropy is selected as the loss function for gradient backpropagation.

[0061] Step 4: Use the trained model to perform facial expression recognition on the newly input test image and output the expression category.

[0062] The facial expression recognition model based on prior knowledge of facial regions is as follows:

[0063] The data processing layer preprocesses the input face image, including resizing, data augmentation, and normalization. First, the image is uniformly scaled to 256×256 pixels. Then, it is randomly cropped within a scale range of 60%-100% and resized to the target size of 224×224 pixels. Spatial robustness is increased through random horizontal flipping, rotation within ±15 degrees, and small affine translations. At the color level, brightness, contrast, saturation, and hue are randomly perturbed by ±30%, and the image is randomly converted to grayscale with a 10% probability. A Gaussian blur with a kernel size of 5×5 and a standard deviation between 0.1 and 2.0 is introduced to simulate slight blurring in real-world imaging. Finally, the image is converted to a tensor and normalized to obtain a standardized input. The normalization operation is represented as follows:

[0064]

[0065] Where μ is the image mean vector [0.485, 0.456, 0.406], and σ is the standard deviation vector [0.229, 0.224, 0.225].

[0066] Dual-path token generation module, such as Figure 3 As shown, it includes a region token generation unit, a spatial feature token generation unit, and a common convolutional augmentation embedding module, which are used to generate a region token sequence that encodes local semantic information and a spatial feature token sequence that encodes global context information. The region token sequence is obtained based on a pre-trained face parsing model.

[0067] The region token generation unit calls a pre-trained face parsing model to obtain a pixel-level semantic segmentation map of the input face image, providing facial semantic priors; it extracts M facial semantic regions according to predefined mapping rules (M=6 in this embodiment), including the left eyebrow region, right eyebrow region, left eye region, right eye region, nose region, and mouth region; for each semantic region, it generates its binary mask and calculates the minimum bounding rectangle, thereby cropping the region image patch from the original image and scaling it to a fixed size S×S×3 (S=64 in this embodiment); each scaled region image patch is independently input into the convolutional enhancement embedding module to obtain the corresponding region feature map. Each region feature map is input into a global average pooling layer, and its spatial dimension (S / 2)×(S / 2) is aggregated to 1, thus obtaining a D-dimensional feature vector, which is the feature vector that encodes local semantic information as a region token; all M region tokens are combined into a region token sequence.

[0068] Spatial Feature Token Generation Unit: The entire input face image is processed by the convolutional enhancement embedding module to obtain a feature map. This feature map is then sequentially input into an adaptive average pooling layer and a tensor flattening layer. The spatial size of the feature map is adjusted to (H / p)×(W / p) using adaptive average pooling, where p is the preset grid size. In this example, p=16. The pooled two-dimensional feature map is flattened in the height and width dimensions to generate a sequence of feature vectors encoding global context information, which serves as the spatial feature token sequence. N=(H / p)×(W / p) is the number of tokens. In this example, N=196.

[0069] Convolutional augmentation embedding modules, such as Figure 2 As shown, the feature extractor, which is shared by the region token generation unit and the spatial feature token generation unit, includes a three-level convolutional structure for progressive feature extraction and channel expansion of the input image.

[0070] The first-level convolutional component includes convolutional layers, batch normalization layers, and activation layers. Its convolutional layers use 7×7 convolutional kernels and perform convolutional operations with a stride of 2 and padding of 3, expanding the number of channels to one-quarter of the hidden dimension.

[0071] The second-level convolutional component includes convolutional layers, batch normalization layers, and activation layers. Its convolutional layers use 3×3 convolutional kernels and perform convolutional operations with a stride of 1 and padding of 1, expanding the number of channels to half of the hidden dimension.

[0072] The third-level convolutional component includes convolutional layers and batch normalization layers. Its convolutional layers use 3×3 convolutional kernels and perform convolutional operations with a stride of 1 and padding of 1, expanding the number of channels to the full hidden dimension.

[0073] Sequence fusion and positional coding layers, such as Figure 4 As shown, the method is used to concatenate the region token sequence and the spatial feature token sequence and add them element by element to the fused position code. After passing through a normalization layer and a random deactivation layer, the encoder input sequence is obtained.

[0074] Specifically, the fused sequence T is obtained by concatenating the region token sequence and the spatial feature token sequence along the sequence length dimension. fused This sequence integrates facial semantic priors and global spatial features. To enable the Transformer encoder to perceive the positional information of each token in the sequence, this invention constructs a fusion positional encoding E... pos It is obtained by concatenating two independent learnable location coding matrices, where the regional location coding E regionBy initializing with a truncated normal distribution and optimizing it during training using backpropagation and gradient descent, the anatomical and functional relationships between semantic regions are adaptively learned; while the spatial location encoding E spatial Based on a two-dimensional image grid, the aim is to capture the spatial relative position of tokens within the two-dimensional image plane. The fused positional encoding and the fused sequence are added element-wise to output a position-enhanced sequence. This position-enhanced sequence is then sequentially input into a normalization layer and a random deactivation layer, finally yielding the encoder input sequence T. output The output is then sent to the subsequent Transformer encoder module.

[0075] The encoder input sequence is represented as:

[0076]

[0077] LayerNorm is a layer normalization function that independently normalizes each feature vector in the sequence; Dropout is a random deactivation operation that achieves regularization by randomly discarding some feature dimensions during forward propagation. In this embodiment, the Dropout rate is set to 0.1.

[0078] The layered prior-guided Transformer encoder module is an L-layer Transformer encoder, such as... Figure 5 As shown, learnable region prior attention bias matrices are introduced into different layers of the encoder to implement facial region prior guidance. The facial region priors include facial semantic priors implemented through a pre-trained face parsing model, and facial region collaborative priors introduced through hierarchical initialization of the bias matrices.

[0079] Facial region collaborative prior is to use the aforementioned encoder input sequence T output As the initial input, let it be denoted as Each layer takes the token sequence output by the previous layer as input. Let l represent the input of the l-th layer encoder, where l = 1, 2, ..., L. Learnable region prior attention bias matrices are introduced into different layers of the Transformer encoder. The self-attention output is calculated, and after standard operations such as residual connection, layer normalization, and feedforward network, the output sequence of the layer is finally obtained. Self-attention calculation, expressed as:

[0080]

[0081] Where Q, K, and V are the query, key, and value matrices obtained by projecting the input from the l-th layer, respectively. The dimension of the key vector, and the scaling factor. Used for stable training, The function normalizes the attention scores into a probability distribution.

[0082] Learnable region prior attention bias matrix The model is constructed as (M+N)×(M+N), with its top-left M×M submatrix (corresponding to the interaction between region tokens) initialized hierarchically to achieve collaborative priors for facial regions. The geometric relationships between spatial feature tokens are sufficiently represented by spatial location encoding, so their corresponding submatrix is ​​initialized to zero. The interaction weights between regions and spatial tokens are adaptively learned by the model during training to avoid introducing inappropriate prior constraints.

[0083] The initial value of the bias matrix of the shallow encoder is based on a region spatial proximity matrix. This matrix is ​​obtained by calculating the Euclidean distance between the center points of the region bounding boxes and transforming it using a Gaussian kernel function. This spatial proximity prior is used to implement spatial proximity guidance, which is based on the spatial relative positional relationships of facial regions. Specifically, a region spatial proximity matrix is ​​constructed based on the spatial relative positional relationships of facial regions. Each element of the matrix This characterizes the spatial association strength between the i-th region and the j-th region, where i, j = 1, 2, ..., M. For each pair of regions (i, j), the Euclidean distance between the center points of their corresponding bounding boxes in the original image is calculated. The Gaussian kernel function is used to convert this distance into affinity, i.e., spatial proximity, which is expressed as:

[0084]

[0085] in, This is a hyperparameter that controls the decay rate; its value is related to the size of the input image and the facial region segmentation. Preferably, The value ranges from 0.05 to 0.15 times the height (or width) of the input image. In this embodiment, the value is 0.1 times the height of the input image, which is 22.4 pixels. Distance The closer, The larger the value, the stronger the spatial connection between regions.

[0086] The initial value of the bias matrix of the mid-layer encoder is based on a dual-attention dynamic fusion mechanism, which includes calculating the self-attention weight matrix and the prototype attention weight matrix and fusing them through a gating signal to implement statistical association prior guidance. This statistical association prior is based on inter-region association patterns statistically learned from large-scale data and encapsulated in a facial prototype feature library. Specifically: from the current layer input sequence... Extract the first M vectors to obtain the region token feature matrix. ,Will Projection as query matrix Bond matrix Calculate the self-attention weights, expressed as:

[0087]

[0088] Using the query matrix facial prototype feature library As the key matrix, the original prototype attention weights are calculated and represented as follows:

[0089]

[0090] right Dimension transformation is performed. Since each semantic region in the facial prototype feature library corresponds to multiple prototypes, the attention weights of each region token to all prototypes belonging to the same semantic region are averaged and pooled to obtain the prototype attention weights. For the region token feature matrix Global average pooling is performed along the sequence dimension to obtain a global feature vector. This vector is then passed through a linear projection layer and activated by a sigmoid function to generate a gated signal G. The two features are then fused using the gated signal G to obtain matrix A, which serves as the prior bias source for the middle layer, as follows:

[0091]

[0092] in, This indicates element-wise multiplication.

[0093] The aforementioned facial prototype feature library is constructed using the following method: On the large-scale face dataset AffectNet, the region token unit generation method in step 2 of this invention is used to extract region tokens for each semantic region of all images; K-Means clustering algorithm is performed on all token sets for each region, and the number of clusters can be selected between 5 and 10 based on the region's expression richness. In this embodiment, the number of clusters is 8; after clustering, each cluster center is a prototype, representing a typical appearance pattern of that region; the prototypes of all regions are combined into a prototype feature library. The facial prototype feature library is a pre-built static knowledge base, constructed offline based on the large-scale face dataset before model training, and does not participate in the online inference process of a single image.

[0094] The bias matrix of the deep encoder is initialized to zero, without imposing explicit priors, so that the model can autonomously learn higher-level semantic associations.

[0095] In summary, the bias matrices of each layer The initialization is represented as:

[0096]

[0097] Among them, L1 and L2 are the boundary layer numbers for dividing the shallow, medium, and deep layers (1 < L1 < L2 < L). In this example, L1 = 4, L2 = 8, and L = 12. 、 are learnable scalar parameters of dimension 1, used to automatically adjust the intensity of the prior at each stage. After being processed by all L layers of the Transformer encoder, the final output sequence .

[0098] The global feature fusion layer is used to fuse the enhanced regional token features output by the Transformer encoder module and output an image-level global feature vector. Specifically, through sequence slicing operations, the first M vectors are extracted from to obtain the enhanced regional token feature matrix for final classification. Pass it through a lightweight adaptive weighting network. The adaptive weighting network includes a weight generation unit and a weighted summation unit. The weight generation unit consists of a linearly connected projection layer and a Softmax layer. The linearly connected projection layer maps each D-dimensional feature vector in to a scalar independently. The Softmax layer normalizes all M scalar scores to generate corresponding adaptive importance weights for each regional token, where m = 1, 2,..., M; the weighted summation unit uses the weight vector output by the weight generation unit to perform weighted summation to obtain the image-level global feature vector Z, expressed as:

[0099]

[0100] Among them, is the m-th vector in the enhanced regional token feature matrix .

[0101] The classification layer fully connects the global feature vector Z output by the global feature fusion layer to 7 nodes in this layer. After Softmax regression, the probabilities that the facial expressions in the input image belong to 7 expression categories are obtained. The expression category corresponding to the maximum probability value is the output facial expression category.

[0102] Example 2: An expression recognition system based on prior guidance of facial regions, including:

[0103] The sample library module is used to obtain and store a face image data set containing expression category labels;

[0104] The facial expression recognition model construction module is used to construct an facial expression recognition model based on facial region prior guidance. The model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer; for details of the specific structure, please refer to Example 1.

[0105] The model training module is used to train an expression recognition model based on facial region prior guidance using samples from the face image dataset, and to save the trained expression recognition model.

[0106] The facial expression recognition module is used to perform facial expression recognition on newly input test images using a trained facial expression recognition model and output the expression category.

[0107] Example 3: The computer device of this example includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the expression recognition method based on facial region prior guidance in Example 1.

[0108] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0109] Those skilled in the art will clearly understand that each implementation can be achieved using software plus the necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Example 4: The computer-readable storage medium of this example stores a computer program that, when executed by a processor, implements the steps of the expression recognition method based on facial region prior guidance in Example 1.

[0111] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0112] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A facial expression recognition method based on prior knowledge guidance of facial regions, characterized in that, Includes the following steps: Step 1: Obtain a dataset of face images containing facial expression category labels; Step 2: Construct an expression recognition model based on facial region prior guidance. The model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer. The data processing layer preprocesses the input face image, including resizing, data augmentation, and normalization. The dual-path token generation module includes a region token generation unit, a spatial feature token generation unit, and a common convolutional augmentation embedding module, which generates a region token sequence that encodes local semantic information and a spatial feature token sequence that encodes global context information. The region token sequence is obtained based on a pre-trained face parsing model. The sequence fusion and location encoding layer concatenates the region token sequence and the spatial feature token sequence and adds them to the fused location encoding element by element. After passing through a normalization layer and a random deactivation layer, the encoder input sequence is obtained. The hierarchical prior-guided Transformer encoder module introduces learnable region prior attention bias matrices at different layers of the encoder to implement facial region prior guidance. The facial region priors include facial semantic priors implemented through a pre-trained face parsing model, and facial region collaborative priors implemented through hierarchical initialization of bias matrices. The global feature fusion layer extracts and enhances the output sequence of the Transformer encoder module, performs weighted fusion on the enhanced region token features, and outputs an image-level global feature vector. The classification layer classifies the global feature vector and outputs the facial expression category; Step 3: Train the expression recognition model based on facial region prior using a face image dataset, adopt the Adam method as the optimization strategy, and select cross-entropy as the loss function for gradient backpropagation. Step 4: Use the trained model to perform facial expression recognition on the newly input test image and output the expression category.

2. The facial expression recognition method based on facial region prior guidance according to claim 1, characterized in that, The region token generation unit is configured to: call a pre-trained face parsing model to obtain a pixel-level semantic segmentation map of the input face image; extract M facial semantic regions according to predefined mapping rules; for each semantic region, generate its binary mask and calculate the minimum bounding rectangle; crop out the region image patch and scale it to a fixed size; independently input each scaled region image patch into the convolutional enhancement embedding module to obtain the corresponding region feature map; input each region feature map into a global average pooling layer to generate a feature vector encoding local semantic information as a region token; and combine all M region tokens into a region token sequence. The spatial feature token generation unit is configured to: input the entire input face image into the convolutional enhancement embedding module for processing to obtain a feature map, and then input the feature map into an adaptive average pooling layer and a single flattening layer in sequence to generate a feature vector sequence that encodes global context information, which serves as the spatial feature token sequence, with the number of tokens being N. The convolutional enhancement embedding module, serving as a shared feature extractor for both the region token generation unit and the spatial feature token generation unit, is a three-level convolutional structure. The first-level convolutional component includes a convolutional layer, a batch normalization layer, and an activation layer. Its convolutional layers use 7×7 kernels with a stride of 2 and padding of 3, expanding the number of channels to one-quarter of the hidden dimension. The second-level convolutional component includes a convolutional layer, a batch normalization layer, and an activation layer. Its convolutional layers use 3×3 kernels with a stride of 1 and padding of 1, expanding the number of channels to one-half of the hidden dimension. The third-level convolutional component includes a convolutional layer and a batch normalization layer. Its convolutional layers use 3×3 kernels with a stride of 1 and padding of 1, expanding the number of channels to the full hidden dimension.

3. The facial expression recognition method based on facial region prior guidance according to claim 1, characterized in that, The fusion positional encoding in the sequence fusion and positional encoding layer is obtained by concatenating two independent learnable positional encoding matrices, including a regional positional encoding for representing the semantic region order relationship and a spatial positional encoding based on a two-dimensional image grid.

4. The facial expression recognition method based on facial region prior guidance according to claim 1, characterized in that, Learnable region prior attention bias matrix in a hierarchical prior-guided Transformer encoder module The structure is (M+N)×(M+N), where the top-left M×M submatrix is ​​initialized according to the hierarchy to achieve collaborative priors for facial regions, and the rest is initialized as a zero matrix.

5. The facial expression recognition method based on facial region prior guidance according to claim 4, characterized in that, The implementation of facial region collaborative priors through submatrix initialization based on hierarchy specifically includes: The initial value of the bias matrix of the shallow encoder is based on the region spatial proximity matrix, which is obtained by calculating the Euclidean distance between the center points of the region bounding box and transforming it using the Gaussian kernel function; The initial value of the bias matrix of the middle layer encoder is based on a dual attention dynamic fusion mechanism, specifically: extracting region token features from the input sequence of the current layer; projecting the region token features into a query matrix and a key matrix, and calculating self-attention weights; projecting the region token features into a query matrix, using the facial prototype feature library as the key matrix, and calculating prototype attention weights; generating a gating signal based on the region token features; using the gating signal to fuse the self-attention weight matrix and the prototype attention weight matrix to obtain a matrix, which serves as the prior bias source for the middle layer. The bias matrix of the deep encoder is initialized to zero, and no explicit prior is applied.

6. The facial expression recognition method based on facial region prior guidance according to claim 5, characterized in that, The facial prototype feature library was constructed using the following method: Obtain a large-scale face dataset; For each image in the large-scale face dataset, the processing flow of the region token generation unit in step 2 is used to extract the region token of each semantic region; For each semantic region, K-Means clustering is performed on all region tokens extracted from the large-scale face dataset, with the number of clusters between 5 and 10, and each cluster center is used as a prototype; Combine the prototypes of all regions to form a facial prototype feature library.

7. The facial expression recognition method based on facial region prior guidance according to claim 1, characterized in that, The global feature fusion layer is configured as follows: The first M vectors are extracted from the output sequence of the hierarchical prior-guided Transformer encoder module to obtain the enhanced region token feature matrix; The enhanced region token feature matrix is ​​input into an adaptive weighted network to generate importance weights for each region token, and then the weighted sums are performed to obtain an image-level global feature vector.

8. The facial expression recognition method based on facial region prior guidance according to claim 7, characterized in that, An adaptive weighted network includes a weight generation unit and a weighted summation unit; The weight generation unit consists of a linear projection layer and a softmax layer connected in sequence. The enhanced region token feature matrix is ​​input into the linear projection layer and mapped to a scalar. Then, it is normalized by the softmax layer to generate a corresponding adaptive importance weight for each region token. The weighted summation unit is configured to use a normalized weight vector to perform a weighted summation on the region token feature matrix, and output an image-level global feature vector.

9. A facial expression recognition system based on prior knowledge guidance of facial regions, characterized in that, include: The sample library module is used to acquire and store a dataset of face images containing facial expression category labels; The facial expression recognition model building module is used to build an facial expression recognition model based on facial region prior guidance. The model includes a data processing layer, a dual-path token generation module, a sequence fusion and position encoding layer, a hierarchical prior-guided Transformer encoder module, a global feature fusion layer, and a classification layer. The model training module is used to train an expression recognition model based on facial region prior guidance using samples from the face image dataset, and save the trained expression recognition model. The facial expression recognition module is used to perform facial expression recognition on newly input test images using a trained facial expression recognition model and output the expression category.

10. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the facial expression recognition method based on facial region prior guidance as described in any one of claims 1-8.

Images