An expression recognition transformer compression method and system based on key feature region protection

Abstract

The application discloses an expression recognition Transformer compression method and system based on key feature area protection, relates to artificial intelligence, and aims at the problem that efficiency and precision are difficult to balance in the prior art. The method comprises the following steps: preprocessing an image to obtain a standardized input image; extracting face key point coordinates in the input image; obtaining a key token index list that needs to be protected; dividing and converting the image into a token sequence; performing a token merging operation with protection; updating the key token index list to adapt to the length of the new token sequence; and processing the final token features through a classification head to output a face expression recognition result. The method has the advantages that the efficient token merging technology and the accurate face key point detection technology are combined, the key feature area in the image is adaptively recognized and protected according to the characteristics of the face expression recognition task, the problem that the recognition precision is significantly reduced due to indiscriminate merging of all tokens is avoided, and the complexity and inference time are significantly reduced on the premise of ensuring the expression recognition accuracy.

Description

Technical Field

[0001] This invention relates to artificial intelligence, and more particularly to a Transformer compression method and system for facial expression recognition based on the protection of key feature regions. Background Technology

[0002] With the rapid development of artificial intelligence technology, facial expression recognition, as a core technology in affective computing and human-computer interaction, has wide-ranging application needs in scenarios such as intelligent customer service, mental health monitoring, and driver fatigue detection. Users' demands for real-time performance and accuracy are increasing. Therefore, how to reduce computational complexity while ensuring recognition accuracy to achieve efficient facial expression recognition is a topic worthy of serious research.

[0003] Facial expression recognition refers to the automatic identification of emotional states, such as happiness, sadness, and anger, in facial images using computer vision technology. Traditional facial expression recognition methods are mainly based on hand-designed feature extractors (such as LBP and HOG) combined with classifiers such as support vector machines or random forests. These methods rely on manually designed features and can quickly complete recognition tasks in specific scenarios. However, due to a lack of deep semantic understanding of subtle changes in facial expressions and insufficient robustness to complex conditions such as different lighting and poses, the recognition accuracy is often unsatisfactory.

[0004] In recent years, with the development of deep learning technology, facial expression recognition methods based on visual Transformers have become the mainstream approach. However, visual Transformer models typically contain a large number of tokens; for example, a 224×224 image might be divided into 14×14=196 tokens plus one classification token, totaling 197 tokens. During the computation of the multi-layer self-attention mechanism, the computational complexity increases quadratically with the number of tokens, making it difficult to deploy widely on mobile devices or in real-time applications.

[0005] To address the aforementioned issues, existing research proposes using Token Merging (ToMe) to reduce the computational complexity of visual Transformers. This approach reduces the number of tokens by merging similar tokens within each Transformer layer, thereby decreasing the computational load of subsequent layers. While this method effectively reduces computation and improves inference speed, it fails to consider the specific characteristics of facial expression recognition: key facial regions such as eyes, eyebrows, and lips contain crucial features expressing emotions, while background regions contribute less to the recognition results. Existing token merging methods treat all tokens equally, potentially merging key feature regions of facial expression images, leading to the loss of important information. Directly applying this to facial expression recognition tasks often significantly reduces accuracy. Therefore, how to finely protect tokens representing key facial regions during token merging, achieving a balance between computational efficiency and recognition accuracy, is a pressing technical challenge. Summary of the Invention

[0006] The purpose of this invention is to provide a Transformer compression method and system for facial expression recognition based on the protection of key feature regions, so as to solve the problems existing in the prior art.

[0007] The facial expression recognition Transformer compression method based on key feature region protection described in this invention includes the following steps:

[0008] S100. Obtain the facial expression image to be recognized, and preprocess the image to obtain a standardized input image;

[0009] S200. Use a facial landmark detection algorithm to extract the coordinates of facial landmarks in the input image;

[0010] S300. Perform index mapping based on the coordinates of facial key points to obtain a list of key token indexes that need to be protected;

[0011] S400. Input the input image into the visual Transformer, segment the image and convert it into a token sequence;

[0012] S500. In each block of the visual Transformer, perform a protected token merging operation based on the list of key token indices;

[0013] S600. After the tokens are merged, update the list of key token indices to accommodate the new token sequence length;

[0014] S700. The final token features are processed by the classification head, and the facial expression recognition result is output.

[0015] The input image size is The token size is The image is divided into With one token and one category token, the total number of tokens is [number missing]. .

[0016] The facial expression recognition Transformer compression system based on key feature region protection described in this invention includes:

[0017] The facial landmark extraction module is used to receive the input facial image and extract the coordinate information of key parts using a facial landmark detection algorithm;

[0018] The key point mapping module is used to convert the coordinates of facial key points into token indices in the visual Transformer, and generate a list of key token indices that need to be protected.

[0019] The image encoding module is used to divide the input image into fixed-size tokens and convert them into a token sequence, add position encoding, and then input it into the Transformer encoder;

[0020] The protected token merging module is used to perform selective token merging in each Transformer Block based on a list of key token indices, protecting key feature tokens from being merged and only merging tokens that are not in key areas;

[0021] The key point index update module is used to update the key token index list according to the index mapping relationship of the merge operation after each token merge, so as to ensure the consistency of cross-layer protection.

[0022] The facial expression classification module is used to extract classification features from the final token features and outputs the probability distribution of facial expression categories through a fully connected layer.

[0023] When the system is running, the aforementioned expression recognition Transformer compression method based on key point protection is executed.

[0024] The Transformer compression method and system for facial expression recognition based on key feature region protection described in this invention has the advantage of combining efficient token merging technology and accurate facial landmark detection technology. It adaptively identifies and protects key feature regions in the image according to the characteristics of facial expression recognition tasks. The token merging strategy is used for non-critical background regions to reduce the amount of computation, while the tokens of key regions such as eyes, eyebrows, and lips containing important expression information are protected to avoid information loss. This avoids the problem of significantly reducing recognition accuracy due to indiscriminate merging of all tokens. While ensuring the accuracy of expression recognition, it significantly reduces computational complexity and inference time, making it possible to deploy the model in mobile devices and real-time application scenarios. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the method described in this invention.

[0026] Figure 2 This is a flowchart illustrating step S200 of the method described in this invention.

[0027] Figure 3 This is a flowchart illustrating step S500 of the method described in this invention.

[0028] Figure 4 This is a flowchart illustrating step S600 of the method described in this invention.

[0029] Figure 5 This is a schematic diagram of the structure of a Transformer compression system for facial expression recognition based on the protection of key feature regions, as described in this invention. Detailed Implementation

[0030] The facial expression recognition Transformer compression method based on key feature region protection described in this invention is as follows: Figures 1 to 4 As shown, it includes the following steps:

[0031] S100. Obtain the face image to be recognized, and preprocess the image to obtain a standardized input image. The input image size is... The token size is The image is then divided into With one token and one category token, the total number of tokens is [number missing]. In this embodiment, , .

[0032] S200. Extract the coordinates of facial landmarks in the input image using a facial landmark detection algorithm, specifically including the following sub-steps:

[0033] S201. Use a face detection algorithm to locate the face region in the input image.

[0034] S202. Apply a facial landmark detection model to the located face region to extract the coordinates of key points including eyes, eyebrows, nose, lips, etc.

[0035] S203. Filter the extracted facial key points, retain key points that contribute highly to expression recognition, and remove key points that contribute little to expression recognition.

[0036] In this invention, facial landmark detection is implemented based on the DLIB framework, which can extract multiple key regions including the chin, left eyebrow, right eyebrow, nose, left eye, right eye, upper lip, and lower lip. The landmark coordinates are expressed in pixel coordinates. The form of representation, in which The x-axis is... The vertical axis is denoted as y.

[0037] Optionally, to focus on the core areas of facial expression recognition, key points such as the chin and nose, which contribute less to facial expression, are removed, while key points of the eyebrows, eyes, and mouth are retained. The filtered set of key points is defined as follows:

[0038]

[0039] in, This represents the total number of key points.

[0040] S300. Calculate the corresponding token index based on the facial key point coordinates to obtain a list of key token indices that need to be protected. This includes the following sub-steps:

[0041] S301. Obtain image dimensions , and token size parameter.

[0042] S302. Based on the x and y coordinates of the keypoint and the token size, calculate the row and column index of the token to which the keypoint belongs:

[0043]

[0044]

[0045] in and Corresponding to the first The horizontal and vertical coordinates of the key points.

[0046] S303. Calculate the global index in the token sequence based on the token row index and column index:

[0047] .

[0048] S304. Deduplicate and sort the token indices corresponding to all key points to obtain a list of key token indices:

[0049]

[0050] in Indicates a sorting operation. This indicates a deduplication operation.

[0051] Because the visual Transformer adds a category token at the beginning of the token sequence, in actual use it is necessary to... Increment each index by 1 to obtain the final list of key token indices. ,in This represents the number of key tokens after deduplication.

[0052] S400. Input the input image into the visual Transformer model, segment the image and convert it into a token sequence.

[0053] S500. In each block of the visual Transformer, a protected token merging operation is performed based on the list of key token indices, specifically including the following sub-steps:

[0054] S501. Calculate the similarity metric matrix between tokens in the self-attention module of the current Transformer Block. ,in Represents the set of real numbers. For batch size, This represents the current number of tokens. For feature dimensions.

[0055] S502. Divide the token sequence into source group a and target group according to parity. Construct a bipartite graph matching structure:

[0056]

[0057] in .

[0058] S503. Source Group and target group After L2 normalization in the last dimension, the cosine similarity score matrix between the source group tokens and the target group tokens is calculated:

[0059]

[0060]

[0061]

[0062] in Indicates source group L2 norm, Indicate target group L2 norm, .

[0063] S504. Based on the key token index list To prevent a key token from being selected as a merging candidate, the similarity score at the corresponding location is set to negative infinity. Specifically, for an index in the key token index list, its local index in source group a is determined: for the global index... ,like If it is even, then it is in the source group. Local indexes in ,make .

[0064] S505. Similarity score matrix Sort the samples and select the ones with the highest scores. Merge individual token pairs:

[0065]

[0066]

[0067] in Indicates along The last dimension takes the maximum value.

[0068] Finally, we obtain the unmerged token index. and the source token index to be merged and target token index ,in The operation is equivalent to: .

[0069] S506. Use a weighted average method to fuse the features of the source tokens to be merged into the corresponding target tokens, generating a merged token sequence. Let the input token sequence be... Divide them according to their odd or even positions:

[0070]

[0071] Extract the unmerged source token: And the source token to be merged: Merge the source tokens to be merged into the target token:

[0072]

[0073] The final merged token sequence is ,in , This indicates a splicing operation.

[0074] S600. After token merging, update the key token index list to accommodate the new token sequence length, specifically including the following sub-steps:

[0075] S601. Get the length of the token sequence before the merge operation. Unmerged token index list List of source token indexes to be merged and target token index list .

[0076] S602. Construct a mapping vector from the old token index to the new token index. All values ​​are initialized to .

[0077] S603. For unmerged source tokens, assign new indices sequentially according to their positions in the new sequence. Even-numbered positions in the source group are... The old index of the unmerged source token is:

[0078]

[0079] Assign a new index to it .

[0080] S604. For all odd-position tokens in the target group, their position in the new sequence immediately follows the unmerged source token. Odd positions are... ,but:

[0081] .

[0082] S605. Traverse the list of key token indices Each index in Find its corresponding index in the new sequence based on the mapping vector: .

[0083] S606. Update the key token index list as follows: .

[0084] S700. The final token features are processed by the classification head, and the facial expression recognition result is output.

[0085] The facial expression recognition Transformer compression system based on key feature region protection described in this invention is as follows: Figure 5 As shown, it includes:

[0086] The facial landmark extraction module is used to receive the input facial image and extract the coordinate information of key parts such as eyes, eyebrows, and lips using a facial landmark detection algorithm.

[0087] The key point mapping module is used to convert the coordinates of facial key points into token indices in the visual Transformer model and generate a list of key token indices that need to be protected.

[0088] The image encoding module is used to divide the input image into fixed-size tokens and convert them into a token sequence, add position encoding, and then input it into the Transformer encoder;

[0089] The protected token merging module is used to perform selective token merging in each Transformer Block based on a list of key token indices, protecting key feature tokens from being merged and only merging tokens that are not in key areas;

[0090] The key point index update module is used to update the key token index list according to the index mapping relationship of the merge operation after each token merge, so as to ensure the consistency of cross-layer protection.

[0091] The facial expression classification module is used to extract classification features from the final token features and outputs the probability distribution of facial expression categories through a fully connected layer.

[0092] Each module can be a software module or a hardware module. When the system is running, it executes the Transformer compression method for facial expression recognition based on key point protection described in this invention. Specifically, the facial key point extraction module executes steps S201-S203, the key point mapping module executes steps S301-S304, the image encoding module executes step S400, the protected token merging module executes steps S501-S506, the key point index update module executes steps S601-S606, and the expression classification module executes step S700.

[0093] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A Transformer compression method for facial expression recognition based on key feature region protection, characterized in that, Includes the following steps: S100. Obtain the facial expression image to be recognized, and preprocess the image to obtain a standardized input image; S200. Use a facial landmark detection algorithm to extract the coordinates of facial landmarks in the input image; S300. Perform index mapping based on the coordinates of facial key points to obtain a list of key token indexes that need to be protected; S400. Input the input image into the visual Transformer, segment the image and convert it into a token sequence; S500. In each block of the visual Transformer, perform a protected token merging operation based on the list of key token indices; S600. After the tokens are merged, update the list of key token indices to accommodate the new token sequence length; S700. The final token features are processed by the classification head, and the facial expression recognition result is output. The input image size is The token size is The image is divided into With one token and one category token, the total number of tokens is [number missing]. .

2. The expression recognition Transformer compression method based on key feature region protection according to claim 1, characterized in that, Step S200 specifically includes the following sub-steps: S201. Use a face detection algorithm to locate the face region in the input image; S202. Apply the facial landmark detection model to the located face region to extract the coordinates of key points in specific areas; S203. Filter the extracted facial key points, retain key points whose contribution to expression recognition is higher than the threshold, and remove key points whose contribution to expression recognition is lower than the threshold.

3. The expression recognition Transformer compression method based on key feature region protection according to claim 2, characterized in that, The specific body parts include the chin, left eyebrow, right eyebrow, nose, left eye, right eye, upper lip, and lower lip; key point coordinates are in pixel coordinates. The form of representation, in which The x-axis is... The vertical axis is denoted as y.

4. The expression recognition Transformer compression method based on key feature region protection according to claim 3, characterized in that, When focusing on the core areas of facial expression recognition, key points corresponding to the chin and nose are removed, while key points corresponding to the eyebrows, eyes, and mouth are retained; the resulting set of key points is... ,in This represents the total number of key points.

5. The expression recognition Transformer compression method based on key feature region protection according to claim 4, characterized in that, Step S300 specifically includes the following sub-steps: S301. Obtain image dimensions , and token size parameter; S302. Based on the x and y coordinates of the keypoint and the token size, calculate the row and column index of the token to which the keypoint belongs: in and Corresponding to the first The horizontal and vertical coordinates of the key points; S303. Calculate the global index in the token sequence based on the token row index and column index: S304. Deduplicate and sort the token indices corresponding to all key points to obtain a list of key token indices: in Indicates a sorting operation. This indicates a deduplication operation.

6. The expression recognition Transformer compression method based on key feature region protection according to claim 5, characterized in that, Step S500 specifically includes the following sub-steps: S501. Calculate the similarity metric matrix between tokens in the self-attention module of the current Transformer Block; S502. Divide the token sequence into source groups according to parity. and target group Construct a bipartite graph matching structure; S503. Source Group and target group After L2 normalization in the last dimension, the cosine similarity score matrix between the source group token and the target group token is calculated. S504. Based on the key token index list Set the similarity score of the key token's corresponding position to negative infinity; for the global index ,like If it is even, let ; S505. Similarity score matrix Sort the samples and select the ones with the highest scores. Merge individual token pairs; S506. Use the weighted average method to fuse the features of the source tokens to be merged into the corresponding target tokens to generate the merged token sequence.

7. The expression recognition Transformer compression method based on key feature region protection according to claim 6, characterized in that, Step S600 includes the following sub-steps: S601. Get the length of the token sequence before the merge operation. Unmerged token index list List of source token indexes to be merged and target token index list ; S602. Construct a mapping vector from the old token index to the new token index. All values ​​are initialized to ; S603. For unmerged source tokens, assign new indices sequentially according to their positions in the new sequence; even-numbered positions in the source group are... The old index of the unmerged source token is: Assign a new index to it ; S604. For all odd-position tokens in the target group, their position in the new sequence immediately follows the unmerged source token; odd positions are... ,but: ； S605. Traverse the list of key token indices Each index in Find its corresponding index in the new sequence based on the mapping vector: ; S606. Update the key token index list as follows: ,in This represents the number of key tokens after deduplication.

8. A Transformer compression system for facial expression recognition based on key feature region protection, characterized in that, include: The facial landmark extraction module is used to receive the input facial image and extract the coordinate information of key parts using a facial landmark detection algorithm; The key point mapping module is used to convert the coordinates of facial key points into token indices in the visual Transformer, and generate a list of key token indices that need to be protected. The image encoding module is used to divide the input image into fixed-size tokens and convert them into a token sequence, add position encoding, and then input it into the Transformer encoder; The protected token merging module is used to perform selective token merging in each Transformer Block based on a list of key token indices, protecting key feature tokens from being merged and only merging tokens that are not in key areas; The key point index update module is used to update the key token index list according to the index mapping relationship of the merge operation after each token merge, so as to ensure the consistency of cross-layer protection. The facial expression classification module is used to extract classification features from the final token features and outputs the probability distribution of facial expression categories through a fully connected layer. When the system is running, it executes the expression recognition Transformer compression method based on key feature region protection as described in any one of claims 1-7.

Images