Transformer face attribute recognition method and system based on mask attribute modeling

Abstract

The application relates to a mask attribute modeling-based Transformer face attribute recognition method and system, which comprises the following steps: 1) obtaining a face attribute recognition dataset and a face attribute label combination; 2) constructing a mask attribute modeling-based face attribute recognition network model, which comprises a dynamic relationship perception coding module, a Transformer encoder, a language encoder and a language decoder, and cooperatively performing FAR and MFAM tasks; the language encoder obtains face attribute relationship coding from a language mode; the dynamic relationship perception coding module fuses information of the language and visual modes to obtain multi-modal fusion features; the Transformer encoder excavates long-distance dependence between the multi-modal fusion features to obtain attention perception multi-modal features; the language decoder performs the MFAM task; and 3) applying the trained face attribute recognition network model to a face attribute recognition task. The method and system are beneficial to obtaining more stable, robust and accurate face attribute recognition results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a Transformer face attribute recognition method and system based on mask attribute modeling. Background Technology

[0002] Facial attribute recognition aims to predict various attributes in a given facial image, and its applications are wide-ranging, including face verification, face recognition, image generation, and image retrieval. However, due to the susceptibility of visual images to factors such as lighting variations, low contrast, noise, and poor visibility, facial attribute recognition has always been a challenging task. Existing facial attribute recognition methods generally employ convolutional neural networks (CNNs) to learn the deep features of facial images, and improve the accuracy of facial attribute recognition by having the model learn features related to facial attributes or by incorporating relationships between facial attributes.

[0003] Currently, mainstream facial attribute recognition methods can be divided into two categories: auxiliary task-based facial attribute recognition methods and multi-task-based facial attribute recognition methods. Auxiliary task-based facial attribute recognition methods aim to help the backbone network learn effective facial attribute features through multiple face-related auxiliary tasks, thereby improving facial attribute recognition. Ni Zhuang et al. (Ni Zhuang, Yan Yan, Si Chen, and Hanzi Wang. 2018. Multi-task learning of cascaded CNN for facial attribute classification. In Proceedings of the IEEE International Conference on Pattern Recognition. 2069–2074.) proposed cascaded face detection and facial landmark detection to assist facial attribute recognition. They simultaneously trained three related face analysis tasks within a unified framework, while multi-task learning was performed by inputting a single attribute in the facial attribute recognition process. Ying Shu et al. (Ying Shu, Yan Yan, Si Chen, JingHao Xue, Chunhua Shen, and Hanzi Wang. 2021. Learning spatial-semantic relationship for facial attribute recognition with limited labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11916–11925.) designed a task including block rotation (PRT), block segmentation (PST), and block classification (PCT) to learn spatial-semantic relationships from large-scale unlabeled face data and apply them to facial attribute recognition.

[0004] For multi-task-based facial attribute recognition methods, the aim is to group attributes based on their correlations and assign each attribute group to a learning task. Emily M. Hand et al. (Emily M. Hand and Rama Chellappa. 2017. Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 4068–4074.) divided facial attributes into 9 groups based on different facial attribute locations and used a multi-task deep CNN with auxiliary networks to discover hierarchical attribute relationships. Longbiao Mao et al. (Longbiao Mao, YanYan, JingHao Xue, and Hanzi Wang. 2022. Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Affective Computing 13,2(2022),818–828.) artificially divided attributes into two groups based on the learning complexity of different attributes and designed two different network architectures to construct the attribute relationships within the two groups. Xiaohui Fanhe et al. (Xiaohui Fanhe, Jie Guo, Zheng Huang, Weidong Qiu, and Yuele Zhang. 2019. Multi-task learning with knowledge transfer for facial attribute classification. In Proceedings of the IEEE International Conference on Industrial Technology. 877–882) considered the different degrees of correlation between facial attributes, used spectral clustering to cluster facial attributes, and designed a multi-task framework based on this.

[0005] However, these facial attribute relationships are obtained either through fixed clustering algorithms or through manual grouping. The relationships between facial attributes obtained in these ways cannot be well modeled or fully utilized. Furthermore, multi-task learning frameworks designed in this way cannot fully express the complex relationships between facial attributes, resulting in insufficient learning of these relationships by the model. Based on a Transformer encoder, this invention combines information from the language modality and the visual modality, enabling the model to obtain and fully utilize the corresponding local facial attribute relationships from the input image, thereby optimizing the model's extraction of image features. Simultaneously, to address the problem of the model's inability to fully learn and model facial attribute relationships, this invention combines the characteristics of the "visual-language" model and designs a masking task as an auxiliary task, enhancing the model's understanding of global facial attribute relationships. Furthermore, this invention is adaptable to various Transformer encoders. Summary of the Invention

[0006] The purpose of this invention is to provide a Transformer face attribute recognition method and system based on mask attribute modeling, which is beneficial for obtaining more stable, robust and accurate face attribute recognition results.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a Transformer face attribute recognition method based on mask attribute modeling, comprising the following steps:

[0008] 1) Obtain a face attribute recognition dataset and a face attribute label combination, wherein the face attribute recognition dataset contains face images and their corresponding face attribute labels; wherein the face attribute label consists of one or more face attribute words;

[0009] 2) Construct a face attribute recognition network model based on mask attribute modeling. This model comprises multiple components: a dynamic relation-aware encoding module, a Transformer encoder, a language encoder, and a language decoder. These components collaboratively perform face attribute recognition and face attribute mask modeling tasks. The language encoder encodes information from the language modality to obtain face attribute relation encodings from the language modality. The dynamic relation-aware encoding module fuses information from the language and visual modalities to obtain multimodal fusion features. The Transformer encoder further mines the long-distance dependencies between the multimodal fusion features to obtain attention-aware multimodal features. The language decoder performs face attribute mask modeling on the attention-aware multimodal features to enhance the model's understanding of global attribute relationships. The constructed face attribute recognition network model is trained using a face attribute recognition dataset.

[0010] 3) The trained face attribute recognition network model is used for face attribute recognition tasks.

[0011] Further, in step 1), a face attribute recognition dataset is obtained from public datasets including CelebA and LFWA, wherein the face attribute label combination includes 40 face attribute labels.

[0012] Furthermore, in step 2), the implementation method of the facial attribute recognition network model is as follows:

[0013] A) Load the pre-trained language encoder weights to initialize the language encoder, and use the language encoder to extract the face attribute relationship code from the face attribute label combination;

[0014] B) Encode the face images in the face attribute recognition dataset to obtain face image features;

[0015] C) The facial attribute relationship encoding and facial image features are fed into the dynamic relationship-aware encoding module, i.e., the DRAE module. The dynamic relationship-aware encoding module first uses an information gating mechanism to dynamically adjust the weight of the language modality information based on the input facial image features, so that it matches the visual modality information in the facial image, reducing the difference between the visual modality and the language modality. Then, the facial attribute relationship encoding and facial image features are fused in the channel dimension. Through the gating mechanism and the guided attention mechanism, the facial attribute relationship in the language modality is effectively integrated into the facial image features, thereby obtaining a multimodal fusion feature that includes facial image and facial attribute relationship.

[0016] D) The multimodal fusion features are input into the Transformer encoder after positional encoding. The Transformer encoder uses an attention mechanism to mine the long-distance dependencies between the multimodal fusion features to obtain attention-aware multimodal features, thereby enhancing the understanding of complex image features and the interrelationships between attributes to achieve the face attribute recognition task, i.e., the FAR task. The attention-aware multimodal features and the face attribute labels of the random mask are input into the language decoder to perform the face attribute mask modeling task, i.e., the MFAM task, to enhance the learning of the global relationships between face attributes.

[0017] E) During training, the face attribute recognition network model simultaneously performs the FAR and MFAM tasks, continuously optimizing the parameters of the Transformer encoder. For face attribute recognition, the FAR task uses the sigmoid cross-entropy function as the loss function and treats the FAR task as multiple binary classification tasks. The MFAM task uses the negative log-likelihood function as the loss function to predict the face attribute labels of randomly masked faces. Both tasks use the SGD optimizer to minimize the error and jointly affect the update of the Transformer encoder parameters during training.

[0018] Further, in step A), the language encoder is a BERT model pre-trained with a Wikipedia corpus dictionary. The language encoder encodes the face attribute labels in the face attribute label combination to obtain the face attribute relationship code.

[0019] Further, in step B), the original image of size 224*224 is first segmented into image blocks of size adapted to the input of the Transformer encoder, and then the image blocks are processed through a linear layer and a classification head and position encoding are added to obtain the face image features.

[0020] Furthermore, in step C), the specific implementation method of the dynamic relationship-aware encoding module is as follows:

[0021] First, the DRAE module accepts face attribute relation encoding. and facial image features As input, N is the number of facial attribute words in the facial attribute label combination, and C, H, and W are the number of channels, height, and width of the image, respectively; the facial image features are expanded into Then, the relationship between the facial attributes and the encoding F is... l Interact to adjust the weights of each facial attribute in the facial attribute relation encoding based on facial image features; the weighted facial attribute relation encoding F l w Represented as:

[0022]

[0023] Where d is the encoding dimension of the Transformer encoder; It is the scaling factor; F l w With F l They have the same dimension N×C;

[0024] To extract the facial attribute relationships in the current image from the weighted facial attribute relationship encoding, the connections between facial attribute words are modeled; the facial attribute relationship Relation is represented as:

[0025]

[0026] Wherein, Conv(·) represents the convolution operation, which is used to extract the relationship between facial attribute words and model the relationship between facial attributes;

[0027] To uniformly integrate the extracted facial attribute relationships into the facial image features, the extracted facial attribute relationships are reshaped to match the dimensions of the facial image features, and the two are connected along the channel dimension; secondly, the connected feature F cat Convolutional operations are performed to restore the original number of channels; then, the features are weighted by an information gate and added to the original face image features to generate a multimodal fusion feature F. mm The information gate adjusts the contribution of facial attribute relationships to multimodal fusion features during the summation process to adapt to different levels of image quality; the concatenated feature F cat and the obtained multimodal fusion features F mm They are represented as follows:

[0028]

[0029] Here, ++ represents the concatenation operation; tanh(·) represents the information gate, which uses a tanh function with a trainable parameter μ to adjust the proportion of facial attribute relationships in multimodal features.

[0030] Further, in step D), face attribute masking modeling is used as an auxiliary task. The specific implementation method of face attribute masking modeling is as follows: First, the face attribute labels corresponding to the face image are randomly sorted, and the face attribute words are randomly masked according to a certain proportion to obtain the face attribute labels of the random mask. Then, the language encoder is used to encode them to obtain the masked language features. The attention-aware multimodal features output by the Transformer encoder are the sum of face image features, face attribute relationship encoding, and position encoding. The attention-aware multimodal features output by the Transformer encoder are input into the language decoder, and the language decoder is used to assist in predicting the face attribute words at the mask position. The language decoder used for face attribute masking modeling has multiple layers. During the mask prediction process, the attention-aware multimodal features output by the Transformer encoder are input into the language decoder of each layer. Among them, the language decoder of the l-th layer uses a cross-attention mechanism to process the attention-aware multimodal features. and the language features masked at layer l-1 Preliminary fusion is performed to obtain preliminary masked language features. Further analysis of masked language features Obtaining masked language features of layer l using a self-attention mechanism and They are represented as follows:

[0031]

[0032] Where, N mask Indicates the length of the masked language features; when l = 0, The masked language features of the initial input;

[0033] Throughout the face attribute masking modeling process, the Transformer encoder continuously refines the extraction of face image features and the modeling of face attribute relationships. To maintain compatibility with different Transformer encoders, the speech encoder and speech decoder will retain the original input dimensions of the Transformer encoder during the encoding and decoding processes.

[0034] Furthermore, in step E), the specific method for constructing the loss functions for the FAR and MFAM tasks is as follows: the parameters of the Transformer encoder will be updated during training due to the influence of the FAR and MFAM tasks; considering that the FAR task is a binary classification problem, the sigmoid cross-entropy function is used. As a loss function, it is expressed as:

[0035]

[0036] in, and Let N represent the predicted value and the true label of the face attribute of the j-th person in the i-th image, respectively; σ(·) represents the sigmoid function; N s A represents the number of training samples; A represents the number of facial attributes.

[0037] For the MFAM task, the negative log-likelihood function is used. As a loss function, it is expressed as:

[0038]

[0039] Where M represents the number of face attribute tags that are masked; m i |V| represents the predicted value of the attribute label of the masked face in the MFAM task; |V| represents the number of words in the Wikipedia corpus dictionary used by the language encoder.

[0040] During training, both loss functions mentioned above jointly affect the update of the Transformer encoder parameters; therefore, the total loss function is expressed as:

[0041]

[0042] The present invention also provides a Transformer face attribute recognition system based on mask attribute modeling, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0043] Compared with the prior art, the present invention has the following beneficial effects: The method and system effectively explore the interaction between the facial attribute relationship of language modality and image features. By using this framework to more effectively guide the model to extract features from facial images, more robust features that are more relevant to facial attributes can be obtained, effectively solving problems such as lighting, occlusion, and low pixel count, and improving the accuracy of facial attribute recognition. Attached Figure Description

[0044] Figure 1 This is an architecture diagram of a face attribute recognition network model based on mask attribute modeling in an embodiment of the present invention;

[0045] Figure 2 This is a structural diagram of the dynamic relationship-aware encoding module in an embodiment of the present invention;

[0046] Figure 3 This is a structural diagram of the language decoder in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0050] This embodiment provides a Transformer-based face attribute recognition method based on mask attribute modeling, including the following steps:

[0051] 1) Obtain a face attribute recognition dataset and a face attribute label combination, wherein the face attribute recognition dataset contains face images and their corresponding face attribute labels; wherein the face attribute label consists of one or more face attribute words;

[0052] 2) Construct a face attribute recognition network model based on mask attribute modeling. This model comprises multiple components, including a dynamic relation-aware encoding module, a Transformer encoder, a language encoder, and a language decoder. These components collaboratively perform face attribute recognition and face attribute mask modeling tasks. The language encoder encodes information from the language modality to obtain face attribute relation encodings from the language modality. The dynamic relation-aware encoding module fuses information from the language and visual modalities to obtain multimodal fusion features. The Transformer encoder further mines the long-distance dependencies between the multimodal fusion features to obtain attention-aware multimodal features. The language decoder performs face attribute mask modeling on the attention-aware multimodal features to enhance the model's understanding of global attribute relationships. The constructed face attribute recognition network model is trained using a face attribute recognition dataset.

[0053] 3) The trained face attribute recognition network model is used for face attribute recognition tasks.

[0054] In this embodiment, a face attribute recognition dataset is obtained from publicly available datasets including CelebA and LFWA. The face attribute label combination includes 40 face attribute labels.

[0055] Figure 1 This is an architecture diagram of the face attribute recognition network model based on mask attribute modeling in this embodiment. For example... Figure 1 As shown, the specific implementation method of the face attribute recognition network model is as follows.

[0056] A) Load the pre-trained language encoder weights from the Wikipedia corpus to initialize the language encoder, and use the language encoder to extract the face attribute relationship code from the face attribute label combination.

[0057] In this embodiment, the language encoder is a BERT model pre-trained with a Wikipedia corpus dictionary. The language encoder encodes the face attribute labels in the face attribute label combination to obtain the face attribute relationship code.

[0058] B) Encode the face images in the face attribute recognition dataset to obtain face image features.

[0059] In this embodiment, the original image with a size of 224*224 is first segmented into image blocks of a size that fits the input of the Transformer encoder. Then, the image blocks are processed through a linear layer and a classification head and position encoding are added to obtain the face image features.

[0060] C) The facial attribute relationship encoding and facial image features are fed together into the Dynamic Relationship-Aware Encoding (DRAE) module. The DRAE module first uses an information gating mechanism to dynamically adjust the weight of the language modality information based on the input facial image features, so that it matches the visual modality information in the facial image, reducing the difference between the visual modality and the language modality. Then, the facial attribute relationship encoding and facial image features are fused in the channel dimension. Through gating and guided attention mechanisms, the facial attribute relationships in the language modality are effectively integrated into the facial image features, thereby obtaining multimodal fusion features that include facial images and facial attribute relationships.

[0061] Figure 2 This is a structural diagram of the Dynamic Relationship Aware Coding (DRAE) module in this embodiment. For example... Figure 2 As shown, the implementation method of the dynamic relationship-aware encoding module is as follows.

[0062] First, the DRAE module accepts face attribute relation encoding. and facial image features As input, N is the number of facial attribute words in the facial attribute label combination, and C, H, and W are the number of image channels, height, and width, respectively; in the DRAE module, the facial image features are expanded into Then, the relationship between the facial attributes and the encoding F is... l Interact to adjust the weights of each facial attribute in the facial attribute relation encoding based on facial image features; the weighted facial attribute relation encoding F l w It can be represented as:

[0063]

[0064] Where d is the encoding dimension of the Transformer encoder; It is the scaling factor; F lw With F l They have the same dimension N×C.

[0065] To extract the facial attribute relationships in the current image from the weighted facial attribute relationship encoding, the connections between facial attribute words are modeled; the facial attribute relationship Relation can be represented as:

[0066]

[0067] Here, Conv(·) represents the convolution operation, which is used to extract the relationship between facial attribute words and model the relationship between facial attributes.

[0068] To uniformly integrate the extracted facial attribute relationships into the facial image features, the extracted facial attribute relationships are reshaped to match the dimensions of the facial image features, and the two are connected along the channel dimension; secondly, the connected feature F cat Convolutional operations are performed to restore the original number of channels; then, the data is weighted by an information gate and added to the original face image features to generate a multimodal fusion feature F. mm Furthermore, the information gate adjusts the contribution of facial attribute relationships to multimodal fusion features during the summation process to adapt to different levels of image quality; therefore, the concatenated feature F cat and the subsequent obtained multimodal fusion features F mm They can be represented as:

[0069]

[0070] Here, ++ represents the concatenation operation; tanh(·) represents the information gate, which uses a tanh function with a trainable parameter μ to adjust the proportion of facial attribute relationships in multimodal features.

[0071] D) The multimodal fusion features are input into the Transformer encoder after positional encoding. The Transformer encoder uses an attention mechanism to mine the long-distance dependencies between the multimodal fusion features to obtain attention-aware multimodal features, thereby enhancing the understanding of complex image features and the interrelationships between attributes to achieve the Face Attribute Recognition (FAR) task. In order to deepen the Transformer encoder's understanding of face attribute relationships, the attention-aware multimodal features and face attribute labels with random masks are input into the language decoder to perform the Masked Facial Attribute Modeling (MFAM) task to enhance the learning of global relationships between face attributes.

[0072] In this method, Face Attribute Masking (MFAM) is used as an auxiliary task. The specific implementation of MFAM is as follows: First, the face attribute labels corresponding to the face image are randomly sorted, and then the face attribute words are randomly masked according to a certain ratio to obtain the randomly masked face attribute labels. Then, a language encoder is used to encode these labels to obtain the masked linguistic features. The language encoder used in the MFAM task is the same as the language encoder used in the DRAE module. For the attention-aware multimodal features output by the Transformer encoder... This can be understood as the sum of facial image features, facial attribute relation encoding, and location encoding. The attention-aware multimodal features output by the Transformer encoder are input into the language decoder, which is then used to assist in predicting facial attribute words at the mask location. In this embodiment, the language decoder used for facial attribute mask modeling has 12 layers. The specific structure of each layer's language decoder is as follows... Figure 3 As shown, during mask prediction, the attention-aware multimodal features output by the Transformer encoder are input into the language decoder of each layer. Specifically, the language decoder of layer l utilizes a cross-attention mechanism to process the attention-aware multimodal features. and the language features masked at layer l-1 Preliminary fusion was performed to obtain preliminary masked language features. Further analysis of masked language features Obtaining masked language features of layer l using a self-attention mechanism and They are represented as follows:

[0073]

[0074] Where, N mask Indicates the length of the masked language features; when l = 0, The masked language features of the initial input;

[0075] Throughout the face attribute masking modeling process, the Transformer encoder continuously refines the extraction of face image features and the modeling of face attribute relationships. For both the speech encoder and speech decoder, this embodiment employs a 12-layer Transformer. To maintain compatibility with different Transformer encoders, the speech encoder and speech decoder retain the original input dimensions of the Transformer encoder during encoding and decoding.

[0076] E) During training, the face attribute recognition network model simultaneously performs the face attribute recognition (FAR) task and the face attribute masking modeling (MFAM) task, and continuously optimizes the parameters of the Transformer encoder. For face attribute recognition, the FAR task uses the sigmoid cross-entropy function as the loss function and treats the FAR task as multiple binary classification tasks. The MFAM task uses the negative log-likelihood function as the loss function to predict the face attribute labels of randomly masked faces. Both tasks use the SGD optimizer to minimize the error and jointly affect the update of the Transformer encoder parameters during training.

[0077] In this embodiment, the specific method for constructing the loss functions for the FAR and MFAM tasks is as follows.

[0078] In this invention, the parameters of the Transformer encoder are updated during training due to the influence of the FAR and MFAM tasks. Considering that the FAR task for recognizing 40 facial attributes is a binary classification problem, the sigmoid cross-entropy function is adopted. As a loss function, it can be expressed as:

[0079]

[0080] in, and Let N represent the predicted value and the true label of the face attribute of the j-th person in the i-th image, respectively; σ(·) represents the sigmoid function; N s A represents the number of training samples; A represents the number of facial attributes.

[0081] For the MFAM task, the negative log-likelihood function is used. As a loss function; therefore, the calculation of the MFAM loss function can be expressed as:

[0082]

[0083] Where M represents the number of face attribute tags that are masked; m i This represents the predicted value of the attribute label of the masked face in the MFAM task; This indicates the number of words in the Wikipedia corpus dictionary used by the language encoder.

[0084] During training, both loss functions jointly affect the update of the Transformer encoder parameters. By integrating the tasks of FAR and MFAM through a joint loss function, the total loss function can be expressed as:

[0085]

[0086] This embodiment also provides a Transformer face attribute recognition system based on mask attribute modeling, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0087] In this embodiment, validation sets of the CelebA and LFWA datasets are used for comparative validation. Table 1 shows the comparison results between the proposed method and other facial attribute recognition methods on the CelebA and LFWA datasets. As can be seen from Table 1, the proposed method has higher accuracy and robustness compared to other facial attribute recognition methods, specifically demonstrating the best ACC (Accuracy Correction).

[0088] Table 1

[0089]

[0090] In Table 1, PANDA corresponds to the method proposed by Ning Zhang et al. (Ning Zhang, Manohar Paluri, Marc'Aurelio Ranzato, Trevor Darrell, and Lubomir Bourdev. 2014. Panda: Pose aligned networks for deep attribute modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1637–1644.)

[0091] LNets+ANets corresponds to the method proposed by Ziwei Liu et al. (Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3730–3738.)

[0092] MCNN-AUX corresponds to the method proposed by Emily M Hand et al. (Emily M Hand and Rama Chellappa. 2017. Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 4068–4074.)

[0093] AFFAIR corresponds to the method proposed by Jianshu Li et al. (Jianshu Li, Fang Zhao, Jiashi Feng, Sujoy Roy, Shuicheng Yan, and Terence Sim. 2018. Landmark-free face attribute prediction. IEEE Transactions on Image Processing 27, 9(2018), 4651–4662.)

[0094] MCFA corresponds to the method proposed by Ni Zhuang et al. (Ni Zhuang, Yan Yan, Si Chen, and Hanzi Wang. 2018. Multi-task learning of cascaded CNN for facial attribute classification. In Proceedings of the IEEE International Conference on Pattern Recognition. 2069–2074.)

[0095] SPLITFACE corresponds to the method proposed by Upal Mahbub et al. (Upal Mahbub, Sayantan Sarkar, and Rama Chellappa. 2020. Segment-based methods for facial attribute detection from partial faces. IEEE Transactions on Affective Computing 11,4(2020),601–613.)

[0096] SSPL corresponds to the method proposed by Ying Shu et al. (Ying Shu, Yan Yan, Si Chen, Jing HaoXue, Chunhua Shen, and Hanzi Wang. 2021. Learning spatial-semantic relationship for facial attribute recognition with limited labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11916–11925.)

[0097] DMM-CNN corresponds to the method proposed by Longbiao Mao et al. (Longbiao Mao, Yan Yan, Jing HaoXue, and Hanzi Wang. 2022. Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Effective Computing 13,2(2022),818–828.)

[0098] MZTS corresponds to the method proposed by Si Chen et al. (Si Chen, Xueyan Zhu, DaHan Wang, Shunzhi Zhu, and Yun Wu. 2023. Multizone transformer based on self-distillation for facial attribute recognition. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 1–7.)

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A Transformer-based face attribute recognition method based on mask attribute modeling, characterized in that, Includes the following steps: 1) Obtain a face attribute recognition dataset and a face attribute label combination, wherein the face attribute recognition dataset contains face images and their corresponding face attribute labels, and the face attribute labels consist of one or more face attribute words; 2) Construct a face attribute recognition network model based on mask attribute modeling. The face attribute recognition network model includes multiple components such as a dynamic relationship-aware encoding module, a Transformer encoder, a language encoder, and a language decoder. These multiple components work together to perform face attribute recognition and face attribute mask modeling tasks. The language encoder encodes information from the language modality to obtain face attribute relationship encoding from the language modality. The dynamic relationship-aware encoding module fuses information from the language modality and the visual modality to obtain multimodal fusion features. The Transformer encoder further mines the long-distance dependencies between multimodal fusion features to obtain attention-aware multimodal features; the language decoder performs face attribute masking modeling on the attention-aware multimodal features to enhance the model's understanding of global attribute relationships; and the constructed face attribute recognition network model is trained using a face attribute recognition dataset. 3) Apply the trained face attribute recognition network model to the face attribute recognition task; The specific implementation method of the dynamic relationship-aware encoding module is as follows: First, the DRAE module accepts face attribute relation encoding. and facial image features As input, N is the number of facial attribute words in the facial attribute label combination, and C, H, and W are the number of channels, height, and width of the image, respectively; the facial image features are expanded into Then, the relationship between the face attributes and the encoding is performed. Interact to adjust the weight of each facial attribute in the facial attribute relationship encoding based on facial image features; Weighted face attribute relationship encoding Represented as: Where d is the encoding dimension of the Transformer encoder; It is the scaling factor; and Having the same dimensions ; To extract facial attribute relationships from weighted facial attribute relationship encoding, the relationships between facial attribute words are modeled; facial attribute relationships Represented as: in, This indicates a convolution operation used to extract the relationships between facial attribute words and to model the relationships between facial attributes. To uniformly integrate the extracted facial attribute relationships into the facial image features, the extracted facial attribute relationships are reshaped to match the dimensions of the facial image features, and the two are connected along the channel dimension; secondly, the connected features... Convolutional operations are performed to restore the original number of channels; then, the features are weighted by information gating and added to the original face image features to generate multimodal fusion features. The information gate adjusts the contribution of facial attribute relationships to multimodal fusion features during the summation process to adapt to different levels of image quality; the concatenated features and the obtained multimodal fusion features They are represented as follows: in, Indicates a splicing operation; Information gates are used, which employ trainable parameters. of The function adjusts the proportion of facial attribute relationships in multimodal features.

2. The Transformer face attribute recognition method based on mask attribute modeling according to claim 1, characterized in that, In step 1), a face attribute recognition dataset is obtained from public datasets including CelebA and LFWA, and the face attribute label combination includes 40 face attribute labels.

3. The Transformer face attribute recognition method based on mask attribute modeling according to claim 1, characterized in that, In step 2), the implementation method of the face attribute recognition network model is as follows: A) Load the pre-trained language encoder weights to initialize the language encoder, and use the language encoder to extract the face attribute relationship code from the face attribute label combination; B) Encode the face images in the face attribute recognition dataset to obtain face image features; C) The facial attribute relationship encoding and facial image features are fed into the dynamic relationship-aware encoding module, i.e., the DRAE module. The dynamic relationship-aware encoding module first uses an information gating mechanism to dynamically adjust the weight of the language modality information based on the input facial image features, so that it matches the visual modality information in the facial image, reducing the difference between the visual modality and the language modality. Then, the facial attribute relationship encoding and facial image features are fused in the channel dimension. Through the gating mechanism and the guided attention mechanism, the facial attribute relationship in the language modality is effectively integrated into the facial image features, thereby obtaining a multimodal fusion feature that includes facial image and facial attribute relationship. D) The multimodal fusion features are input into the Transformer encoder after positional encoding. The Transformer encoder uses an attention mechanism to mine the long-distance dependencies between the multimodal fusion features to obtain attention-aware multimodal features, thereby enhancing the understanding of complex image features and the interrelationships between attributes to achieve the face attribute recognition task, i.e., the FAR task. The attention-aware multimodal features and the face attribute labels of the random mask are input into the language decoder to perform the face attribute mask modeling task, i.e., the MFAM task, to enhance the learning of the global relationships between face attributes. E) During training, the face attribute recognition network model simultaneously performs the FAR and MFAM tasks, continuously optimizing the parameters of the Transformer encoder. For face attribute recognition, the FAR task uses the sigmoid cross-entropy function as the loss function and treats the FAR task as multiple binary classification tasks. The MFAM task uses the negative log-likelihood function as the loss function to predict the face attribute labels of randomly masked faces. Both tasks use the SGD optimizer to minimize the error and jointly affect the update of the Transformer encoder parameters during training.

4. The Transformer face attribute recognition method based on mask attribute modeling according to claim 3, characterized in that, In step A), the language encoder is a BERT model pre-trained with a corpus dictionary. The language encoder encodes the face attribute labels in the face attribute label combination to obtain the face attribute relationship code.

5. The Transformer face attribute recognition method based on mask attribute modeling according to claim 3, characterized in that, In step B), the original image of size 224*224 is first segmented into image blocks of size adapted to the input of the Transformer encoder. Then, the image blocks are passed through a linear layer and a classification head and position encoding are added to obtain the face image features.

6. The Transformer face attribute recognition method based on mask attribute modeling according to claim 3, characterized in that, In step D), face attribute masking modeling is used as an auxiliary task. The specific implementation method of face attribute masking modeling is as follows: First, the face attribute labels corresponding to the face image are randomly sorted, and the face attribute words are randomly masked according to a certain proportion to obtain the face attribute labels of the random mask. Then, the language encoder is used to encode them to obtain the masked language features. The attention-aware multimodal features output by the Transformer encoder are the sum of face image features, face attribute relationship encoding, and position encoding. The attention-aware multimodal features output by the Transformer encoder are input into the language decoder, and the language decoder is used to assist in predicting the face attribute words of the mask position. The language decoder used for face attribute masking has multiple layers. During mask prediction, the attention-aware multimodal features output by the Transformer encoder are input into the language decoder of each layer. The language decoder of the l-th layer utilizes a cross-attention mechanism to process the attention-aware multimodal features. and the language features masked at layer l-1 Preliminary fusion is performed to obtain preliminary masked language features. Then, the masked language features Obtaining masked language features of layer l using a self-attention mechanism ; and They are represented as follows: in, Indicates the length of the masked language features; When l=0, The masked language features of the initial input; Throughout the face attribute masking modeling process, the Transformer encoder continuously refines the extraction of face image features and the modeling of face attribute relationships. To maintain compatibility with different Transformer encoders, the speech encoder and speech decoder will retain the original input dimensions of the Transformer encoder during the encoding and decoding processes.

7. The Transformer face attribute recognition method based on mask attribute modeling according to claim 3, characterized in that, In step E), the specific method for constructing the loss functions for the FAR and MFAM tasks is as follows: the parameters of the Transformer encoder will be updated during training due to the influence of the FAR and MFAM tasks; considering that the FAR task is a binary classification problem, the sigmoid cross-entropy function is used. As a loss function, it is expressed as: in, and They represent the first The first image Predicted values ​​and true labels of individual facial attributes; Represents the sigmoid function; Indicates the number of training samples; Indicates the number of facial attributes; For the MFAM task, the negative log-likelihood function is used. As a loss function, it is expressed as: in, Indicates the number of face attribute tags that are masked; m i This represents the predicted value of the attribute label of the masked face in the MFAM task; This indicates the number of words in the corpus dictionary used by the language encoder; During training, both loss functions mentioned above jointly affect the update of the Transformer encoder parameters; therefore, the total loss function is expressed as: 。 8. A Transformer face attribute recognition system based on mask attribute modeling, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images