A method and system for facial attribute recognition based on visual language models

Abstract

This invention relates to a method and system for facial attribute recognition based on a visual language model. The method includes: 1) Given all facial attribute labels to be predicted, acquiring multiple facial images and their corresponding facial attribute labels to form a facial attribute recognition dataset; 2) Constructing a facial attribute recognition network model based on a visual language model, which mainly includes a visual encoder, a language encoder, and multiple visual-language interaction encoders. Each visual-language interaction encoder includes a visual-language cross-attention module and a self-attention module. The visual-language cross-attention module includes a visual-language modulated attention mechanism and a language-visual guided attention mechanism; training the network model using the facial attribute recognition dataset and a combination of 40 facial attribute labels; 3) Using the trained facial attribute recognition network model for facial attribute recognition. This method and system are beneficial for obtaining more stable, robust, and accurate facial attribute recognition results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for facial attribute recognition based on a visual language model. Background Technology

[0002] Facial attribute recognition aims to predict various attributes in a given facial image, and its applications are wide-ranging, including face verification, face recognition, image generation, and image retrieval. However, due to the susceptibility of visual images to factors such as lighting variations, low contrast, noise, and poor visibility, facial attribute recognition has always been a challenging task. Existing facial attribute recognition methods generally employ convolutional neural networks (CNNs) to learn the deep features of facial images, and improve the accuracy of facial attribute recognition by having the model learn features related to facial attributes or by incorporating relationships between facial attributes.

[0003] Currently, mainstream facial attribute recognition methods can be divided into two categories: auxiliary task-based facial attribute recognition methods and multi-task-based facial attribute recognition methods. Auxiliary task-based methods aim to help the backbone network learn effective facial attribute features through multiple face-related auxiliary tasks, thereby improving facial attribute recognition. Ni Zhuang et al. (Ni Zhuang, Yan Yan, Si Chen, and Hanzi Wang. 2018. Multi-task learning of cascaded CNN for facial attribute classification. In Proceedings of the IEEE International Conference on Pattern Recognition. 2069–2074.) proposed cascaded face detection and facial landmark detection to assist facial attribute recognition. They simultaneously trained three related face analysis tasks within a unified framework, while multi-task learning was performed by inputting a single attribute in the facial attribute recognition process. Ying Shu et al. (Ying Shu, Yan Yan, Si Chen, JingHao Xue, Chunhua Shen, and Hanzi Wang. 2021. Learning spatial-semantic relationship for facial attribute recognition with limited labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11916–11925.) designed a task including block rotation (PRT), block segmentation (PST), and block classification (PCT) to learn spatial-semantic relationships from large-scale unlabeled face data and apply them to facial attribute recognition.

[0004] For multi-task-based facial attribute recognition methods, the aim is to group attributes based on their correlations and assign each attribute group to a learning task. Emily M Hand et al. (Emily M Hand and RamaChellappa. 2017. Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 4068–4074.) divided facial attributes into 9 groups based on different facial attribute locations and used a multi-task deep CNN with auxiliary networks to discover hierarchical attribute relationships. Longbiao Mao et al. (Longbiao Mao, YanYan, JingHao Xue, and Hanzi Wang. 2022. Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Affective Computing 13,2(2022),818–828.) artificially divided attributes into two groups based on the learning complexity of different attributes and designed two different network architectures to construct the attribute relationships within the two groups. Xiaohui Fanhe et al. (Xiaohui Fanhe, Jie Guo, Zheng Huang, Weidong Qiu, and Yuele Zhang. 2019. Multi-task learning with knowledge transfer for facial attribute classification. In Proceedings of the IEEE International Conference on Industrial Technology. 877–882) considered the different degrees of correlation between facial attributes, used spectral clustering to cluster facial attributes, and designed a multi-task framework based on this.

[0005] However, these facial attribute relationships are obtained either through fixed clustering algorithms or through manual grouping, and the relationships between facial attributes obtained in these ways cannot be well modeled or fully utilized. Furthermore, existing facial attribute recognition methods only focus on the visual modality, and when the input image contains interfering factors or has low resolution, these methods cannot effectively learn the relationships between facial attributes, thus failing to help the model recognize facial attributes. Inspired by the field of natural language processing, we hope that the model can learn facial attribute relationships from the linguistic modality and use the learned relationships to guide the extraction of image features. Summary of the Invention

[0006] The purpose of this invention is to provide a face attribute recognition method and system based on a visual language model, which is beneficial for obtaining more stable, robust and accurate face attribute recognition results.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a face attribute recognition method based on a visual language model, comprising the following steps:

[0008] 1) Given all the facial attribute labels that need to be predicted, obtain multiple facial images and their facial attribute labels to form a facial attribute recognition dataset;

[0009] 2) Construct a face attribute recognition network model based on a visual language model. The face attribute recognition network model mainly includes a visual encoder, a language encoder, and multiple visual-language interaction encoders. Each visual-language interaction encoder includes a visual-language cross-attention module and a self-attention module. The visual-language cross-attention module includes a visual-language modulating attention mechanism and a language-visual guiding attention mechanism. The face attribute recognition network model is trained using a face attribute recognition dataset and 40 combinations of face attribute labels.

[0010] 3) Use the trained face attribute recognition network model for face attribute recognition.

[0011] Further, in step 1), the face attribute recognition dataset is the CelebA and LFWA datasets.

[0012] Furthermore, in step 2), the implementation method of the facial attribute recognition network model is as follows:

[0013] A) Load the pre-trained language encoder weights from the Wikipedia corpus dataset to initialize the language encoder, enabling it to extract the semantic relationships between the 40 facial attribute labels from the 40 facial attribute label combinations and convert them into a text sequence that can be input into the visual-language interaction encoder.

[0014] B) Encode the face images in the face attribute recognition dataset using a visual encoder, making them an image sequence that can be input into the visual language interaction encoder;

[0015] C) After concatenating, normalizing, and splitting the text sequence and image sequence, the input is fed into the visual-language interaction encoder. The input sequence first passes through the visual-language cross-attention module, where the visual-language modulated attention mechanism modulates the facial attribute relationships from semantic knowledge in the text sequence and adapts them to the image sequence. Then, the language-visual guided attention mechanism uses the modulated text sequence to guide the distribution of the image sequence, making it more prominent in terms of attribute-related feature information. Finally, the resulting image sequence guided by the text sequence is input into the self-attention module for feature extraction. The modulated text sequence and the image sequence with features extracted by the self-attention module are concatenated, added to the original input to obtain an intermediate sequence, then normalized and mapped to a multilayer perceptron, and summed with the original intermediate sequence. After being split according to the previous size, it becomes a new text sequence and image sequence, which are then input into the next visual-language interaction encoder.

[0016] D) In ​​order to align the visual modality with the language modality and enable better fusion of information from the two modalities, loss calculations are performed on the classification heads of the text sequence and image sequence after all visual-language interaction encoders. The error between the two is minimized by the SGD optimizer to enhance the alignment between the two modalities.

[0017] E) To ensure the correctness of language modality guidance, the classification head of the text sequence obtained after all visual-language interaction encoders is mapped by a multilayer perceptron. The loss is calculated between the recognition result and the true result, and the error between the two is minimized by the SGD optimizer to ensure that the language modality can correctly guide the visual modality and highlight the attribute-related features.

[0018] F) The classification head of the image sequence obtained after all visual language interaction encoders is mapped by a linear layer to obtain the face attribute prediction result. The loss is calculated by comparing it with the real result, and the error between the two is minimized by the SGD optimizer to perform face attribute recognition.

[0019] Further, in steps A) and B), the language encoder uses a BERT model pre-trained on a Wikipedia corpus to encode a set of 40 facial attribute labels, resulting in a text sequence of size 124*768; the visual encoder divides the original image of size 3*224*224 into 16*16 image blocks, and then passes the image blocks through a linear layer and adds a classification head and position encoding to obtain an image sequence of size 197*768.

[0020] Further, in step C), the implementation method of the visual language interaction encoder is as follows:

[0021] First, in the visual attentional regulation mechanism of language, the text sequence is adjusted according to the image sequence to adapt it to the visual modality. This process can be represented as follows:

[0022]

[0023] in, and This represents normalized image and text sequences. This represents the adjusted text sequence; D and H are the set dimensions and number of heads.

[0024] Secondly, in the language-guided attention mechanism of vision, the distribution of image sequences is guided by modulated text sequences to make them more prominent in attribute-related feature information. The process is represented as follows:

[0025]

[0026]

[0027] in, This indicates an image sequence guided by a modified text sequence.

[0028] Finally, in the self-attention module, feature extraction is performed on the image sequence guided by the adjusted text sequence. The process is as follows:

[0029]

[0030] Further, in step D), the difference between the classification heads in the text sequence and the image sequence output by the last visual-language interaction encoder is calculated, and learnable parameters are added as weights for adjustment to obtain the modality fusion loss. Updating the network model using the SGD optimizer is represented as follows:

[0031]

[0032] Where λ represents the learnable parameter, Denotes KL divergence, and These represent the classification heads for text sequences and image sequences, respectively.

[0033] Further, in step E), the difference between the recognition result obtained after the classification head of the text sequence is mapped by a multilayer perceptron and the true result is calculated to obtain the text conditioning loss. Updating the network model using the SGD optimizer is represented as follows:

[0034]

[0035] in, This represents the result of predicting the j-th attribute of the i-th image after the classification head of the text sequence is mapped by a multilayer perceptron.

[0036] Further, in step F), the difference between the recognition result obtained after the classification head of the image sequence is mapped through a linear layer and the true result is calculated to obtain the face attribute classification loss. Updating the network model using the SGD optimizer is represented as follows:

[0037]

[0038] Where σ represents the Sigmoid activation function; log represents the log function; M represents the Mth training image; and A represents the A attributes of each image. This represents the true label of the j-th attribute of the i-th image; This represents the result of predicting the j-th attribute of the i-th image after the classification head of the image sequence has passed through a linear layer mapping.

[0039] The present invention also provides a face attribute recognition system based on a visual language model, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0040] Compared with the prior art, the present invention has the following beneficial effects: The method and system effectively utilize the semantic relationship between facial attributes in natural language. By using this facial attribute relationship from the language modality to guide the model to extract features from facial images, more robust features that are more relevant to facial attributes can be obtained, effectively solving problems such as lighting, occlusion, and low pixel count, and improving the accuracy of facial attribute recognition. Attached Figure Description

[0041] Figure 1 This is an architecture diagram of a face attribute recognition network model based on a visual language model in an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] This embodiment provides a face attribute recognition method based on a visual language model, including the following steps:

[0046] 1) Given all the facial attribute labels that need to be predicted, obtain multiple facial images and their facial attribute labels to form a facial attribute recognition dataset.

[0047] 2) Construct a face attribute recognition network model based on a visual language model. This network model mainly includes a visual encoder, a language encoder, and multiple visual-language interaction encoders. Each visual-language interaction encoder includes a visual-language cross-attention module and a self-attention module. The visual-language cross-attention module contains a visual-to-language modulating attention mechanism and a language-to-visual guiding attention mechanism. The language encoder is initialized by loading pre-trained language encoder weights from a Wikipedia corpus dataset. The entire face attribute recognition network model is trained using a face attribute recognition dataset and a combination of 40 face attribute labels. In this embodiment, the architecture of the face attribute recognition network model is as follows: Figure 1 As shown.

[0048] 3) Use the trained face attribute recognition network model for face attribute recognition.

[0049] In step 1), the face attribute recognition dataset is the CelebA and LFWA datasets.

[0050] In step 2), the implementation method of the face attribute recognition network model is as follows:

[0051] A) Load the pre-trained language encoder weights from the Wikipedia corpus dataset to initialize the language encoder, enabling it to extract the semantic relationships between the 40 facial attribute labels from the 40 facial attribute label combinations and convert them into a text sequence that can be input into the visual-language interaction encoder.

[0052] In step A), the language encoder uses a BERT model pre-trained on a Wikipedia corpus to encode a set of 40 facial attribute labels. The resulting text sequence is 124*768 in size.

[0053] B) Encode the face images in the face attribute recognition dataset using a visual encoder, making them an image sequence that can be input into the visual language interaction encoder.

[0054] In step B), the visual encoder divides the original image of size 3*224*224 into 16*16 image blocks, and then passes the image blocks through a linear layer and adds a classification head and position encoding to obtain an image sequence of size 197*768.

[0055] C) The text sequence and image sequence are concatenated, normalized, and split before being input into the visual-language interaction encoder. The input sequence first passes through the visual-language cross-attention module, where the visual-language modulated attention mechanism modulates the facial attribute relationships from semantic knowledge in the text sequence and adapts them to the image sequence. Then, the language-visual guided attention mechanism uses the modulated text sequence to guide the distribution of the image sequence, making it more prominent in terms of attribute-related feature information. Finally, the resulting image sequence guided by the text sequence is input into the self-attention module for feature extraction. The modulated text sequence and the image sequence with features extracted by the self-attention module are concatenated, added to the original input to obtain an intermediate sequence, then normalized and mapped to a multilayer perceptron, and added to the original intermediate sequence. After being split according to the previous size, it becomes a new text sequence and image sequence, which are then input into the next visual-language interaction encoder.

[0056] In step C), the implementation method of the visual language interaction encoder is as follows:

[0057] First, in the visual attentional regulation mechanism of language, the text sequence is adjusted according to the image sequence to adapt it to the visual modality. This process can be represented as follows:

[0058]

[0059] in, and This represents normalized image and text sequences. This represents the adjusted text sequence; D and H are the set dimensions and number of heads.

[0060] Secondly, in the language-guided attention mechanism of vision, the distribution of image sequences is guided by modulated text sequences to make them more prominent in attribute-related feature information. The process is represented as follows:

[0061]

[0062]

[0063] in, This refers to an image sequence guided by a modified text sequence.

[0064] Finally, in the self-attention module, feature extraction is performed on the image sequence guided by the adjusted text sequence. The process is as follows:

[0065]

[0066] D) In ​​order to align the visual modality with the language modality and enable better fusion of information from the two modalities, loss calculations are performed on the respective classification heads of the text sequence and image sequence after all visual-language interaction encoders. The error between the two is minimized by the SGD optimizer to enhance the alignment between the two modalities.

[0067] In step D), the specific method for constructing the modal alignment loss is as follows:

[0068] The difference between the classifier heads in the text sequence and the image sequence output by the last visual-language interactive encoder is calculated, and learnable parameters are added as weights for adjustment to obtain the modality fusion loss. Updating the network model using the SGD optimizer is represented as follows:

[0069]

[0070] Where λ represents the learnable parameter, Denotes KL divergence, and These represent the classification heads for text sequences and image sequences, respectively.

[0071] E) To ensure the correctness of language modality guidance, the classification head of the text sequence obtained after all visual-language interaction encoders is mapped by a multilayer perceptron. The loss between the obtained recognition result and the true result is calculated, and the error between the two is minimized by the SGD optimizer to ensure that the language modality can correctly guide the visual modality and highlight the attribute-related features.

[0072] In step E), the specific method for constructing the language-aware classification loss is as follows:

[0073] The text conditioning loss is obtained by calculating the difference between the recognition result obtained by the classification head of the text sequence after being mapped by a multilayer perceptron and the true result. Updating the network model using the SGD optimizer is represented as follows:

[0074]

[0075] in, This represents the result of predicting the j-th attribute of the i-th image after the classification head of the text sequence is mapped by a multilayer perceptron.

[0076] F) The classification head of the image sequence obtained after all visual language interaction encoders is mapped by a linear layer to obtain the face attribute prediction result. The loss is calculated by comparing it with the real result, and the error between the two is minimized by the SGD optimizer to perform face attribute recognition.

[0077] In step F), the specific method for constructing the face attribute classification loss is as follows:

[0078] The difference between the recognition result obtained after the classification head of the image sequence is mapped through a linear layer and the true result is calculated to obtain the face attribute classification loss. Updating the network model using the SGD optimizer is represented as follows:

[0079]

[0080] Where σ represents the Sigmoid activation function; log represents the log function; M represents the Mth training image; and A represents the A attributes of each image. This represents the true label of the j-th attribute of the i-th image; This represents the result of predicting the j-th attribute of the i-th image after the classification head of the image sequence has passed through a linear layer mapping.

[0081] This embodiment also provides a face attribute recognition system based on a visual language model, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0082] In this embodiment, validation sets of the CelebA and LFWA datasets are used for comparative validation. Table 1 shows the comparison results between the proposed method and other facial attribute recognition methods on the CelebA and LFWA datasets. As can be seen from Table 1, the proposed method has higher accuracy and robustness compared to other facial attribute recognition methods, specifically demonstrating the best ACC (Accuracy Correction).

[0083] Table 1

[0084]

[0085] In Table 1, PANDA corresponds to the method proposed by Ning Zhang et al. (Ning Zhang, Manohar Paluri, Marc'Aurelio Ranzato, Trevor Darrell, and Lubomir Bourdev. 2014. Panda: Pose aligned networks for deep attribute modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1637–1644.)

[0086] LNets+ANets corresponds to the method proposed by Ziwei Liu et al. (Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3730–3738.)

[0087] MCNN-AUX corresponds to the method proposed by Emily M Hand et al. (Emily M Hand and Rama Chellappa. 2017. Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 4068–4074.)

[0088] AFFAIR corresponds to the method proposed by Jianshu Li et al. (Jianshu Li, Fang Zhao, Jiashi Feng, Sujoy Roy, Shuicheng Yan, and Terence Sim. 2018. Landmark-free face attribute prediction. IEEE Transactions on Image Processing 27, 9(2018), 4651–4662.)

[0089] MCFA corresponds to the method proposed by Ni Zhuang et al. (Ni Zhuang, Yan Yan, Si Chen, and Hanzi Wang. 2018. Multi-task learning of cascaded CNN for facial attribute classification. In Proceedings of the IEEE International Conference on Pattern Recognition. 2069–2074.)

[0090] SPLITFACE corresponds to the method proposed by Upal Mahbub et al. (Upal Mahbub, Sayantan Sarkar, and Rama Chellappa. 2020. Segment-based methods for facial attribute detection from partial faces. IEEE Transactions on Affective Computing 11,4(2020),601–613.)

[0091] SSPL corresponds to the method proposed by Ying Shu et al. (Ying Shu, Yan Yan, Si Chen, Jing Hao Xue, Chunhua Shen, and Hanzi Wang. 2021. Learning spatial-semantic relationship for facial attribute recognition with limited labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11916–11925.)

[0092] DMM-CNN corresponds to the method proposed by Longbiao Mao et al. (Longbiao Mao, Yan Yan, JingHaoXue, and Hanzi Wang. 2022. Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Affective Computing 13,2(2022),818–828.)

[0093] MZTS corresponds to the method proposed by Si Chen et al. (Si Chen, Xueyan Zhu, DaHan Wang, Shunzhi Zhu, and Yun Wu. 2023. Multizone transformer based on self-distillation for facial attribute recognition. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 1–7.)

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A face attribute recognition method based on a visual language model, characterized in that, Includes the following steps: 1) Given all the facial attribute labels that need to be predicted, obtain multiple facial images and their facial attribute labels to form a facial attribute recognition dataset; 2) Construct a face attribute recognition network model based on a visual language model. The face attribute recognition network model mainly includes a visual encoder, a language encoder, and multiple visual-language interaction encoders. Each visual-language interaction encoder includes a visual-language cross-attention module and a self-attention module. The visual-language cross-attention module includes a visual-language modulating attention mechanism and a language-visual guiding attention mechanism. The face attribute recognition network model is trained using a face attribute recognition dataset and 40 combinations of face attribute labels. 3) Use the trained face attribute recognition network model for face attribute recognition; In step 2), the implementation method of the face attribute recognition network model is as follows: A) Load the pre-trained language encoder weights from the Wikipedia corpus dataset to initialize the language encoder, enabling it to extract the semantic relationships between the 40 facial attribute labels from the 40 facial attribute label combinations and convert them into a text sequence that can be input into the visual-language interaction encoder. B) Encode the face images in the face attribute recognition dataset using a visual encoder, making them an image sequence that can be input into the visual-language interaction encoder; C) The text sequence and image sequence are concatenated, normalized, and split before being input into the visual-language interaction encoder. The input sequence first passes through the visual-language cross-attention module, where the visual-language modulated attention mechanism modulates the facial attribute relationships from semantic knowledge in the text sequence and adapts them to the image sequence. Then, the language-visual guided attention mechanism uses the modulated text sequence to guide the distribution of the image sequence, making it more prominent in terms of attribute-related feature information. Finally, the resulting image sequence guided by the text sequence is input into the self-attention module for feature extraction. The modulated text sequence and the image sequence with features extracted by the self-attention module are concatenated, added to the original input to obtain an intermediate sequence, then normalized and mapped to a multilayer perceptron, and added to the original intermediate sequence. After being split according to the previous size, it becomes a new text sequence and image sequence, which are then input into the next visual-language interaction encoder. D) In ​​order to align the visual modality with the language modality and enable better fusion of information from the two modalities, loss calculations are performed on the classification heads of the text sequence and image sequence after all visual-language interaction encoders. The error between the two is minimized by the SGD optimizer to enhance the alignment between the two modalities. E) To ensure the correctness of language modality guidance, the classification head of the text sequence obtained after all visual-language interaction encoders is mapped by a multilayer perceptron. The loss is calculated between the recognition result and the true result, and the error between the two is minimized by the SGD optimizer to ensure that the language modality can correctly guide the visual modality and highlight the attribute-related features. F) The classification head of the image sequence obtained after all visual language interaction encoders is mapped by a linear layer to obtain the face attribute prediction result. The loss is calculated by comparing it with the real result, and the error between the two is minimized by the SGD optimizer to perform face attribute recognition.

2. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In step 1), the face attribute recognition dataset is the CelebA and LFWA datasets.

3. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In steps A) and B), the language encoder uses a BERT model pre-trained on a Wikipedia corpus to encode a set of 40 facial attribute labels, resulting in a text sequence of size 124×768. The visual encoder divides the original image of size 3×224×224 into 16×16 image blocks, and then passes the image blocks through a linear layer and adds a classification head and position encoding to obtain an image sequence of size 197×768.

4. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In step C), the implementation method of the visual language interaction encoder is as follows: First, in the visual attentional regulation mechanism of language, the text sequence is adjusted according to the image sequence to adapt it to the visual modality. This process can be represented as follows: in, and This represents normalized image and text sequences. This indicates a regulated text sequence; and The dimensions and number of multiple heads are set; Secondly, in the language-guided attention mechanism of vision, the distribution of image sequences is guided by modulated text sequences to make them more prominent in attribute-related feature information. The process is represented as follows: in, This indicates an image sequence guided by a modified text sequence. Finally, in the self-attention module, feature extraction is performed on the image sequence guided by the adjusted text sequence. The process is as follows: 。 5. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In step D), the difference between the classifier heads in the text sequence and the image sequence output by the last visual-language interaction encoder is calculated, and learnable parameters are added as weights for adjustment to obtain the modality fusion loss. The network model is updated using the SGD optimizer, as shown below: in, Indicates learnable parameters, Denotes KL divergence, and These represent the classification heads for text sequences and image sequences, respectively.

6. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In step E), the difference between the recognition result obtained by mapping the classification head of the text sequence through a multilayer perceptron and the true result is calculated to obtain the text conditioning loss. The network model is updated using the SGD optimizer, as shown below: in, This represents the result of predicting the j-th attribute of the i-th image after the classification head of the text sequence is mapped by a multilayer perceptron.

7. The face attribute recognition method based on a visual language model according to claim 1, characterized in that, In step F), the difference between the recognition result obtained after the classification head of the image sequence is mapped through a linear layer and the true result is calculated to obtain the face attribute classification loss. The network model is updated using the SGD optimizer, as shown below: in, This represents the Sigmoid activation function; Represents the log function; Indicates the first Zhang training photos; This indicates that each image has a total of One attribute; This represents the true label of the j-th attribute of the i-th image; This represents the result of predicting the j-th attribute of the i-th image after the classification head of the image sequence has passed through a linear layer mapping.

8. A facial attribute recognition system based on a visual language model, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the method as described in any one of claims 1-7.

Images