Face attribute recognition method and system based on multistage multi-scale feature fusion

Abstract

The invention belongs to the field of face attribute recognition, and particularly relates to a face attribute recognition method and system based on multistage multi-scale feature fusion. The method comprises the following steps: acquiring a to-be-detected face image; performing multi-detection attribute object positioning on the to-be-detected face image by adopting a positioning network to obtain a key position of each attribute object, and obtaining an attribute weight based on the key position of each attribute object; performing multi-scale and multi-level fusion processing on the to-be-detected face image based on the attribute weight of each attribute object to obtain multi-scale and multi-level fusion features; and identifying the multi-scale multi-level fusion features by using a multi-attribute classifier to obtain a detection result of each attribute object of the to-be-detected face image. According to the method, the attribute weight is obtained through the key position of each attribute object of the face image, and the importance in subsequent feature extraction and feature fusion can be accurately measured by using the attribute weight of the face attribute; and thus, the recognition performance of face attribute recognition is improved.

Description

Technical Field

[0001] The present invention relates to a method and system for recognizing human face attributes, and in particular to a method and system for recognizing human face attributes based on multi-level and multi-scale feature fusion. Background Art

[0002] Facial attribute recognition aims to identify whether a certain facial attribute exists, such as whether there are bangs, whether wearing glasses, whether the hair is black, etc. It has broad application value in computer vision and pattern recognition fields such as face recognition, image generation, and image retrieval.

[0003] Since the introduction of deep learning, convolutional neural networks (CNNs) have dominated a number of computer vision tasks such as image classification, with typical examples being ResNet, SENet, MobileNet, and EfficientNet. In recent years, due to the outstanding performance of Transformer in natural language processing tasks, people have begun to use it for computer vision tasks and have achieved performance results comparable to CNN, with representative results including ViT, DeiT, and Swin Transformer. Si Chen et al. (Chen, S., et al. Multi-Zone Transformer Based on Self-Distillation for Facial Attribute Recognition. IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG). 2023) used Transformer for the first time in the task of facial attribute recognition; Lixiong Qin et al. (Qin, L., et al., SwinFace: A Multi-Task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation. IEEE Transactions on Circuits and Systems for Video Technology, 2024. 34(4): 2223-2234) also used Swin Transformer and combined it with multi-task learning to simultaneously complete face recognition, expression recognition, age recognition, and face attribute recognition tasks, achieving good facial attribute recognition results.

[0004] However, although the Transformer uses a self-attention mechanism that has the ability to capture long-distance information, that is, the use of the Transformer is conducive to the introduction of global information, it usually ignores local feature information. On the contrary, CNN uses a small convolution kernel for convolution operations, which has good local characteristics, but the limited receptive field makes it difficult for the network to obtain global information. In the face attribute recognition task, face attributes may contain multiple different attribute objects. The fusion of multi-scale global features and local features is very important for accurate recognition, but it is obvious that the use of CNN or Transformer alone cannot simultaneously achieve the effective use of local features and global features of different face attribute objects. Summary of the invention

[0005] The purpose of the present invention is to provide a facial attribute recognition method and system that combines CNN and Transformer for multi-level and multi-scale feature extraction and fusion. The attribute weights based on the facial attribute objects can accurately measure their importance in subsequent feature extraction and feature fusion. It has the advantages of CNN in extracting low-level features and strong locality and the advantages of Transformer in establishing long-distance dependencies. It can effectively extract and fuse multi-level and multi-scale local features and global features, thereby improving the recognition performance of facial attribute recognition.

[0006] To achieve the above object, the present invention provides a method and system for face attribute recognition based on multi-level and multi-scale feature fusion, including the following technical solutions:

[0007] In a first aspect of the present application, the present application provides a method for face attribute recognition based on multi-level and multi-scale feature fusion, the method comprising:

[0008] Acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition;

[0009] Using a positioning network to locate multiple detection attribute objects on the face image to be detected, obtaining a key position of each attribute object, and obtaining an attribute weight based on the key position of each attribute object;

[0010] Based on the attribute weight of each attribute object, the face image to be detected is subjected to multi-scale multi-level fusion processing to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by the interaction of multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network;

[0011] The multi-scale and multi-level fusion features are identified using a multi-attribute classifier to obtain detection results of various attribute objects of the face image to be detected.

[0012] In a second aspect of the present application, the present application further provides a face attribute recognition system based on multi-level and multi-scale feature fusion, the system comprising:

[0013] A face image acquisition module is used to acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition;

[0014] A face image positioning module, used for positioning the multi-detection attribute objects of the face image to be detected using a positioning network, obtaining the key position of each attribute object, and obtaining the attribute weight based on the key position of each attribute object;

[0015] A multi-scale multi-level feature fusion module is used to perform multi-scale multi-level fusion processing on the face image to be detected based on the attribute weight of each attribute object to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by the interaction of multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network;

[0016] The face attribute recognition module is used to recognize the multi-scale and multi-level fusion features using a multi-attribute classifier to obtain detection results of various attribute objects of the face image to be detected.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] This method and system obtain the attribute weight through the key position of each attribute object of the face image. The attribute weight of the face attribute can accurately measure its importance in subsequent feature extraction and feature fusion; it effectively combines the advantages of CNN locality and Transformer in capturing global information, and can extract and fuse multi-level and multi-scale local features and global features, thereby improving the recognition performance of face attribute recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0020] Figure 1This is a flow chart of a method for face attribute recognition based on multi-level and multi-scale feature fusion in an embodiment of the present application;

[0021] Figure 2 This is a flow chart of the dual attention mechanism fusion method of multi-scale and multi-level fusion features in the embodiment of the present application;

[0022] Figure 3 This is a flow chart of the processing method of the channel attention module in the embodiment of the present application;

[0023] Figure 4 This is a flow chart of the processing method of the spatial attention module in the embodiment of the present application;

[0024] Figure 5 This is a flow chart of the processing method of the inverted residual multilayer perceptron in an embodiment of the present application;

[0025] Figure 6 A flowchart of the interaction method of the multi-level interactive local features of the embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of the face attribute recognition network model structure of the embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of the operation of the face attribute recognition network model in the embodiment of the present application;

[0028] Fig. 9 This is a schematic diagram of the architecture of a face attribute recognition system based on multi-level and multi-scale feature fusion according to an embodiment of the present application;

[0029] Fig.10 This is a schematic diagram of the structure of the multi-scale and multi-level feature fusion module of an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] This embodiment provides a face attribute method based on multi-level and multi-scale feature fusion, which can be applied to electronic devices, and the electronic devices can be servers on the network side or terminal devices on the user side. The terminal devices can be: PC (Personal Computer), or smart phones, or tablet computers, or car computers, or digital cameras, or game consoles, or smart TVs, etc. Here, this embodiment does not specifically limit what kind of device the terminal device is.

[0032] Figure 1 FIG. 4 is a flow chart of a method for face attribute recognition based on multi-level and multi-scale feature fusion in an embodiment of the present invention; Figure 1 As shown, the method comprises the following steps:

[0033] 101. Acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition;

[0034] Specifically, the face image to be detected refers to the face image to be subjected to face attribute recognition. In the embodiment of the present application, a face image can be taken in real time by a photographing device as the face image to be detected, or a face image uploaded by a user through a client can be received as the face image to be detected.

[0035] Optionally, the acquired face image is subjected to image processing to obtain a face image to be detected. Specifically, the acquired color face image is subjected to grayscale processing to obtain a grayscale image of the face, and then the grayscale image is histogram-equalized to obtain a processed face image as the target face image. Grayscale processing reduces redundant information in the original face image, which is more conducive to the recognition of face attributes.

[0036] Specifically, the attribute object to be detected refers to the attribute object for face attribute recognition, and the attribute object to be detected may be multiple attributes, such as the age of the face, whether it is an oval face, whether it is a round face, whether it is a high cheekbone, whether it is a high nose bridge, whether it is big eyes, whether it has a beard, whether it is male or female, whether it is made up, whether it is a young person, whether it is attractive, etc. Another example is age, gender, and expression. Among the multiple attributes, age is a regression attribute, and gender and expression are not regression attributes. In practical applications, each attribute can be represented in numerical form. For example, there are two genders, 0 can be used to represent male, and 1 can be used to represent female. There can be 6 expressions: happy, sad, fear, anger, surprise and disgust. The 6 expressions can be numbered, respectively 0, 1, 2, 3, 4, and 5.

[0037] In an embodiment of the present application, CelebA and LFWA data sets can be collected for training of face attribute recognition network models. Taking the CelebA data set as an example, CelebA contains 202,599 facial images of celebrities, each of which is annotated with 40 different facial attributes. The CelebA data set is divided into three parts, the first 162,770 face images are divided into a training set, the 162,771st to 182,637th face images are divided into a verification set, and the 182,638th to 202,599th face images are divided into a test set; the network model in the face attribute recognition method of the present application can be trained, verified and tested through the training set, verification set and test set to achieve the recognition accuracy of face attributes.

[0038] 102. Using a positioning network to locate multiple detection attribute objects on the face image to be detected, obtain a key position of each attribute object, and obtain an attribute weight based on the key position of each attribute object;

[0039] In an embodiment of the present application, when processing a face image, a positioning network, such as a target detection network in deep learning, is used to locate the position information of multiple attribute objects; since the position information of each attribute object has been determined, the attribute weight of the corresponding attribute object can be obtained based on the saliency of the attribute object in the image, the distance from the face center, the size ratio, etc. For example, the distance between the attribute object and the face center is used as the weight basis, and the attribute object with a closer distance may have a higher weight, and vice versa; or, for example, the weight is calculated by combining the detection confidence and position information of the attribute object, and the attribute object with a higher confidence and a more important position may obtain a higher weight.

[0040] 103. Based on the attribute weight of each attribute object, the face image to be detected is subjected to multi-scale multi-level fusion processing to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by interacting multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network;

[0041] In some embodiments of the present application, Figure 2 As shown, the dual attention mechanism fusion method of the multi-scale and multi-level fusion features includes:

[0042] 301. Process the multi-scale fusion features of the previous level through convolution and average pooling to obtain the first intermediate fusion feature vector of the current level;

[0043] 302. Concatenate the interactive global features, the interactive local features, and the first intermediate fusion feature vector of the current level to obtain a second intermediate fusion feature vector of the current level;

[0044] 303. After the interactive global features and the interactive local features of the current level are concatenated, they are processed by the channel attention module and the spatial attention module to obtain the third intermediate fusion feature vector of the current level;

[0045] 304. Process the second intermediate fusion feature vector and the third intermediate fusion feature vector of the current level through an inverted residual multilayer perceptron to obtain a multi-scale fusion feature of the current level.

[0046] In the embodiments of the present application, Figure 3 As shown, the processing method of the channel attention module includes:

[0047] 311. The concatenated features of the current level are processed by maximum pooling and multi-layer perceptron to obtain the first channel features;

[0048] 312. The concatenated features of the current level are processed by average pooling and multi-layer perceptron to obtain the second channel features;

[0049] 313. The first channel feature and the second channel feature of the current level are processed through an activation function layer to obtain a channel attention feature.

[0050] In the embodiments of the present application, Figure 4 As shown, the processing method of the spatial attention module includes:

[0051] 321. The channel attention feature of the current level is processed by maximum pooling to obtain the first spatial feature;

[0052] 322. The channel attention features of the current level are processed by average pooling to obtain the second spatial features;

[0053] 323. Concatenate and convolve the first spatial feature and the second spatial feature of the current level to obtain a spatial attention feature map;

[0054] 324. The spatial attention feature map of the current level is processed through an activation function layer to obtain the spatial attention feature.

[0055] In the embodiments of the present application, Figure 5 As shown, the processing method of the inverted residual multilayer perceptron includes:

[0056] 331. Concatenate the first intermediate fused feature vector and the second intermediate fused feature vector of the current level to obtain a first fused feature vector.

[0057] 332. Perform layer normalization processing on the first fused feature vector of the current level to obtain a first normalized feature;

[0058] 333. Process the first fused feature vector of the current level through layer normalization and one convolution layer to obtain a first convolution feature;

[0059] 334. Process the first normalized feature and the first convolution feature of the current level through an adder to obtain a first added feature;

[0060] 335. The first addition feature of the current level is processed by a convolution layer to obtain an inverted residual feature;

[0061] 336. The inverted residual feature of the current level is concatenated with the first intermediate fusion feature vector to obtain a multi-scale fusion feature of the current level.

[0062] In some embodiments of the present application, Figure 6 As shown, the interaction mode of the multi-level interactive local features includes:

[0063] 341. Embed the local features of each level into the global features of the corresponding level based on the attribute weights of each attribute object to obtain interactive local features of each attribute object at each level;

[0064] 342. Based on the attribute weights of each attribute object, the global features of each level are embedded into the local features of the corresponding level to obtain interactive global features of all attribute objects at each level.

[0065] For example, assuming that the attribute weight of hair color is 0.4, the attribute weight of hair shape is 0.1, the local feature of hair color in the second layer is represented by a, the local feature of hair shape in the second layer is represented by a', and the global feature in the second layer is represented by b, then the interactive local feature of hair color in the second layer can be represented as 0.4a(f(b)). Similarly, the interactive global feature in the second layer can be represented as b(0.4f(a)+0.1f(a')+…), where f() represents the interactive function, which can be a common multi-layer perceptron, etc.

[0066] It can be understood that the two-way feature interaction mechanism of the embodiment of the present application can extract global and local features of facial attributes. By realizing two-way feature interaction, complex patterns and relationships in the data can be captured, thereby improving the predictive ability and robustness of the model.

[0067] In some embodiments of the present application, in order to obtain multi-scale and multi-level fusion features, such as Figure 7As shown, the embodiment of the present application constructs a face attribute recognition network model that combines multi-level and multi-scale feature extraction and fusion of CNN and Transformer. The face attribute recognition network model mainly includes a local feature extraction module based on CNN, a global feature extraction module based on Transformer, and a dual attention feature fusion module; wherein, the face attribute recognition network model may include a four-level structure, and of course, the corresponding cascade structure may be increased or decreased in actual application; each level of the structure is a stage, and each stage may include a local feature extraction module, a global feature extraction module, and a dual attention feature fusion module (DAFF).

[0068] In some embodiments of the present application, the local feature extraction module uses the Inception-ResNet block to integrate and construct a CNN local feature extraction network. First, the Inception-ResNet block uses a series of parallel convolutional layers with different convolution kernel sizes to extract multi-scale features from the input tensor. Secondly, the outputs of these parallel layers are spliced ​​in the channel dimension to generate a high-dimensional feature map, and the dimension of the feature map is reduced by a 1×1 convolutional layer. Finally, the input feature vector and the output feature vector are residually connected to further extract the local features while retaining the integrity of the local feature information.

[0069] In some embodiments of the present application, when a face image is input into a face attribute recognition training network and enters a local feature extraction module, feature extraction and downsampling operations are performed through a convolution layer and a maximum pooling layer to pre-process the input image. The pre-processed features are subjected to layer normalization operations, and the feature vectors subjected to layer normalization operations are subjected to multiple layers of convolution layers of different sizes to obtain feature maps of different sizes. The multi-layer feature splicing is residually connected with the original input feature vector to obtain local features, and feature information at different levels is obtained while maintaining the content information of the feature vector without loss.

[0070] In some embodiments of the present application, the global feature extraction module is constructed based on the Swin Transformer block, and a single Swin Transformer block is used to extract feature maps at different levels. First, the face image is segmented by patch segmentation and linear embedding and converted into a high-dimensional embedding vector. Then, the obtained embedding vector is input into the Swin Transformer block, and multi-scale feature extraction is realized through a hierarchical structure and a sliding window mechanism, thereby obtaining a more hierarchical and global feature representation.

[0071] In some embodiments of the present application, a face image is input into a face attribute recognition network. When entering the global feature extraction module, the input face image is segmented into non-overlapping small blocks through Patch segmentation, and each small block can be regarded as a separate "small image". Each segmented small block is then flattened into a vector through a linear embedding layer, and projected to a higher dimension through a linear layer to obtain an embedded vector. The purpose is to convert each image block into a high-dimensional vector representation for subsequent attention mechanism and feature processing. The embedded vector obtained through the linear embedding layer is subjected to feature extraction to obtain a global feature, and the global feature is adjusted to a feature shape through a linear layer to facilitate input into the next layer of the global feature extraction module.

[0072] In some embodiments, the global feature extraction module can divide the original image of size 3*224*224 into image blocks of size 16*16, and then pass the image blocks through a linear layer and add a classification head and position encoding to obtain an image vector of size 197*768.

[0073] In some embodiments of the present application, the dual attention feature fusion module includes a channel attention mechanism and a spatial attention mechanism, which is used to realize multi-level feature fusion and information interaction between different feature extraction modules. Finally, the fused features are input into a multi-attribute classifier to predict and recognize facial attributes.

[0074] In some embodiments of the present application, the dual attention feature fusion module first obtains the first intermediate fusion feature vector of the current level by convolution and average pooling of the multi-scale fusion features obtained in the previous layer. Then the interactive global features, interactive local features, and the first intermediate fusion feature vector of the current level are spliced ​​to obtain the second intermediate fusion feature vector of the current level. At the same time, the interactive global features and interactive local features of the current level are spliced ​​and input into the channel attention module and the spatial attention module. The input feature vector first passes through the channel attention module, and obtains global context information through operations such as global average pooling and maximum pooling, and automatically learns and enhances important channel features in the network by linking the channel feature information of global features and local features. The channel attention features obtained after the channel attention module are then input into the spatial attention module, and the input feature map is subjected to global average pooling and global maximum pooling in the channel dimension. The pooling operation compresses information in the spatial dimension to obtain two feature maps, which respectively represent the average response and maximum response of each spatial position in different channels. The obtained average pooling and maximum pooling feature maps are spliced, and a spatial attention feature map is generated through a convolutional layer. The spatial information from different channels is aggregated through convolution operations, and a single-channel attention map is generated. The second intermediate fusion feature vector obtained by dual attention fusion and the third intermediate fusion feature vector are concatenated, and the fusion feature is obtained by inverted residual multi-layer perceptron, and the multi-scale fusion feature is obtained by residual connection with the second intermediate fusion feature vector.

[0075] Among them, the local feature extraction module is used to capture the detail features in the face image, focusing on extracting key information such as texture and edge of the local area. The global feature extraction module focuses on the overall structure and global semantic information of the image, and is responsible for integrating the features of different regions, thereby providing a comprehensive description of the overall features of the face. In some embodiments of the present application, in the training stage, the face images in the training set and their corresponding multiple face attribute labels are input into the face attribute recognition training network. The model first extracts the local detail features and global features of the face image respectively through the local feature extraction module and the global feature extraction module. Then, these two types of feature information are passed to the dual attention feature fusion module, in which the channel attention mechanism and the spatial attention mechanism are used to effectively fuse local features with global features. Finally, the fused features are input into the classifier for attribute prediction, and the gap between the prediction result and the real face attribute label is calculated by the classification loss function, and then the gradient of the model is updated to optimize the training process. After the training of the current stage is completed, the validation set data is used to verify the model of the current stage, which is used to tune the hyperparameters of the model during the training process and prevent the model from overfitting.

[0076] In the embodiment of the present application, since there are a total of multi-level structures in the face attribute recognition network model, each structure includes a global feature extraction, a local feature extraction and a dual attention feature fusion operation; for the convenience of description, the following steps are described by taking the i-th layer in the model as an example, i≠1, such as Figure 8 As shown:

[0077] First, the multi-scale fusion feature F obtained in the previous layer i-1 After convolution and average pooling, the first intermediate fusion feature vector of the current level is obtained. The process is expressed as:

[0078]

[0079] Among them, F i-1 Represents the multi-scale fusion features input by the double attention feature fusion module of the previous layer; represents the first intermediate fusion feature vector of the current level; i represents the i-th layer of the model; f 1×1 Represents a convolution operation, and the size of the convolution kernel is 1×1;

[0080] Secondly, the first intermediate fusion feature vector of the current level Interacting with global features Interactive local features f l i Further splicing and fusion are performed to obtain intermediate fusion features The process is expressed as:

[0081]

[0082] in, Represents the second intermediate fusion feature vector of the current level; i represents the i-th layer of the model; Concat[·] represents the concatenation operation, which concatenates the vectors in the channel direction; f 3×3 Represents the convolution operation, the size of the convolution kernel is 3×3;

[0083] At the same time, the interactive global features of the current level Interactive local features f l i The splicing is performed and input into the channel attention module and the spatial attention module for dual attention fusion. The process is expressed as:

[0084]

[0085] in, Represents the global eigenvector Local eigenvector f l iThe feature vector after double attention fusion, that is, the third intermediate fusion feature vector of the current level; i represents the i-th layer of the model; The CA(·) and SA(·) processes are expressed as follows:

[0086]

[0087] CA(x)=sigmoid(MLP(AvgPool(x))+MLP(MaxPool(x))) (5)

[0088] SA(x)=sigmoid(f 7×7 (Concat[AvgPool(x),MaxPool(x)])) (6)

[0089] Among them, the vector Represents interactive global features and the interaction local feature f l i The concatenated vector; ca(·) and SA(·) represent the process of channel attention and spatial attention; x represents input; MLP(·) represents multi-layer perceptron; i represents the i-th layer of the model; f 7×7 Represents the convolution operation, the size of the convolution kernel is 7×7;

[0090] Finally, the second intermediate fusion feature vector The third intermediate fused feature vector after dual attention fusion Perform splicing and fusion to obtain the final multi-scale fusion feature F i :

[0091]

[0092] Among them, F i represents the multi-scale fusion features of the current level; IrM(·) represents the inverted residual multilayer perceptron; Concat[·] represents the concatenation operation; the IrM(·) process is expressed as follows:

[0093] IrM(x)=f 1×1 (f 1×1 (f 3×3 (LN(x))+LN(x))) (8)

[0094] Among them, f 1×1 、f 3×3 represents the convolution operation, and the sizes of the convolution kernels are 1×1 and 3×3 respectively; LN(·) represents layer normalization; x represents input;

[0095] In the embodiment of the present application, by calculating each layer of local feature vector fl ;i The local mutual information maximization loss function of the vector FC1 obtained with the output of the first fully connected layer is used to improve the model's prediction ability for complex and changeable facial attributes by maximizing the mutual information between the local area of ​​the image and the corresponding attribute label.

[0096] In a preferred embodiment of the present application, the loss function based on local mutual information maximization includes:

[0097] The interactive local features of the current level and the feature vector obtained by the first fully connected layer of the multi-attribute classifier are processed through local mutual information to obtain the corresponding real loss and false loss;

[0098] The sum of the real loss and the false loss of all levels is maximized to optimize the parameters of the multi-attribute classifier.

[0099] Specifically, the local mutual information maximization loss function and the classification loss function are implemented as follows:

[0100] First, by maximizing the mutual information between the local area of ​​the image and the corresponding attribute label, the model's ability to predict complex and changeable facial attributes is improved. The process is expressed as:

[0101]

[0102] Among them, L LMIM represents the local mutual information maximization loss function, L real represents the loss of predicting the true label, L fake represents the loss of the predicted false label; σ(·) represents the sigmoid function; i represents the corresponding i-level structure; j and k represent the jth and kth face images in the dataset respectively; y i,j ,y i,k Respectively represent the predicted labels corresponding to the jth and kth face images in the i-th Stage; f l i,j 、f l i,k Represents the local feature map corresponding to the jth and kth face images in the i-th Stage; represents the FC1 vector corresponding to the first fully connected layer in the classifier of the local feature map corresponding to the jth face image in the i-th stage; Concat[·] represents the concatenation operation; (k≠j; y i,j ,y i,k The true labels are 1 and 0 respectively)

[0103] Secondly, by calculating the binary cross entropy loss function value between the predicted face attribute results and the true results, the gradient is updated to reduce the error of the predicted results. The process is expressed as:

[0104]

[0105] Among them, L FAC represents the face attribute classification loss; N represents the number of face images in the training set; A represents the number of attributes to be predicted; y j,a represents the predicted label of the a-th attribute of the j-th image; Represents the predicted value of the a-th attribute of the j-th image.

[0106] Finally, the total loss function of the network model is the sum of the face attribute classification loss and the local mutual information maximization loss, and the process is expressed as:

[0107] L total =L FAC +L LmIM (13)

[0108] Among them, L total Represents the total loss function of the entire network model; L FAC represents the face attribute classification loss function; L LMIM represents the local mutual information maximization loss function.

[0109] 104. Use a multi-attribute classifier to identify the multi-scale and multi-level fusion features to obtain detection results of various attribute objects of the face image to be detected.

[0110] In the embodiment of the present application, the last layer of multi-scale multi-level fusion features is mapped through a linear layer to obtain a facial attribute prediction result, and the classification loss is calculated with the real result, and the error between the two is minimized through the AdamW optimizer to perform facial attribute recognition. The multi-scale multi-level fusion features are passed through a convolution layer, layer normalization, activation function, and then through two fully connected layers to obtain the facial attribute prediction result.

[0111] In some embodiments of the present application, the present invention further provides a face attribute recognition system based on multi-level and multi-scale feature fusion, the face attribute recognition system 200 comprising:

[0112] The face image acquisition module 201 is used to acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition;

[0113] The face image positioning module 202 is used to use a positioning network to perform multi-detection attribute object positioning on the face image to be detected, obtain the key position of each attribute object, and obtain the attribute weight based on the key position of each attribute object;

[0114] The multi-scale multi-level feature fusion module 203 is used to perform multi-scale multi-level fusion processing on the face image to be detected based on the attribute weight of each attribute object to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by the interaction of multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network;

[0115] The face attribute recognition module 204 is used to recognize the multi-scale and multi-level fusion features using a multi-attribute classifier to obtain detection results of various attribute objects of the face image to be detected.

[0116] In the embodiments of the present application, Fig.10 As shown, the multi-scale multi-level feature fusion module 203 includes:

[0117] A first feature extraction unit 211 is used to extract multi-level local features through a CNN network;

[0118] A second feature extraction unit 212 is used to extract multi-level global features through Transformer;

[0119] A first feature interaction unit 213, configured to embed the local features of each level into the global features of the corresponding level based on the attribute weights of each attribute object, to obtain interactive local features of each attribute object at each level;

[0120] A second feature interaction unit 214, configured to embed the global features of each level into the local features of the corresponding level based on the attribute weights of each attribute object, to obtain interactive global features of all attribute objects at each level;

[0121] The first attention fusion unit 215 is used to process the splicing features of the current level through maximum pooling and multi-layer perceptron to obtain the first channel features; process the splicing features of the current level through average pooling and multi-layer perceptron to obtain the second channel features; process the first channel features and the second channel features of the current level through the activation function layer to obtain the channel attention features;

[0122] The second attention fusion unit 216 is used to process the channel attention features of the current level through maximum pooling to obtain the first spatial features; process the channel attention features of the current level through average pooling to obtain the second spatial features; concatenate and convolve the first spatial features and the second spatial features of the current level to obtain a spatial attention feature map; process the spatial attention feature map of the current level through an activation function layer to obtain the spatial attention features.

[0123] It can be understood that the face attribute recognition method based on multi-level and multi-scale feature fusion and the face attribute recognition system based on multi-level and multi-scale feature fusion of the present invention belong to the same inventive concept, and their corresponding features can be referenced to each other, and the present invention will not describe them one by one.

[0124] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A face attribute recognition method based on multi-level and multi-scale feature fusion, characterized in that: The method comprises: Acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition; Using a positioning network to locate multiple detection attribute objects on the face image to be detected, obtaining a key position of each attribute object, and obtaining an attribute weight based on the key position of each attribute object; Based on the attribute weight of each attribute object, the face image to be detected is subjected to multi-scale multi-level fusion processing to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by the interaction of multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network; The multi-scale and multi-level fusion features are identified using a multi-attribute classifier to obtain detection results of various attribute objects of the face image to be detected.

2. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 1, characterized in that: The dual attention mechanism fusion method of the multi-scale and multi-level fusion features includes: The multi-scale fusion features of the previous level are processed by convolution and average pooling to obtain the first intermediate fusion feature vector of the current level; The interactive global features, the interactive local features, and the first intermediate fused feature vector of the current level are concatenated to obtain a second intermediate fused feature vector of the current level; The interactive global features and interactive local features of the current level are concatenated and processed through the channel attention module and the spatial attention module to obtain the third intermediate fusion feature vector of the current level; The second intermediate fusion feature vector and the third intermediate fusion feature vector of the current level are processed by an inverted residual multilayer perceptron to obtain the multi-scale fusion features of the current level.

3. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 2, characterized in that: The processing method of the channel attention module includes: The concatenated features of the current level are processed by maximum pooling and multi-layer perceptron to obtain the first channel features; The concatenated features of the current level are processed by average pooling and multi-layer perceptron to obtain the second channel features; The first channel features and the second channel features of the current level are processed through the activation function layer to obtain the channel attention features.

4. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 3, characterized in that: The processing method of the spatial attention module includes: The channel attention features of the current level are processed by maximum pooling to obtain the first spatial features; The channel attention features of the current level are processed by average pooling to obtain the second spatial features; The first spatial feature and the second spatial feature of the current level are concatenated and convolved to obtain a spatial attention feature map; The spatial attention feature map of the current level is processed through the activation function layer to obtain the spatial attention feature.

5. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 2, characterized in that: The processing method of the inverted residual multilayer perceptron includes: Concatenate the first intermediate fused feature vector and the second intermediate fused feature vector of the current level to obtain a first fused feature vector; The first fused feature vector of the current level is processed by layer normalization to obtain the first normalized feature; The first fused feature vector of the current level is processed by layer normalization and one convolution layer to obtain the first convolution feature; The first normalized feature and the first convolution feature of the current level are processed by an adder to obtain a first additive feature; The first additive feature of the current level is processed through a convolutional layer to obtain an inverted residual feature; The inverted residual feature of the current level is concatenated with the first intermediate fusion feature vector to obtain the multi-scale fusion feature of the current level.

6. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 2, characterized in that: The interaction mode of the multi-level interactive local features includes: Embedding the local features of each level into the global features of the corresponding level based on the attribute weights of each attribute object to obtain interactive local features of each attribute object at each level; Based on the attribute weights of each attribute object, the global features of each level are embedded into the local features of the corresponding level to obtain the interactive global features of all attribute objects at each level.

7. A method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 1 or 2, characterized in that: The method also includes optimizing parameters of the multi-attribute classifier based on a local mutual information maximization loss function and a classification loss function.

8. The method for face attribute recognition based on multi-level and multi-scale feature fusion according to claim 7, characterized in that: The loss function based on local mutual information maximization includes: The interactive local features of the current level and the feature vector obtained by the first fully connected layer of the multi-attribute classifier are processed through local mutual information to obtain the corresponding real loss and false loss; The sum of the real loss and the false loss of all levels is maximized to optimize the parameters of the multi-attribute classifier.

9. The face attribute recognition system based on multi-level and multi-scale feature fusion according to claim 1, characterized in that: The system comprises: A face image acquisition module is used to acquire a face image to be detected; the face image to be detected includes a plurality of attribute objects to be detected; the attribute objects to be detected include hair shape, hair color, glasses wearing condition and / or hat wearing condition; A face image positioning module, used for positioning the multi-detection attribute objects of the face image to be detected using a positioning network, obtaining the key position of each attribute object, and obtaining the attribute weight based on the key position of each attribute object; A multi-scale multi-level feature fusion module is used to perform multi-scale multi-level fusion processing on the face image to be detected based on the attribute weight of each attribute object to obtain multi-scale multi-level fusion features; the multi-scale multi-level fusion features are obtained by fusing multi-level interactive global features and multi-level interactive local features through a double attention mechanism; the multi-level interactive global features and the multi-level interactive local features are obtained by the interaction of multi-level local features and multi-level global features; the multi-level local features are extracted by a CNN network, and the multi-level global features are extracted by a Transformer network; The face attribute recognition module is used to recognize the multi-scale and multi-level fusion features using a multi-attribute classifier to obtain detection results of various attribute objects of the face image to be detected.

10. The face attribute recognition system based on multi-level and multi-scale feature fusion according to claim 9, characterized in that: The multi-scale and multi-level feature fusion module includes: A first feature extraction unit is used to extract multi-level local features through a CNN network; The second feature extraction unit is used to extract multi-level global features through Transformer; A first feature interaction unit, configured to embed the local features of each level into the global features of the corresponding level based on the attribute weights of the attribute objects, so as to obtain interactive local features of the attribute objects at each level; A second feature interaction unit, configured to embed the global features of each level into the local features of the corresponding level based on the attribute weights of each attribute object, so as to obtain interactive global features of all attribute objects at each level; The first attention fusion unit is used to process the splicing features of the current level through maximum pooling and multi-layer perceptron to obtain the first channel features; the splicing features of the current level are processed through average pooling and multi-layer perceptron to obtain the second channel features; the first channel features and the second channel features of the current level are processed through the activation function layer to obtain the channel attention features; The second attention fusion unit is used to process the channel attention features of the current level through maximum pooling to obtain the first spatial features; process the channel attention features of the current level through average pooling to obtain the second spatial features; concatenate and convolve the first spatial features and the second spatial features of the current level to obtain a spatial attention feature map; process the spatial attention feature map of the current level through an activation function layer to obtain the spatial attention features.

Images