Face-wearing article recognition method and system based on feature fusion

Abstract

The application discloses a face-wearing object recognition method and system based on feature fusion, relates to the technical field of computer vision, and solves the technical problems of serious feature cross interference, poor posture adaptability and low recognition accuracy in the existing face-wearing object recognition; the method comprises the following steps: firstly, initial features are extracted through a lightweight identity encoder and a wearing object encoder with a double-attention module; then, a single generator-single discriminator adversarial learning network is constructed in combination with a double collaborative judgment mechanism; subsequently, local region features of the wearing object, face component relationship features and global face features are extracted to construct a multi-dimensional feature set; then, the feature weights are dynamically adjusted through a posture perception attention mechanism and fusion is completed; finally, a double-branch classification structure is adopted in combination with a secondary recognition mechanism to output the wearing object category; and the application realizes efficient and accurate recognition of various face-wearing objects.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for facial adornment recognition based on feature fusion. Background Technology

[0002] With the widespread application of computer vision technology in fields such as security, attendance, and smart terminals, the demand for accurate recognition of face-worn items (such as glasses, masks, hats, and headphones) is becoming increasingly urgent.

[0003] However, existing face recognition technologies still suffer from the following key bottlenecks: In traditional methods, facial identity features and accessory attributes are often coupled, leading to interference from identity information in accessory feature extraction. This is especially true when the accessory fits the face closely (e.g., glasses, masks), resulting in severe feature confusion and a significant drop in recognition accuracy. Facial pose changes (lateral turning, tilting) can cause the accessory to occlude or deform in the image. Existing technologies often use fixed feature extraction methods, which cannot dynamically adjust feature weights based on pose, leading to a sharp drop in recognition accuracy in non-frontal poses. Most technologies only focus on local features of the accessory or global features of the face, ignoring the spatial relationship between facial components (such as eyes and nose) and the accessory. This results in insufficient feature representation capabilities, making it difficult to distinguish between similar accessories (sunglasses and regular glasses, baseball caps and flat caps). Furthermore, existing classification models often use a single-branch structure, which does not adequately cover niche accessory categories and lacks an effective confidence verification mechanism, making misclassification likely. In particular, stability is difficult to guarantee in complex backgrounds or lighting conditions.

[0004] Meanwhile, the core objective of existing technologies is mostly to eliminate the interference of wearing objects on facial recognition, rather than to accurately identify the wearing objects themselves. The technical approach does not match the actual needs of wearing object recognition and cannot meet the recognition requirements in diverse scenarios.

[0005] Therefore, a face-wearing recognition method and system based on feature fusion has emerged. Summary of the Invention

[0006] This invention aims to solve at least one of the technical problems existing in the prior art; to this end, this invention proposes a face-wearing recognition method and system based on feature fusion, to solve the following technical problem:

[0007] In traditional methods, facial identity features and the attributes of the worn items are often coupled, leading to interference from identity information in the extraction of item features. This is especially true when the item fits the face closely (e.g., glasses, masks), resulting in severe feature confusion and a significant drop in recognition accuracy. Changes in facial posture (turning, tilting) can cause the item to be occluded or deformed in the image. Existing technologies often use fixed feature extraction methods, which cannot dynamically adjust feature weights according to posture, leading to a sharp drop in recognition accuracy in non-frontal poses. Most technologies only focus on local features of the item or global features of the face, ignoring the spatial relationship between facial components (such as eyes and nose) and the item. This results in insufficient feature representation capabilities, making it difficult to distinguish similar items (sunglasses and regular glasses, baseball caps and flat caps). Furthermore, existing classification models often use a single-branch structure, which has insufficient coverage of niche item categories and lacks an effective confidence verification mechanism, making them prone to misclassification. In particular, stability is difficult to guarantee in complex backgrounds or scenes with changing lighting.

[0008] To address the above problems, a first aspect of the present invention provides a face-wearing recognition method based on feature fusion, comprising the following steps: S1: Obtain facial image data containing the worn items, construct an identity encoder and an item encoder, and extract facial identity features and item attribute features respectively by taking the facial image data as input. S2: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. S3: Process the decoupled wearable attribute features, extract local region features of the wearable, and perform face component detection on the face image data to determine the location information of key face components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; S4: Construct a pose-aware attention mechanism model. Input the global facial features, local features of the worn object, and component relationship features into the attention mechanism model. Perform pose detection on the facial image data to obtain facial pose information. Then, dynamically adjust the weight coefficients of the global facial features, local features of the worn object, and component relationship features. Perform dynamic weighted fusion on the global facial features, local features of the worn object, and component relationship features to obtain fused features. S5: Input the fused features into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearer.

[0009] Preferably, in step S1, constructing the identity encoder and the wearable encoder includes the following steps: The identity encoder employs a lightweight convolutional neural network structure, comprising: one input layer, four feature extraction units, and one fully connected layer. Each feature extraction unit is a concatenated structure of a convolutional layer, a batch normalization layer, a ReLU activation function, and a max-pooling layer. The convolutional layers uniformly employ 3... The convolutional kernel is 3 pixels in size, with a fixed stride of 1. The first and third feature extraction units contain only 1 convolutional layer, while the second and fourth feature extraction units contain 2 convolutional layers. The fully connected layer flattens the feature map output by the fourth feature extraction unit into a one-dimensional vector and outputs the final face identity feature vector. The wearable encoder is based on the ResNet-18 framework. After each residual block in its third and fourth stages, a channel-spatial dual attention module is embedded. The module calculates the weight coefficients of each channel in the feature map output by the residual block through the channel attention branch, and then inputs the channel-weighted feature map into the spatial attention branch to locate the spatial region of the wearable on the feature map. The pixels in the region are assigned high weights and the background region is assigned low weights. Finally, the wearable attribute feature vector is output.

[0010] Preferably, in step S2, constructing the adversarial learning network includes the following steps: The adversarial learning network adopts a single generator-single discriminator architecture. The generator is a parallel structure of the identity encoder and the wearable encoder, which synchronously outputs facial identity features. Characteristics of the worn item The discriminator employs a two-layer fully connected layer combined with a sigmoid activation structure to determine whether the input features are pure feature pairs without cross-interference. .

[0011] Preferably, determining whether the input features are pure feature pairs without cross-interference includes the following steps: The discriminator employs a dual collaborative judgment mechanism embedded within a two-layer fully connected structure combined with a Sigmoid activation structure. This mechanism includes a first judgment stage and a second judgment stage, specifically: The first judgment step is to judge the input. and Perform dimensional compliance checks by extracting dimensions using a pre-set dimensional mask matrix. The first 128 dimensions of features and The full 128-dimensional features are used to calculate the cosine similarity between the two features. If the cosine similarity is less than or equal to the preset similarity threshold, the second judgment is performed; otherwise, the feature pair is directly defined as having cross-interference features. The second judgment step is to determine whether the first judgment step has been passed. and Calculate the variance within each feature and the interference evaluation value. ,in, for The variance value, for The variance value; If the interference evaluation value If the value is less than or equal to the preset interference threshold, then... and Input to the fully connected layer; For feature pairs that pass through the dual collaborative judgment mechanism, after processing by the first fully connected layer and the second fully connected layer, the probability value is output by activating the Sigmoid function. If the probability value is greater than 0.85, it is determined to be a pure feature pair without cross interference.

[0012] Preferably, in step S2, optimizing the parameters of the two encoders through adversarial training includes the following steps: Construct a composite loss function, specifically as follows: ; ; ; ; in, This represents the total loss value of the composite loss function. , and For the weights of the corresponding loss function, Loss due to loss in preserving identity features The facial identity features of the anchor sample. Facial identity features of positive samples Facial identity features of negative samples For interval parameters, For the loss of classification of wearable attributes, For the sample size, For sample index, For the true attributes of the wearable item, To predict probabilities, To balance the categories, the weights are set inversely based on the number of items worn in each category in the sample set. To combat the losses, Let be the expected function. The discriminator outputs the probability. For the true feature distribution, To add Gaussian noise interference feature distribution, and These are the interfering facial identity features and interfering accessory attribute features after adding Gaussian noise, respectively; The parameters of the two encoders are optimized by iterating through an adaptive estimator until the composite loss function converges.

[0013] Preferably, in step S3, the decoupled wearable attribute features are processed, including the following steps: A multi-scale sliding window mechanism is adopted, with three window sizes set, each with a step size of 8 pixels. The decoupled wearable attribute feature map is then sampled to obtain multi-scale local feature blocks. An adaptive feature selection unit is introduced, which calculates the information entropy of each feature unit and selects feature units whose information entropy is greater than a preset information entropy threshold. After unifying the dimensions of the first convolutional layer, the features of the local area of ​​the wearer are obtained by stitching them together. The preset information entropy threshold is obtained by calculating the average information entropy of all feature units. An improved cascaded convolutional neural network is used. In the first stage, a lightweight convolutional layer is used to locate the entire face region. In the second stage, a bounding box regression branch is introduced to correct the entire face region. In the third stage, six component detection heads are set up to detect the eyes, nose, mouth, eyebrows, cheeks and forehead respectively, and output the bounding box coordinates of each component.

[0014] Preferably, in step S3, constructing the component relationship model includes the following steps: The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model G=V,E, where V=v1, v2, ..., v6 represent 6 face component nodes and E is the edge between nodes. Define edge weights ,in, Let be the Euclidean distance between nodes vi and vj. The angle between the line connecting the two nodes and the horizontal direction; Feature learning is performed on an undirected graph model using a graph convolutional network to output 64-dimensional component relationship features; The decoupled facial identity feature map is subjected to a dual pooling operation combining global average pooling and global max pooling to obtain two 128-dimensional feature vectors. These vectors are then fused by attention weighting to obtain 256-dimensional global facial features.

[0015] Preferably, step S4 includes the following steps: A 3D pose regression network is used to take face image data as input, output face Euler angles, and classify five pose levels based on the Euler angles. A pose-attention mapping table is set up, and an initial attention weight matrix is ​​preset for each pose level. A dynamic weight adjustment module is introduced. With face pose information as input, the weight correction coefficient is output through a multilayer perceptron to correct the initial attention weight matrix. After obtaining the attention weights for pose adaptation, the global face features, local features of the worn object, and component relationship features are multiplied by the corresponding attention weights respectively. Calculate the feature importance scores of the three types of augmented features. Specifically: ; in, For characteristic variance, The maximum value of the variance of the three types of features. , and These are respectively global facial features, local features of the worn object, and features related to the components; Scoring based on feature importance The three types of features are dynamically weighted and fused to obtain the fused features.

[0016] Preferably, in step S5, the preset classification and recognition model includes the following steps: A feature adaptation layer combined with a dual-branch classification structure is adopted. The feature adaptation layer is passed through two fully connected layers. The dual-branch classification structure includes a coarse classification branch and a fine classification branch. The coarse classification branch is used to identify the major category of the wearable item, and the fine classification branch is used to identify the sub-category of the wearable item. The fused features are input into the classification and recognition model. After dimensional mapping and feature optimization through the feature adaptation layer, they are input into the dual-branch classification layer. The coarse classification branch outputs the predicted probability of the major category of the wearable item, and selects the category with the highest probability as the coarse classification result. The fine classification branch calls the classifier of the corresponding subclass based on the coarse classification result, outputs the predicted probability of the wearable item subclass, and selects the category with the highest probability as the fine classification result. If the maximum probability of the coarse classification result is less than 0.6, a secondary recognition mechanism is activated to re-extract the local features of the wearer and splice them with the fused features. The results are then input into the classification recognition model again for recognition, and finally the major category and sub-category information of the wearer are output.

[0017] A second aspect of the present invention provides a face-wearing recognition system based on feature fusion, comprising the following modules: Feature extraction module: Acquires facial image data including the worn object, constructs an identity encoder and an object encoder, and extracts facial identity features and object attribute features respectively by taking the facial image data as input; Feature decoupling optimization module: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. Multi-dimensional feature refinement and extraction module: Processes the decoupled wearable attribute features, extracts local region features of the wearable, and performs facial component detection on facial image data to determine the location information of key facial components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each facial component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; Pose-aware feature fusion module: Constructs a pose-aware attention mechanism model, inputs global facial features, local features of the worn object, and component relationship features into the attention mechanism model, performs pose detection on the facial image data, obtains facial pose information, dynamically adjusts the weight coefficients of global facial features, local features of the worn object, and component relationship features, and performs dynamic weighted fusion of the global facial features, local features of the worn object, and component relationship features to obtain fused features; Wearable item classification and recognition module: The fused features are input into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearable item.

[0018] The beneficial effects of this invention are: This invention designs a single generator-single discriminator adversarial learning network. The generator is connected in parallel with an identity encoder and an object encoder, and the discriminator is embedded with cosine similarity. This initially filters feature pairs with no obvious coupling. Then, the interference evaluation value is calculated by feature variance to accurately identify latent cross interference. Finally, a composite loss function containing identity fidelity loss, attribute classification loss and adversarial loss is constructed. The encoder parameters are iteratively optimized by an adaptive optimizer to ensure the purity of the decoupled features and greatly reduce the feature cross interference rate. This invention outputs Euler angles of the face and classifies them into 5 pose levels through a 3D pose regression network, achieving accurate classification of different poses. Then, a pose-attention mapping table is constructed with preset initial weights. The weight matrix is ​​dynamically adjusted by outputting weight correction coefficients through a multilayer perceptron. This enables the dynamic weight correction coefficients to adjust the weights of each feature in real time according to pose changes. Finally, the feature importance score is calculated by combining feature variance. The global features of the face, the local features of the worn object, and the features related to the parts are dynamically weighted and fused to further optimize the rationality of weight allocation, thereby improving the recognition accuracy under non-frontal poses and breaking through the limitations of fixed feature extraction. This invention employs a multi-scale sliding window combined with adaptive information entropy filtering to extract multi-scale local features of the wearable object, retaining key feature units with information entropy higher than the mean. It detects six facial components using an improved cascaded convolutional neural network, constructs an undirected graph model with the component center coordinates as nodes, and extracts 64-dimensional component relationship features using a graph convolutional network. The decoupled identity features are then subjected to a dual pooling operation combining global average pooling and global max pooling, combined with attention-weighted fusion to obtain 256-dimensional global facial features. These three types of features work together to achieve a full-dimensional representation of details, associations, and contours, significantly improving the ability to distinguish similar wearable objects. This invention designs a feature adaptation layer combined with a dual-branch classification structure. The coarse classification branch identifies the major categories of wearable items, while the fine classification branch calls the corresponding subclass classifier based on the coarse classification result. When the maximum probability of the coarse classification is less than 0.6, a secondary recognition mechanism is activated to re-extract local features of the wearable item and concatenate them with the fused features for re-identification. The category balance weight is set inversely according to the number of samples to solve the classification bias caused by insufficient samples of niche categories. The secondary recognition mechanism enhances the feature representation ability of low-confidence samples by supplementing local features, thereby greatly improving the overall classification accuracy. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the module flow of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 As shown, this invention is a face-wearing recognition method based on feature fusion, comprising the following steps: S1: Obtain facial image data containing the worn items, construct an identity encoder and an item encoder, and extract facial identity features and item attribute features respectively by taking the facial image data as input. S2: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. S3: Process the decoupled wearable attribute features, extract local region features of the wearable, and perform face component detection on the face image data to determine the location information of key face components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; S4: Construct a pose-aware attention mechanism model. Input the global facial features, local features of the worn object, and component relationship features into the attention mechanism model. Perform pose detection on the facial image data to obtain facial pose information. Then, dynamically adjust the weight coefficients of the global facial features, local features of the worn object, and component relationship features. Perform dynamic weighted fusion on the global facial features, local features of the worn object, and component relationship features to obtain fused features. S5: Input the fused features into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearer.

[0022] In one embodiment of the present invention, step S1, constructing the identity encoder and the wearable encoder, includes the following steps: The identity encoder employs a lightweight convolutional neural network structure, comprising: one input layer, four feature extraction units, and one fully connected layer. Each feature extraction unit is a concatenated structure of a convolutional layer, a batch normalization layer, a ReLU activation function, and a max-pooling layer. The convolutional layers uniformly employ 3... The convolutional kernel is 3 pixels in size, with a fixed stride of 1. The first and third feature extraction units contain only 1 convolutional layer, while the second and fourth feature extraction units contain 2 convolutional layers. The fully connected layer flattens the feature map output by the fourth feature extraction unit into a one-dimensional vector and outputs the final face identity feature vector. The wearable encoder is based on the ResNet-18 framework. After each residual block in its third and fourth stages, a channel-spatial dual attention module is embedded. The module calculates the weight coefficients of each channel in the feature map output by the residual block through the channel attention branch, and then inputs the channel-weighted feature map into the spatial attention branch to locate the spatial region of the wearable on the feature map. The pixels in the region are assigned high weights and the background region is assigned low weights. Finally, the wearable attribute feature vector is output.

[0023] Specifically, the four feature extraction units include: Feature Extraction Unit 1: Contains one 3×3 convolutional layer (32 kernels, stride = 1, padding = 1 to ensure the feature map size is not reduced) → Batch Normalization Layer (momentum = 0.9, epsilon = 1e-5) → ReLU activation function → 2×2 max pooling layer (stride = 2, padding = 0, compressing the feature map size from 224×224 to 112×112); Feature Extraction Unit 2: Contains one 3×3 convolutional layer (128 kernels, stride = 1, padding = 1) → Batch Normalization Layer → ReLU activation function → 2×2 max pooling layer (stride = 2, compressing the feature map size to 28×28); Feature Extraction Unit 3: Contains one 3×3 convolutional layer (128 kernels, stride = 1, padding = 1) → Batch Normalization Layer → ReLU activation function → 2×2 max pooling layer (stride = 2, compressing the feature map size to 28×28); 128 (stride=1, padding=1) → Batch normalization layer → ReLU activation function → 2×2 max pooling layer (stride=2, feature map size compressed to 28×28); 4th feature extraction unit: containing 2 layers of 3×3 convolutional layers (first layer convolutional kernel number=256, second layer convolutional kernel number=256, stride=1, padding=1) → Batch normalization layer → ReLU activation function → 2×2 max pooling layer (stride=2, feature map size compressed to 14×14); The channel attention branch operation involves: The feature map output from the residual block (denoted as X, dimension = H × W × C, where H is height, W is width, and C is the number of channels); global average pooling (H × W → 1 × 1) is performed on X to obtain the channel feature vector (dimensional = 1 × 1 × C); two fully connected layers (the first layer output dimension = C / 4, activation function is ReLU; the second layer output dimension = C, activation function is Sigmoid) are used to generate the channel weight coefficient vector (dimensional = 1 × 1 × C, denoted as W_c); the original feature map X is multiplied by the channel weight coefficients W_c for each channel to obtain the channel-weighted feature map X_c = X × W_c. The spatial attention branch operation involves: inputting a channel-weighted feature map X_c, performing max pooling and average pooling (both with 1×1 pooling kernels) along the channel dimension to obtain two single-channel feature maps (both with dimensions = H×W×1), and concatenating them into a feature map of H×W×2; compressing the feature map dimension to H×W×1 through a 3×3 convolutional layer (1 kernel, 1 padding), and then generating a spatial weight coefficient matrix (dimension = H×W×1, denoted as W_s) using a sigmoid activation function; multiplying the channel-weighted feature map X_c and the spatial weight coefficient W_s according to their spatial positions to obtain the final attention-enhanced feature map X_sc = X_c × W_s, where the pixel weights in the wearable area are ≥0.7 and the pixel weights in the background area are ≤0.3.

[0024] In one embodiment of the present invention, step S2, constructing an adversarial learning network, includes the following steps: The adversarial learning network adopts a single generator-single discriminator architecture. The generator is a parallel structure of the identity encoder and the wearable encoder, which synchronously outputs facial identity features. Characteristics of the worn item The discriminator employs a two-layer fully connected layer combined with a sigmoid activation structure to determine whether the input features are pure feature pairs without cross-interference. .

[0025] Specifically, the parallel structure is constructed as follows: A pre-processed face image (size 224×224×3, same as the encoder input standard) is received. The same image is synchronously transmitted to the identity encoder and the wearable encoder via an image splitter. The identity encoder outputs 128-dimensional face identity features, and the wearable encoder synchronously outputs 128-dimensional wearable attribute features. The generator then combines the two feature vectors... in front, The subsequent sequential concatenation forms a 256-dimensional feature pair. As input data to the discriminator, the initial values ​​of the parameters of the identity encoder and the wearable encoder in the generator are the same as their optimal parameters after independent training. During the training process, they are updated together with the discriminator parameters to ensure that the feature extraction capability does not degrade. The discriminator is used to judge the input 256-dimensional feature pairs. Whether they are pure feature pairs without cross-interference, i.e. Excluding information on the attributes of the wearable item Without facial identity information, the two fully connected layers consist of: the first fully connected layer with an input dimension of 256 and an output dimension of 128, using LeakyReLU activation function (negative slope = 0.2) to avoid gradient vanishing, and a Dropout layer (dropout probability = 0.5) to improve the discriminator's generalization ability; and the second fully connected layer with an input dimension of 128 and an output dimension of 1, using Sigmoid activation function to map the output value to the [0, 1] interval. The discriminator loss function adopts binary cross-entropy loss, the generator loss function adopts adversarial loss, the generator optimizer adopts AdamW (initial learning rate = 5e-5, weight decay = 1e-5) and connects with the encoder optimizer parameters to avoid parameter oscillation, and the discriminator optimizer adopts Adam (initial learning rate = 1e-4, β1 = 0.5, β2 = 0.999) to adapt to the gradient update requirements of binary classification tasks.

[0026] In one embodiment of the present invention, determining whether the input features are pure feature pairs without cross-interference includes the following steps: The discriminator employs a dual collaborative judgment mechanism embedded within a two-layer fully connected structure combined with a Sigmoid activation structure. This mechanism includes a first judgment stage and a second judgment stage, specifically: The first judgment step is to judge the input. and Perform dimensional compliance checks by extracting dimensions using a pre-set dimensional mask matrix. The first 128 dimensions of features and The full 128-dimensional features are used to calculate the cosine similarity between the two features. If the cosine similarity is less than or equal to the preset similarity threshold, the second judgment is performed; otherwise, the feature pair is directly defined as having cross-interference features. The second judgment step is to determine whether the first judgment step has been passed. and Calculate the variance within each feature and the interference evaluation value. ,in, for The variance value, for The variance value; If the interference evaluation value If the value is less than or equal to the preset interference threshold, then... and Input to the fully connected layer; For feature pairs that pass through the dual collaborative judgment mechanism, after processing by the first fully connected layer and the second fully connected layer, the probability value is output by activating the Sigmoid function. If the probability value is greater than 0.85, it is determined to be a pure feature pair without cross interference.

[0027] Specifically, the preset similarity threshold is determined by experimentally analyzing the cosine similarity distribution of pure feature pairs in the CelebA-Wearing dataset (generally concentrated in the range of 0.1-0.3), and the preset similarity threshold is set to 0.4 (the similarity of 95% of pure feature pairs is ≤0.4). Preset interference threshold setting: Through experimental verification, the interference evaluation value of pure feature pairs is concentrated in the range of 0.1-0.2. The preset interference threshold is set to 0.3 (the preset interference threshold for 90% pure feature pairs is ≤0.3). If the output probability value is greater than 0.85, it is determined to be a pure feature pair without cross-interference; otherwise, it is determined to be a non-pure feature pair with cross-interference.

[0028] In one embodiment of the present invention, step S2, optimizing the parameters of the two encoders through adversarial training, includes the following steps: Construct a composite loss function, specifically as follows: ; ; ; ; in, This represents the total loss value of the composite loss function. , and For the weights of the corresponding loss function, Loss due to loss in preserving identity features The facial identity features of the anchor sample. Facial identity features of positive samples Facial identity features of negative samples For interval parameters, For the loss of classification of wearable attributes, For the sample size, For sample index, For the true attributes of the wearable item, To predict probabilities, To balance the categories, the weights are set inversely based on the number of items worn in each category in the sample set. To combat the losses, Let be the expected function. The discriminator outputs the probability. For the true feature distribution, To add Gaussian noise interference feature distribution, and These are the interfering facial identity features and interfering accessory attribute features after adding Gaussian noise, respectively; The parameters of the two encoders are optimized by iterating through an adaptive estimator until the composite loss function converges.

[0029] Specifically, the parameter optimization process uses the Adaptive Moment Estimator (Adam) optimizer, with an initial learning rate of 1e-4. Every 50 iterations, the learning rate is reduced to 0.5 of the previous iteration. The process continues until the composite loss function converges (the fluctuation of the composite loss function is less than 1e-5 for 10 consecutive iterations, thus completing the parameter optimization of the two encoders). in, , and To determine the weights of the corresponding loss function, before training, 1000 samples were selected for forward computation of the single loss function. The magnitude ranges of the three types of losses were statistically analyzed, and based on the principle of magnitude balance, the weights were set... , and The values ​​are 0.3, 0.4 and 0.3 respectively.

[0030] In one embodiment of the present invention, step S3 involves processing the decoupled wearable attribute features, including the following steps: A multi-scale sliding window mechanism is adopted, with three window sizes set, each with a step size of 8 pixels. The decoupled wearable attribute feature map is then sampled to obtain multi-scale local feature blocks. An adaptive feature selection unit is introduced, which calculates the information entropy of each feature unit and selects feature units whose information entropy is greater than a preset information entropy threshold. After unifying the dimensions of the first convolutional layer, the features of the local area of ​​the wearer are obtained by stitching them together. The preset information entropy threshold is obtained by calculating the average information entropy of all feature units. An improved cascaded convolutional neural network is used. In the first stage, a lightweight convolutional layer is used to locate the entire face region. In the second stage, a bounding box regression branch is introduced to correct the entire face region. In the third stage, six component detection heads are set up to detect the eyes, nose, mouth, eyebrows, cheeks and forehead respectively, and output the bounding box coordinates of each component.

[0031] Specifically, for local feature extraction of the wearable object: a multi-scale sliding window mechanism is adopted, setting three window sizes, including a small window (2 2 pixels), middle window (3) 3 pixels) and large window (4 (4 pixels), during sampling, the feature map is fully scanned from left to right and from top to bottom. If the window edge exceeds the feature map boundary, the boundary pixels are filled with zero padding (the padding value is 0, which does not affect the effective features). The improved cascaded convolutional neural network used in face component detection has the following configuration: First, the network structure is configured as follows: The first stage uses a lightweight convolutional neural network (based on MobileNetV2), containing 3 convolutional layers. The second stage regression branch configuration adds a bounding box regression branch at the output of the first stage network, containing 2 fully connected layers. The first fully connected layer has an input dimension equal to the flattened dimension of the probability map from the first stage, and an output dimension of 128. The second fully connected layer has an output dimension of 4. Bounding box correction is calculated by predicting the offset (Δx1, Δy1, Δx2, Δy2) of the initial bounding box B1 through the regression branch. Here, (x1, y1) represents the pixel coordinates of the top-left vertex of bounding box B1, and (x2, y2) represents the offset of the top-left vertex of bounding box B1. The pixel coordinates of the bottom right corner vertex are given. The corrected bounding box B2 coordinates are (x1+Δx1, y1+Δy1, x2+Δx2, y2+Δy2), where (x1+Δx1, y1+Δy1) are the pixel coordinates of the top left corner vertex of the corrected bounding box B2, and (x2+Δx2, y2+Δy2) are the pixel coordinates of the top right corner vertex of the corrected bounding box B2. The offset values ​​range from [-0.1×(x2-x1), 0.1×(x2-x1)] (to avoid over-correction). If the corrected bounding box B2... If the intersection-union ratio (IoU) of the face region is ≥0.85 (compared with the manually labeled face region), then proceed to the third stage; otherwise, re-execute the first stage of localization. Third stage: Detector configuration: On the face region corrected in the second stage (based on the image region cropped from B2, with a uniform size of 224×224), build 6 independent component detectors, corresponding to the eyes, nose, mouth, eyebrows, cheeks, and forehead, respectively. Input the corrected face region image into the 6 detectors, and each detector outputs the bounding box coordinates of the corresponding component. Use the non-maximum suppression (NMS) algorithm to remove overlapping bounding boxes of the same component (IoU threshold = 0.3), retain the bounding box with the highest confidence, and output the final bounding box coordinates of the 6 key face components.

[0032] In one embodiment of the present invention, step S3, constructing the component relationship model, includes the following steps: The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model G=V,E, where V=v1, v2, ..., v6 represent 6 face component nodes and E is the edge between nodes. Define edge weights ,in, Let be the Euclidean distance between nodes vi and vj. The angle between the line connecting the two nodes and the horizontal direction; Feature learning is performed on an undirected graph model using a graph convolutional network to output 64-dimensional component relationship features; The decoupled facial identity feature map is subjected to a dual pooling operation combining global average pooling and global max pooling to obtain two 128-dimensional feature vectors. These vectors are then fused by attention weighting to obtain 256-dimensional global facial features.

[0033] Specifically, the goal of decoupling is to ensure that facial identity features do not contain any information about what the person is wearing, and that the attribute features of the person wearing do not contain any information that can be used to identify a specific person. Based on the obtained pure features after decoupling, in order to further improve the robustness to delicate wearable objects and different postures, this invention further refines the feature extraction from three dimensions: local, relational and global.

[0034] In one embodiment of the present invention, step S4 includes the following steps: A 3D pose regression network is used to take face image data as input, output face Euler angles, and classify five pose levels based on the Euler angles. A pose-attention mapping table is set up, and an initial attention weight matrix is ​​preset for each pose level. A dynamic weight adjustment module is introduced. With face pose information as input, the weight correction coefficient is output through a multilayer perceptron to correct the initial attention weight matrix. After obtaining the attention weights for pose adaptation, the global face features, local features of the worn object, and component relationship features are multiplied by the corresponding attention weights respectively. Calculate the feature importance scores of the three types of augmented features. Specifically: ; in, For characteristic variance, The maximum value of the variance of the three types of features. , and These are respectively global facial features, local features of the worn object, and features related to the components; Scoring based on feature importance The three types of features are dynamically weighted and fused to obtain the fused features.

[0035] Specifically, a 3D pose regression network is used. Face image data is input, and Euler angles (pitch angle β1, yaw angle β2, roll angle β3) are output. Five pose levels are defined based on the Euler angle values: frontal (β1≤15° and β2≤15°); slight leftward deviation (15°<β2≤30°); slight rightward deviation (-30°≤β2<-15°); slight upward deviation (15°<β1≤≤30°); slight downward deviation (-30°≤β1<-15°); corresponding to levels 1 to 5 respectively. Scoring based on feature importance The three types of features are dynamically weighted and fused to obtain the fused features, specifically: ; in, As a feature of fusion, , and These are respectively global facial features, local features of the worn object, and features related to the components. , and These are the weight coefficients corresponding to the three features. In this embodiment, , and The values ​​were 0.6, 1.0, and 0.4, respectively; The weight matrix dimensions are defined as follows: the attention weight matrices for global facial features, local features of the wearable device, and features related to the device are all column vectors of feature dimension × 1, denoted as W_g, W_l, and W_r, respectively. The initial weight preset rules are as follows: based on the effectiveness of features under different poses (e.g., when the face is turned to the side, the features related to the device are occluded and their weights need to be reduced; when the head is tilted up, the local features of the wearable device are more obvious and their weights need to be increased), the initial weights of each level are preset. The pose-attention mapping table is shown in Table 1:

[0036] In one embodiment of the present invention, step S5, the preset classification and recognition model includes the following steps: A feature adaptation layer combined with a dual-branch classification structure is adopted. The feature adaptation layer is passed through two fully connected layers. The dual-branch classification structure includes a coarse classification branch and a fine classification branch. The coarse classification branch is used to identify the major category of the wearable item, and the fine classification branch is used to identify the sub-category of the wearable item. The fused features are input into the classification and recognition model. After dimensional mapping and feature optimization through the feature adaptation layer, they are input into the dual-branch classification layer. The coarse classification branch outputs the predicted probability of the major category of the wearable item, and selects the category with the highest probability as the coarse classification result. The fine classification branch calls the classifier of the corresponding subclass based on the coarse classification result, outputs the predicted probability of the wearable item subclass, and selects the category with the highest probability as the fine classification result. If the maximum probability of the coarse classification result is less than 0.6, then the secondary recognition mechanism is activated. The local features of the worn object are re-extracted and spliced ​​with the fused features, then input into the classification and recognition model again for recognition, and finally the major category and sub-category information of the worn object are output.

[0037] Specifically, a structure combining a feature adaptation layer and a two-branch classification layer is adopted. The feature adaptation layer maps the fused features to 256-dimensional adaptation features through two fully connected layers (with 512 and 256 hidden neurons respectively). The two-branch classification layer includes a coarse classification branch and a fine classification branch. The coarse classification branch is used to identify the major categories of worn items (such as glasses, masks, hats, and accessories), using the Softmax activation function and cross-entropy loss. The fine classification branch is used to identify the subcategories of worn items (such as myopia glasses, sunglasses, medical masks, and KN95 masks), using a weighted Softmax activation function (the subclass weights are dynamically adjusted according to the proportion of subclass samples within the major category). The specific process of the secondary recognition mechanism is as follows: if the maximum probability of coarse classification is less than the threshold of 0.6, the system backtracks to the feature extraction step of the local area of ​​the wearer, uses more refined parameters (smaller sliding window step size) to re-extract features, and then concatenates the new local features with the original fused features and inputs them again into the dual-branch classifier for decision-making.

[0038] Please see Figure 2 As shown, this invention is a face recognition system based on feature fusion, comprising the following modules: Feature extraction module: Acquires facial image data including the worn object, constructs an identity encoder and an object encoder, and extracts facial identity features and object attribute features respectively by taking the facial image data as input; Feature decoupling optimization module: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. Multi-dimensional feature refinement and extraction module: Processes the decoupled wearable attribute features, extracts local region features of the wearable, and performs facial component detection on facial image data to determine the location information of key facial components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each facial component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; Pose-aware feature fusion module: Constructs a pose-aware attention mechanism model, inputs global facial features, local features of the worn object, and component relationship features into the attention mechanism model, performs pose detection on the facial image data, obtains facial pose information, dynamically adjusts the weight coefficients of global facial features, local features of the worn object, and component relationship features, and performs dynamic weighted fusion of the global facial features, local features of the worn object, and component relationship features to obtain fused features; Wearable item classification and recognition module: The fused features are input into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearable item.

[0039] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A face recognition method based on feature fusion, characterized in that, Includes the following steps: S1: Obtain facial image data containing the worn items, construct an identity encoder and an item encoder, and extract facial identity features and item attribute features respectively by taking the facial image data as input. S2: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. S3: Process the decoupled wearable attribute features, extract local region features of the wearable, and perform face component detection on the face image data to determine the location information of key face components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; S4: Construct a pose-aware attention mechanism model. Input the global facial features, local features of the worn object, and component relationship features into the attention mechanism model. Perform pose detection on the facial image data to obtain facial pose information. Then, dynamically adjust the weight coefficients of the global facial features, local features of the worn object, and component relationship features. Perform dynamic weighted fusion on the global facial features, local features of the worn object, and component relationship features to obtain fused features. S5: Input the fused features into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearer; Step S4 includes the following steps: A 3D pose regression network is used to take face image data as input, output face Euler angles, and classify five pose levels based on the Euler angles. A pose-attention mapping table is set up, and an initial attention weight matrix is ​​preset for each pose level. A dynamic weight adjustment module is introduced. With face pose information as input, the weight correction coefficient is output through a multilayer perceptron to correct the initial attention weight matrix. After obtaining the attention weights for pose adaptation, the global face features, local features of the worn object, and component relationship features are multiplied by the corresponding attention weights respectively. Calculate the feature importance scores of the three types of augmented features. ; Scoring based on feature importance The three types of features are dynamically weighted and fused to obtain the fused features.

2. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S1, constructing the identity encoder and the wearable encoder includes the following steps: The identity encoder employs a lightweight convolutional neural network structure, comprising: one input layer, four feature extraction units, and one fully connected layer. Each feature extraction unit is a concatenated structure of a convolutional layer, a batch normalization layer, a ReLU activation function, and a max-pooling layer. The convolutional layers uniformly employ 3... The convolutional kernel is 3 pixels in size, with a fixed stride of 1. The first and third feature extraction units contain only 1 convolutional layer, while the second and fourth feature extraction units contain 2 convolutional layers. The fully connected layer flattens the feature map output by the fourth feature extraction unit into a one-dimensional vector and outputs the final face identity feature vector. The wearable encoder is based on the ResNet-18 framework. After each residual block in its third and fourth stages, a channel-spatial dual attention module is embedded. The module calculates the weight coefficients of each channel in the feature map output by the residual block through the channel attention branch, and then inputs the channel-weighted feature map into the spatial attention branch to locate the spatial region of the wearable on the feature map. The pixels in the region are assigned high weights and the background region is assigned low weights. Finally, the wearable attribute feature vector is output.

3. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S2, constructing the adversarial learning network includes the following steps: The adversarial learning network adopts a single generator-single discriminator architecture. The generator is a parallel structure of the identity encoder and the wearable encoder, which synchronously outputs facial identity features. Characteristics of the worn item The discriminator employs a two-layer fully connected layer combined with a sigmoid activation structure to determine whether the input features are pure feature pairs without cross-interference. .

4. The face accessory recognition method based on feature fusion according to claim 3, characterized in that, The process of determining whether the input features are pure feature pairs without cross-interference includes the following steps: The discriminator employs a dual collaborative judgment mechanism embedded within a two-layer fully connected structure combined with a Sigmoid activation structure. This mechanism includes a first judgment stage and a second judgment stage, specifically: The first judgment step is to judge the input. and Perform dimensional compliance checks by extracting dimensions using a pre-set dimensional mask matrix. The first 128 dimensions of features and The full 128-dimensional features are used to calculate the cosine similarity between the two features. If the cosine similarity is less than or equal to the preset similarity threshold, the second judgment is performed; otherwise, the feature pair is directly defined as having cross-interference features. The second judgment step is to determine whether the first judgment step has been passed. and Calculate the variance within each feature and the interference evaluation value. ,in, for The variance value, for The variance value; If the interference evaluation value If the value is less than or equal to the preset interference threshold, then... and Input to the fully connected layer; For feature pairs that pass through the dual collaborative judgment mechanism, after processing by the first fully connected layer and the second fully connected layer, the probability value is output by activating the Sigmoid function. If the probability value is greater than 0.85, it is determined to be a pure feature pair without cross interference.

5. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S2, the parameters of the two encoders are optimized through adversarial training, including the following steps: Construct a composite loss function, specifically as follows: ； ； ； ； in, This represents the total loss value of the composite loss function. , and For the weights of the corresponding loss function, Loss due to loss in preserving identity features The facial identity features of the anchor sample. Facial identity features of positive samples Facial identity features of negative samples For interval parameters, For the loss of classification of wearable attributes, For the sample size, For sample index, For the true attributes of the wearable item, To predict probabilities, To balance the categories, the weights are set inversely based on the number of items worn in each category in the sample set. To combat the losses, Let be the expected function. The discriminator outputs the probability. For the true feature distribution, To add Gaussian noise interference feature distribution, and These are the interfering facial identity features and interfering accessory attribute features after adding Gaussian noise, respectively; The parameters of the two encoders are optimized by iterating through an adaptive estimator until the composite loss function converges.

6. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S3, the decoupled wearable attribute features are processed, including the following steps: A multi-scale sliding window mechanism is adopted, with three window sizes set, each with a step size of 8 pixels. The decoupled wearable attribute feature map is then sampled to obtain multi-scale local feature blocks. An adaptive feature selection unit is introduced, which calculates the information entropy of each feature unit and selects feature units whose information entropy is greater than a preset information entropy threshold. After unifying the dimensions of the first convolutional layer, the features of the local area of ​​the wearer are obtained by stitching them together. The preset information entropy threshold is obtained by calculating the average information entropy of all feature units. An improved cascaded convolutional neural network is used. In the first stage, a lightweight convolutional layer is used to locate the entire face region. In the second stage, a bounding box regression branch is introduced to correct the entire face region. In the third stage, six component detection heads are set up to detect the eyes, nose, mouth, eyebrows, cheeks and forehead respectively, and output the bounding box coordinates of each component.

7. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S3, the component relationship model is constructed, including the following steps: The component relationship model is constructed by using the center coordinates of the bounding box of each face component as nodes to build an undirected graph model G=V,E, where V=v1, v2, ..., v6 represent 6 face component nodes and E is the edge between nodes. Define edge weights ,in, Let be the Euclidean distance between nodes vi and vj. The angle between the line connecting the two nodes and the horizontal direction; Feature learning is performed on an undirected graph model using a graph convolutional network to output 64-dimensional component relationship features; The decoupled facial identity feature map is subjected to a dual pooling operation combining global average pooling and global max pooling to obtain two 128-dimensional feature vectors. These vectors are then fused by attention weighting to obtain 256-dimensional global facial features.

8. The face accessory recognition method based on feature fusion according to claim 1, characterized in that, In step S5, the preset classification and recognition model includes the following steps: A feature adaptation layer combined with a dual-branch classification structure is adopted. The feature adaptation layer is passed through two fully connected layers. The dual-branch classification structure includes a coarse classification branch and a fine classification branch. The coarse classification branch is used to identify the major category of the wearable item, and the fine classification branch is used to identify the sub-category of the wearable item. The fused features are input into the classification and recognition model. After dimensional mapping and feature optimization through the feature adaptation layer, they are input into the dual-branch classification layer. The coarse classification branch outputs the predicted probability of the major category of the wearable item, and selects the category with the highest probability as the coarse classification result. The fine classification branch calls the classifier of the corresponding subclass based on the coarse classification result, outputs the predicted probability of the wearable item subclass, and selects the category with the highest probability as the fine classification result. If the maximum probability of the coarse classification result is less than 0.6, a secondary recognition mechanism is activated to re-extract the local features of the wearer and splice them with the fused features. The results are then input into the classification recognition model again for recognition, and finally the major category and sub-category information of the wearer are output.

9. A face recognition system based on feature fusion, used to implement the face recognition method based on feature fusion according to any one of claims 1-8, characterized in that, Includes the following modules: Feature extraction module: Acquires facial image data including the worn object, constructs an identity encoder and an object encoder, and extracts facial identity features and object attribute features respectively by taking the facial image data as input; Feature decoupling optimization module: Construct an adversarial learning network. The adversarial learning network adopts a single generator-single discriminator architecture. The facial identity features and the attribute features of the worn items are input into the adversarial learning network. The discriminator adopts a dual collaborative judgment mechanism in a 2-layer fully connected layer combined with a Sigmoid activation structure. The facial identity features and the attribute features of the worn items are decoupled through its internal preset adversarial mechanism to obtain the decoupled facial identity features and attribute features of the worn items. Multi-dimensional feature refinement and extraction module: Processes the decoupled wearable attribute features, extracts local region features of the wearable, and performs facial component detection on facial image data to determine the location information of key facial components in order to construct a component relationship model. The component relationship model is constructed by using the center coordinates of the bounding box of each facial component as nodes to build an undirected graph model. The undirected graph model is then used to learn features through a graph convolutional network to output 64-dimensional component relationship features. After extracting component relationship features from the component relationship model, global feature extraction is performed on the decoupled face identity features to obtain global face features; Pose-aware feature fusion module: Constructs a pose-aware attention mechanism model, inputs global facial features, local features of the worn object, and component relationship features into the attention mechanism model, performs pose detection on the facial image data, obtains facial pose information, dynamically adjusts the weight coefficients of global facial features, local features of the worn object, and component relationship features, and performs dynamic weighted fusion of the global facial features, local features of the worn object, and component relationship features to obtain fused features; Wearable item classification and recognition module: The fused features are input into a preset classification and recognition model for analysis and processing, and finally output the category information of the wearable item.

Images