Face wrinkle detection method and system based on double-branch feature fusion and multi-input attention mechanism

Abstract

The invention discloses a face wrinkle detection method and system based on dual-branch feature fusion and a multi-input attention mechanism. The method comprises the following steps: dual-branch feature extraction: extracting local texture features through a ResNet branch, and modeling global context dependence through a Swin Transform branch; the feature fusion module adopts 1 * 1 convolution to align channel dimensions (ResNet branch compression ratio is 4: 1, and Swin branch compression ratio is 1.5: 1) output by double branches, and feature fusion is realized through splicing or adding; a multi-input channel attention mechanism: performing multi-scale convolution on the fusion features to generate three paths of inputs, adding the three paths of inputs, performing compression-excitation operation to generate three groups of independent channel weights, and performing weighted fusion to output enhanced features; and the classification module is used for outputting the fine-grained wrinkle category probability. The system comprises a processor and a memory and is used for implementing the method. According to the method, the wrinkle detection precision is remarkably improved, and the method is superior to a mainstream model in fine-grained classification tasks such as nasolabial folds and crow's feet, and is suitable for the fields of medical beauty evaluation and skin analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a double-branch network architecture combining Swin Transformer and ResNet, a method and system for high-precision facial wrinkle detection through multi-input channel attention mechanism, and is especially suitable for facial skin quality analysis, medical beauty evaluation and other fields. BACKGROUND

[0002] Due to various factors such as environment, living habits, and genetics, the skin quality of each person is unique. Skin quality detection technology can accurately identify skin problems such as wrinkle type and inflammation type. With key indicators, targeted solutions can be proposed, and suitable skin care products or skin care drugs can be selected to avoid misuse. Traditional methods based on convolutional neural networks (CNN), such as ResNet, perform well in local texture feature extraction, but their limited receptive fields make it difficult to effectively model long-range dependencies and global context information of wrinkle distribution. While models based on pure Transformers (such as Vision Transformer) have strong global modeling capabilities, they have high computational complexity and may ignore local detail features that are crucial for wrinkle detection.

[0003] In view of the shortcomings of existing methods, the present application proposes a facial wrinkle detection method and system based on double-branch feature fusion and multi-input attention mechanism. This method can effectively capture local micro-texture and global distribution patterns of wrinkles through double-branch feature fusion and attention mechanism, and further achieve accurate detection of wrinkle categories through training. SUMMARY

[0004] In view of the shortcomings of existing wrinkle detection methods in feature fusion sufficiency, multi-source information utilization, and fine-grained detection accuracy, the present application proposes an innovative facial wrinkle detection method and system based on double-branch feature fusion and multi-input attention mechanism. This method significantly improves the accuracy and robustness of wrinkle detection through carefully designed double-branch feature extraction, feature alignment fusion, and multi-input channel attention mechanism.

[0005] The technical solutions adopted by the present application are as follows:

[0006] A facial wrinkle detection method based on double-branch feature fusion and multi-input attention mechanism, comprising the following steps:

[0007] S1: Preprocessing of human face image: Obtain the human face RGB image to be detected. Perform preprocessing operations on the input image, including but not limited to face detection and alignment, illumination normalization, image size standardization (preferably 224x224 pixels), and optional data augmentation operations (such as random rotation, flipping, cropping), to obtain preprocessed image data.

[0008] S2: For the preprocessed image in step S1, perform wrinkle labeling by labelme, establish a data set, and divide it into a training set, a validation set, and a test set.

[0009] S3: Construct and train a wrinkle detection model: Construct a neural network model (hereinafter referred to as RSTE-Net model) containing a double-branch feature extraction module, a feature fusion module, a multi-input channel attention module, and a classification module, and train it using the training data set labeled with wrinkle category information. The specific modules are as follows:

[0010] A double-branch feature extraction module:

[0011] First, ResNet branch feature extraction based on CNN skeleton network: input the preprocessed image into the ResNet branch. This branch successively contains an initial convolution layer (preferably 7x7 convolution), a batch normalization layer, an activation function (preferably ReLU), a pooling layer (preferably maximum pooling), and multiple stacked residual blocks (preferably ResNet-50 conv2_x to conv5_x stages). Finally, output the first high-level feature map (preferably a feature map with a final output size of 7x7xC1 in the ResNet architecture, C1 being the number of channels, such as 2048).

[0012] Second, Swin Transformer branch feature extraction: input the preprocessed image into the Swin Transformer branch. This branch successively contains image partition (Patch Partition, preferably 4x4 non-overlapping blocks), linear embedding, and a hierarchical structure composed of multiple Swin Transformer blocks containing local window attention mechanism (W-MSA) and shift window multi-head self-attention (SW-MSA). Finally, output the second high-level feature map (preferably with a size of 7x7xC2, C2 being the number of channels, such as 768).

[0013] The linear embedding layer serves as an initial feature extractor, converting all pixel values of each image block into a high-dimensional feature vector, compressing and encoding spatial information into feature channel dimension C, and uniformly converting the input data format into a sequence form that can be directly processed by the Swin Transformer.

[0014] The image patching of the Swin Transformer architecture utilizes a Patch Merging module to divide the image into 4x4 non-overlapping patches; a local window attention mechanism W-MSA with a window size of 7x7; and a Swin Transformer architecture for image cross-window interaction.

[0015] B Feature Fusion Module: receives the first high-level feature map from the ResNet branch and the second high-level feature map from the Swin Transformer branch. Since the spatial dimensions of the two feature maps are the same (preferably 7x7) but the number of channels is different (C1≠C2), this module first performs feature alignment, applying a 1x1 convolution layer to each of the two feature maps to adjust the number of channels, so that the final number of channels is consistent (preferably, the ResNet branch adjusts the number of channels with a compression ratio of 4:1, and the Transformer branch adjusts the number of channels with a compression ratio of 1.5:1, so that the output number of channels of the two is the same). Perform fusion operations (preferably Concat or element-wise addition (Add)) on the two compressed and aligned feature maps to obtain a fused feature map.

[0016] C Multi-Input Channel Attention Module (Improved SENets Module): Process the feature fusion map with three convolution layers of different kernel sizes in sequence to generate three feature maps of different scales, add the three feature maps of different scales to fuse multi-scale information. Then perform compression operation, perform global pooling operation (preferably global average pooling GAP) on the fused feature map to generate a channel descriptor vector (dimension 1x1xC). Then perform excitation operation, use two fully connected layers to build a model of channel correlation, increase the dimension of the second fully connected layer to 3C, C refers to the dimension of the feature vector, that is, the feature channel dimension C mentioned in the double-branch feature extraction module above, and then divide the obtained feature vector into three equal parts, each part is processed by the Sigmoid function. Generate their corresponding weight representations. Finally, perform weighted processing on the three scale information according to the weights, and add the weighted results to obtain the final output of the entire module.

[0017] D Classification Module: Flatten the feature map enhanced by the multi-input channel attention module, input into the fully connected layer, perform nonlinear transformation and dimension reduction, and finally output the prediction probability distribution of the wrinkle class through the output layer.

[0018] S4 Model Training: Train the RSTE-Net model constructed using the annotated wrinkle dataset (training set and validation set), and evaluate the model performance using the test set.

[0019] S4 Preprocessing of Images to be Detected: Perform the same preprocessing operations on the face images to be detected as in S1 to obtain image data that meets the input requirements of the model.

[0020] S5 Wrinkle detection: input the preprocessed image to be detected into the trained RSTE-Net model. The model automatically analyzes the image and outputs fine-grained wrinkle detection results, including whether there are wrinkles and the wrinkle category to which they belong.

[0021] The second aspect of the application relates to a wrinkle detection system based on dual-branch fusion and multi-input channel attention, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the wrinkle detection method based on dual-branch fusion and multi-input channel attention proposed in the application.

[0022] The technical concept of the application is as follows: first, a wrinkle detection neural network model (RSTE-Net) containing dual branches (ResNet branch and Swin Transformer branch) is constructed. ResNet focuses on extracting local detailed texture features of wrinkles; the Swin Transformer branch focuses on modeling global context information and long-range dependencies of wrinkle distribution. Then, the high-level feature maps extracted by the two branches are spatially aligned and adjusted in channel number before being fused to combine local and global information. Next, the fused features are input into an improved multi-input channel attention module (SENets). The module constructs multiple inputs through multiple convolutional layers of different scales, learns the inter-channel dependencies through compression-excitation operations after fusing multi-scale information, and generates multiple independent channel weight vectors to respectively recalibrate each input feature. Finally, the enhanced features are fused and output. This mechanism can fully utilize complementary information from different scales and different branches, adaptively strengthen feature channels that are crucial to wrinkle detection, and suppress irrelevant or interfering channels. Finally, the enhanced features are input into a classification module to output wrinkle category prediction results. The trained model can perform high-precision wrinkle detection and classification on input human face images.

[0023] The beneficial effects of the application mainly include:

[0024] 1. The application creatively combines the advantages of ResNet and Swin Transformer dual-branch architecture. The ResNet branch effectively captures local detailed texture and edge information of wrinkles, and the Swin Transformer branch efficiently models global context and long-range dependencies of wrinkle distribution. Through a feature fusion module, the advantages of the two branches are complementary, providing a more comprehensive and richer feature basis for subsequent fine-grained detection.

[0025] 2.The application proposes an innovative multi-input channel attention mechanism (improved SENets module). The module effectively fuses multi-scale information by constructing multiple input branches of different scales, overcoming the limitations of single input information. The core is to generate independent channel weight vectors for each input branch to perform feature re-labeling, and finally fuse the output.

[0026] 3.The application effectively solves the problem of inconsistent sizes of double-branch output feature maps through channel compression (1x1 convolution) operation in the feature fusion module, ensuring the accuracy and effectiveness of the fusion process.

[0027] 4.The RSTE-Net model proposed in the application is fully verified on a self-built high-quality wrinkle dataset, and significantly outperforms existing mainstream methods (such as CNN, ViT, MobileNetV3, DermaNet) in terms of precision, recall, F1 score, and improved mIoU. Especially in the detection of key wrinkle types such as command lines, fish tail lines, and tear grooves, and micro-fine wrinkles, the method shows superior performance, proving the effectiveness and advancement of the method in the task of fine-grained facial wrinkle detection. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of the method proposed in the application;

[0029] Figure 2 is a schematic diagram of the improved multi-input SENets channel attention module in the method proposed in the application;

[0030] Figure 3 is a structural schematic diagram of the RSTE-Net neural network model proposed in the application;

[0031] Figure 4 is an mIoU comparison chart of different models in various types of wrinkles in the embodiment of the application. DETAILED DESCRIPTION

[0032] Embodiment 1

[0033] Referring to Figure 1 and Figure 3 , the wrinkle detection method based on double-branch fusion and multi-input channel attention includes the following steps:

[0034] S1 face image preprocessing:

[0035] The unified shooting is performed by a face collection device under daily light, and a total of 1500 pictures are obtained. The RGB image of the face to be detected is acquired, and face detection and alignment are first performed to make the two eyes in the center and the face centered. Then, the light normalization processing is performed, and the histogram equalization is used to reduce the influence of light difference. Then, the image size table is converted to the preset input size (224x224 pixels). Optionally, data enhancement strategies such as random horizontal flip (probability 0.5), random rotation (±15 degrees), random crop scaling (scale 0.8-1.0) are applied in the training stage to increase data diversity and improve model generalization ability. Finally, the preprocessed image data 224x224x3 is obtained.

[0036] For the above preprocessed image, wrinkle labeling is performed by labelme, and the wrinkle contains seven types of wrinkle labeling, including fish tail wrinkle, command wrinkle, tear groove, corner wrinkle, lifting wrinkle, under-eye fine lines and nose wrinkle. A data set is established, and the data set is divided into a training set, a verification set and a test set according to a ratio of 7:1.5:1.5.

[0037] S2, a wrinkle detection model is constructed and trained.

[0038] A dual-branch feature extraction module:

[0039] A1 ResNet-50 branch: after the image is input into the ResNet-50 branch, 7x7 convolution, batch normalization and ReLU activation operations are performed on the image, and 112x112x64 feature maps are output. Then, 3x3 pooling operation is performed through the maximum pooling layer, and 56x56x64 feature maps are output. Finally, residual block stacking is performed, and the residual block stacking is the conv2_x to conv5_x stage of ResNet-50, and 7x7x2048 feature maps are output.

[0040] A2 Swin Transformer branch: the image is divided into 4x4 non-overlapping blocks through patch division, and 56x56x48 feature maps are output. Then, through the local window attention mechanism W-MSA, the window size is 7x7, and 56x56x96 feature maps are output. Then, the Swin Transformer architecture interacts through the cross-window, and the adjacent 2x2 image blocks are spliced and the channels are compressed through the patch merging module, and 28x28x192 feature maps are output. Finally, the global relationship modeling (Swim Transformer through the application of sliding window, focusing on the relationship of global context) is performed, and 7x7x768 feature maps are output.

[0041] B Feature fusion module: ResNet outputs a 7x7x2048 feature map, and Swin Transformer outputs a 7x7x768 feature map. The sizes are the same, but the channel numbers are different. A 1x1 convolution is used for channel dimension compression. The compression ratio of the ResNet path is 4:1, and the compression ratio of the Swin-Transformer path is 1.5:1. Finally, the channel numbers are consistent to facilitate fusion.

[0042] C Multi-input channel attention module (improved SENets, the principle diagram is shown in Figure 2 ): To extract multi-scale feature information, three convolutional layers can be used in turn, and different convolution kernel sizes are set for each convolutional layer. The convolution kernel sizes of the three convolutional layers are 7x7, 5x5, and 3x3, respectively. After processing by these three convolutional layers, three different inputs are generated. Then, the three inputs are added to realize the fusion of multi-scale information. Next, global pooling is used to process the fused features to obtain channel descriptors, which corresponds to the compression operation. In the excitation stage, two fully connected layers are used to construct a model of the correlation between channels. Since there are three inputs, it is not feasible to directly restore the channel number to C in the second fully connected layer. Therefore, the dimension needs to be raised to 3C in the second fully connected layer. Then, the obtained feature vector is evenly divided into three parts, and each part is processed by a Sigmoid function to generate its own weight representation. Finally, the three types of information are weighted and processed according to the weights, and the weighted results are added to obtain the final output of the entire module.

[0043] D Classification module: The feature map enhanced by the multi-input channel attention module is flattened and input into a fully connected layer for nonlinear transformation and dimension reduction. Finally, the output layer outputs the prediction probability distribution of the wrinkle class.

[0044] S3 Model training: The RSTE-Net model constructed in step S2 is trained using the wrinkle dataset (training set and validation set) labeled in step S1, and the test set is used to evaluate the model performance.

[0045] S4 Preprocessing of the image to be detected: The same preprocessing steps as S1 are performed on the new face image to be detected: face detection and alignment, illumination normalization, and size standardization to 224x224 pixels.

[0046] S5 Wrinkle detection: The test image is input into the trained RSTE-Net model. The model goes through double-branch feature extraction, feature fusion, multi-input channel attention weighting, classification, and other steps, and finally outputs a prediction vector. The class with the highest probability is taken as the detected wrinkle class. All class probabilities can also be output as a fine-grained detection result.

[0047] The mIoU chart of training the RSTE-Net model (the model is referred to as RT-SE) by using the wrinkle data set (training set and validation set) labeled in step S1 in embodiment 1 in step S3 is as shown in the following table. Figure 4 Meanwhile, multiple models are set for comparative experiments, including CNN, Vision Transformer, MobileNetV3, Derma Net, and a model adopting a standard SENet single-channel attention module. The models are constructed and trained by using the wrinkle data set (training set and validation set) labeled in step S1 in embodiment 1, and the experimental results are as shown in the following table. Figure 4

[0048] Embodiment 2

[0049] The embodiment provides a wrinkle detection system based on double-branch fusion and multi-input channel attention, including a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the wrinkle detection method based on double-branch fusion and multi-input channel attention in embodiment 1. The system can be a server, a workstation, an embedded device, or a terminal device equipped with corresponding processing capability.

[0050] The above only describes the preferred embodiments of the present application, and does not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art, such as replacing different CNN skeletons (such as VGG, DenseNet), using other types of Transformers (such as PVT, Twins), adjusting the number or size of convolution kernels in the multi-input channel attention module, changing the feature fusion method (such as weighted addition, gate fusion) or the structure of the classification module, etc., should fall within the protection scope determined by the claims of the present application.​

Claims

1. A facial wrinkle detection method based on dual-branch feature fusion and multi-input attention mechanism, characterized in that: include: S1 Face Image Preprocessing: Use a face acquisition device to uniformly shoot under normal lighting conditions, obtain RGB face images, and perform preprocessing, including face detection and alignment, illumination normalization, and size standardization to a predetermined size. S2: For the images preprocessed in step S1, wrinkle annotation is performed using LabelMe to establish a dataset, which is divided into a training set, a validation set, and a test set; S3 builds and trains the wrinkle detection model, namely the RSTE-Net model, which includes: A. Dual-branch feature extraction module: The first branch extracts local texture features based on the CNN skeleton network and outputs the first high-level feature map; The second branch extracts global context features based on the Transformer skeleton network and outputs the second high-level feature map; B. Feature fusion module: aligns the number of channels of the two feature maps through 1×1 convolution and performs a fusion operation to generate a fused feature map; C. Multi-input channel attention module: Multi-scale convolution is performed on the fused feature map to generate three-way features. After addition and fusion, a compression-excitation operation is performed to generate three sets of channel weights. The weighted fusion output is enhanced features. D. Classification module: Flattens the enhanced features and outputs wrinkle category probabilities through a fully connected layer; S4: Use the training set and validation set of step S2 to train the RSTE-Net model described in step S3, and use the test set to evaluate the model performance; S5: perform preprocessing of S1 on the face image to be detected; S6: Input the preprocessed image into the trained RSTE-Net model and output the wrinkle detection results and categories.

2. The method according to claim 1, wherein: The first branch adopts the ResNet architecture, which includes the initial convolution layer, batch normalization layer, activation function, pooling layer and residual block stacking in sequence; The second branch adopts the Swin Transformer architecture, which includes image segmentation, linear embedding and Swin Transformer block hierarchy in sequence; the linear embedding layer serves as the initial feature extractor, converts all pixel values ​​of each image block into a high-dimensional feature vector, compresses and encodes the spatial information into the feature channel dimension C, unifies the input data format and converts it into a sequence form that can be directly processed by Swin Transformer; the Swin Transformer block hierarchy includes the local window attention mechanism W-MSA and the shifted window multi-head self-attention SW-MSA.

3. The method according to claim 2, wherein: The initial convolutional layer of the ResNet architecture uses 7×7 convolution, the activation function is the ReLU function, and the residual block stack is the conv2_x to conv5_x stages of ResNet-50. The ResNet architecture finally outputs a feature map of size 7×7×C1, where C1 is the number of channels; The image segmentation of the Swin Transformer architecture uses the Patch Merging module to divide the image into 4×4 non-overlapping blocks; the local window attention mechanism W-MSA, the window size is set to 7×7; the Swin Transformer architecture interacts with the image across windows, and finally outputs a feature map of size 7×7×C2, where C2 is the number of channels.

4. The method according to claim 1, wherein: In the feature fusion module, the channel alignment operation satisfies: the number of channels of the first branch feature map is adjusted according to a compression ratio of 4:1; the number of channels of the second branch feature map is adjusted according to a compression ratio of 1.5:1, and finally the number of channels of the two are made consistent for fusion; The fusion operation uses the concatenation method or the element-by-element addition method.

5. The method according to claim 1, wherein: The multi-input channel attention module performs the following sub-steps: (a) Three convolutional layers with different kernel sizes are used to process the input features and generate three-way multi-scale features; (b) Add the three multi-scale features to achieve the fusion of multi-scale information; (c) Perform global average pooling on the fused features to generate channel descriptors; (d) Processing the channel descriptor through two fully connected layers, the output dimension of the second fully connected layer is expanded to 3C, where C refers to the dimension of the feature vector; (e) Divide the 3C-dimensional vector into three equal parts and generate three sets of channel weights through the Sigmoid function; (f) The three multi-scale features are channel-weighted and then added together for output.

6. The method according to claim 1, wherein: The preprocessing of step S1 also includes a data enhancement operation, and the data enhancement operation includes at least one of random horizontal flipping, ±15 degree rotation, and random cropping and scaling at a ratio of 0.8-1.

0.

7. The method according to claim 1, wherein: The size is standardized to 224×224 pixels, and illumination normalization uses histogram equalization.

8. The method according to claim 1, wherein: The wrinkle categories output by the classification module include at least multiple fine-grained types of crow's feet, nasolabial folds, and tear grooves.

9. A wrinkle detection system based on dual-branch fusion and multi-input channel attention, characterized in that: include: a memory storing executable code; One or more processors, when executing the executable code, implement the method according to any one of claims 1 to 8.

Images