Multi-granularity face feature aggregation method based on attention mechanism
Through the multi-grained face feature aggregation method based on attention mechanism, the feature fusion problem under complex lighting and expression changes in the prior art is solved, and a three-dimensional face reconstruction with high accuracy and robustness is achieved.
Patent Information
- Application Number
- CN202311294393.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-09
- Publication Date
- 2025-04-11
AI Technical Summary
The existing three-dimensional face reconstruction technology is not effective in dealing with complex lighting scenes and facial expression changes, and it is difficult to effectively integrate feature information of different granularity, resulting in feature loss and insufficient accuracy.
A multi-grained face feature aggregation method based on attention mechanism is adopted, and multi-grained feature aggregation is performed by adjusting the backbone network, combining deep hierarchical features and shallow features, and a progressive training is used to learn complementary information between different image granularities.
It realizes richer feature information extraction, enhances the robustness and anti-overfitting ability of the network, and improves the accuracy and stability of face reconstruction.
Smart Images
Figure CN119810877A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and mainly to the field of face reconstruction. Specifically, the present invention provides a multi-granularity face feature aggregation method based on an attention mechanism. Background Art
[0002] Faces and facial expressions are important information carriers in the process of communication between people, and can convey rich emotions more directly than language. Early research mainly focused on two-dimensional faces, such as face recognition, face verification, face key point detection, expression recognition and other research directions. With the improvement of computing power and the advancement of algorithms, two-dimensional face features are increasingly unable to meet the requirements of many applications. Compared with two-dimensional faces, three-dimensional face reconstruction based on 2D images refers to the reconstruction of a three-dimensional model of the face, that is, it has one more dimension compared to a two-dimensional face, has richer features, can express richer and more realistic information, and has higher accuracy and robustness in areas such as face recognition.
[0003] At present, scholars at home and abroad have proposed many methods to solve the problem of 3D face reconstruction. However, when 2D face images are captured unconstrained in the wild, they are easily affected by factors such as viewpoint changes, lighting changes, and facial expression changes. Previous 3D face reconstruction mainly revolves around methods based on shape from shading (SFS), 3DMM-based methods, and convolutional neural network-based methods. Shape from shading mainly uses gradient shadow changes to reconstruct 3D faces. Due to these strict restrictions, SFS is limited in efficiency and accuracy. In order to alleviate the limitations, many works have introduced additional constraints such as smoothness, intensity gradients, and different types of prior domain knowledge to expand the original paper. Some researchers have proposed embedding surface normals in the SFS framework to replace the statistical model of surface depth, mainly using the correlation between surface direction and image intensity to recover the surface normal field from a single intensity image. SFS is suitable for uniform concave or convex objects, but it will fail when it involves overly complex concave or convex surfaces, because slight changes in surface normals will lead to significant changes in the corresponding integrated surface. The SFS-based method cannot handle 3D reconstruction under complex lighting scenes and is only applicable to frontal facial images. Therefore, over the years, 3D face reconstruction based on SFS has become less and less common in real scenes. In order to overcome the limitations of SFS, researchers proposed a 3D deformable reconstruction method 3DMM in 1999. The core idea is that faces can be matched one-to-one in 3D space and can be obtained by weighted linear addition of multiple other faces by orthogonal basis. The deformable face model comes from a prototype 3D scan dataset of faces. It helps in the two main steps of face manipulation: (1) deriving a 3D face model from a novel image and (2) modifying the shape and texture in a natural way. In a sense, 3D face modeling based on deformable models is based on typical face images in a certain class of face images to form a corresponding type space, and the linear combination of the space basis is used to realize the 3D representation of the target face belonging to this class of face images. In addition to the 3DMM-based methods, many scholars have focused on learning the mapping function from the image space to the 3DMM parameter space. For example, the coupled radial basis function network is a nonlinear subspace modeling technology. Given paired 2D and 3D data, the network learns their intrinsic representations and corresponding nonlinear mapping functions. When a 2D face image is given, its 2D intrinsic representation is first calculated, and then the linear combination of its k nearest neighbors is used to approximate the representation. After that, the same coefficients are used to obtain the intrinsic representation of the 3D face and the corresponding 3D face model. At the same time, some researchers have proposed using a deep convolutional network to approximate the mapping function from a 2D image to a 3D model as perspective projection and 3D deformation model parameters. In their framework, multiple deep regression networks are stacked to regress and update perspective projection parameters and 3DMM parameters.However, there is a limitation that in addition to 2D facial images, additional features are often required to be input into deep neural networks.
[0004] Considering that the feature information extracted at different levels of the neural network will be quite different, the features extracted by the shallow network contain a lot of detailed information of the image content, with strong geometric information representation ability and weak semantic information representation ability, while the deep network contains a lot of high-level semantic features and relatively weak geometric information representation ability. And as the network structure continues to deepen, each layer will lose some information, so when the feature reaches the last layer, it only contains a small amount of information. For the reconstruction of the face, not only the overall structure and special details of the face are needed, that is, it is not enough to just identify the distinguishing parts, but also how these parts interact with each other in a complementary way. Conventional feature networks often focus on the amplification of features, that is, focusing on distinguishing features, ignoring other parts, and not considering how to fuse the features of different amplified parts in a synergistic way. Therefore, when designing the network, it is necessary not only to further identify the components and the most discriminative granularity, but also to simultaneously identify how to effectively merge components of different granularities. To solve the problem raised, the present invention modifies the network structure to achieve deep aggregation of features. During training, the method starts from a more stable fine-grained granularity and gradually transitions to a coarser granularity, continuously mining information of a larger granularity so that the network can retain more effective feature information. This is a multi-granularity face feature aggregation method based on an attention mechanism. Summary of the invention
[0005] The present invention aims to make up for the deficiencies in the prior art in feature aggregation and multi-granularity feature fusion research, and proposes a multi-granularity face feature aggregation method based on the attention mechanism with scientific thinking and rigorous attitude.
[0006] The present invention aggregates features of different granularities in the network, avoids the loss of facial features, extracts richer features while ensuring the introduction of contextual information, and thus proposes a high-precision face reconstruction method with good generalization ability, excellent and stable effect.
[0007] The technical solution of the present invention is as follows:
[0008] A multi-granularity face feature aggregation method based on attention mechanism, the key point is to perform multi-granularity feature aggregation on image information based on an adjusted backbone network. The network part includes designing an efficient upsampling encoder to extract features for preliminary feature information processing, and deep aggregation of specific granularity in the backbone network to achieve semantic and spatial fusion. During the network training process, multi-granularity progressive training is performed to learn the complementary information between different image granularities.
[0009] The shallow features of face images mainly include contours, edges, colors, textures, and shapes that can reflect the features of the image content. The shallow features have less semantic information, but the target position is accurate; the deep features of the image are mainly semantic features that are understandable by computers, but the target position is relatively rough. For face images, they are mainly divided into coarse-grained, medium-grained, and fine-grained for training, starting from the more stable fine-grained and gradually transitioning to coarser granularity, avoiding the confusion caused by differences within large categories in large areas. It is equivalent to the network focusing on a local area, and then shrinking the larger patch around this area until the entire image is completed. Specifically, at the end of each training step, the parameters of the current training step are passed to the next step as parameter initialization. This transfer enables the network to mine larger granularity information based on the areas learned in the previous training steps, and finally connect the features extracted in all stages.
[0010] The specific steps include:
[0011] First, the face dataset is processed initially, using sub-pixel convolution for simple upsampling to generate a feature map of a specific size;
[0012] Then the neural network structure is optimized, and the backbone is modified to Deep Layer Aggregation, or DLA. Deep aggregation mainly has 6 stages, and the first two stages are composed of convolutional layers. The next four stages are hierarchical structures linked by tree structures, which can better propagate features and gradients. In each stage, the HDA structure is used to fuse the internal features of the stage, and the aggregation root node is shared between stages. Each stage uses max pooling with a kernel size of 2 and a step size of 2 for downsampling. Considering that facial feature information is relatively concentrated, the simam attention mechanism is introduced in the first stage. The output features of each encoder block are continuously corrected through the spatial domain suppression effect. Through multi-granular information propagation, the model parameters are reduced while maintaining good model learning ability.
[0013] Next, we process the multi-granularity training of face images. We use a random jigsaw patch generator to evenly split and shuffle image blocks to generate different granularity levels. During network training, we start from a more stable fine granularity and gradually transition to a coarser granularity to avoid confusion caused by differences within large categories that appear in large areas. In order for the convolutional neural network to find the most distinguishable parts more effectively, we use the cross entropy loss to train the network.
[0014] The last module is the 3DMM coefficient estimation module, which inputs the fused features into the fully connected layer and regresses the 62 coefficients of the 3DMM.
[0015] The present invention has the following beneficial effects:
[0016] The present invention provides a multi-granularity face feature aggregation method based on the attention mechanism, which aggregates shallow features and deep features at a specific granularity, and learns complementary information between different image granularities through multi-granularity progressive training. The network introduces more feature information while introducing context information. In addition, information across different granularities helps to avoid the influence of large intra-class differences, and can effectively achieve effective collaborative fusion of features of different granularities. The present invention not only enriches feature information, enhances the robustness of the network structure, and has a better anti-overfitting effect, but also makes the final classification result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Workflow diagram of multi-granularity face feature aggregation method based on attention mechanism;
[0018] Figure 2 The improved feature aggregation module of the present invention;
[0019] Figure 3 The network structure diagram after optimization of the present invention; DETAILED DESCRIPTION
[0020] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0021] The operating environment required for this example is pytorch1.6, ubuntu18.04, cuda11.4, and the server is k8036GB. The training parameters are batch_size of 128, 60 epochs, and the learning rate is set to 0.02. It takes 7 days to complete a whole task.
[0022] This example provides a multi-granularity face feature aggregation method based on the attention mechanism. The workflow diagram is as follows Figure 1 As shown in the figure, first, a dual-branch structure is used to perform sub-pixel convolution on the input image and output a 128*128 feature map.
[0023] Then the neural network structure is optimized. DLA consists of 6 levels, divided into 6 stage modules. The first two stages are composed of two convolutional layers, and the last four stages are hierarchical structures linked by tree structures, which can better propagate features and gradients. The first two convolutional layers use 3×3 convolution kernels for feature extraction. The step size of the first convolutional layer is 1, and the step size of the second convolutional layer is 2. After normalization, the Relu activation function is introduced in the convolutional layer, so that the network has more nonlinear transformations, further enhancing the network's ability to learn features. Starting from the third stage, the tree structure is introduced. In order to further integrate the feature maps of each stage module and channel, the simam attention mechanism is introduced in each stage. This mechanism does not require additional parameters to derive attention weights for feature maps. The operations are all selected based on the defined energy function, avoiding excessive structural adjustments. Each stage absorbs blocks and levels in the tree structure to retain and combine feature channels. Through the tree structure, shallow and deep network layers can be combined together, and such combination information can cross various levels to learn richer combination information. At the same time, the structure of the current stage will be aggregated into an aggregation node as the input of the next tree structure, so that the information of all previous blocks can be aggregated into subsequent processing instead of the previous blocks alone. These nodes are trained to select appropriate and important information to project to the output with the same scale as the input dimension. Each stage is carried out along the repeatedly iterative stacked backbone network, dividing the stacked blocks in the network into layers according to resolution, and refining the resolution and aggregation scale step by step.
[0024] Information across different granularities helps avoid the influence of large intra-class differences. For face reconstruction, not only the overall structure and special details of the face are needed, that is, it is not enough to just identify the distinguishable parts, but also how these parts interact with each other in a complementary way. In network design, a random jigsaw patch generator is used to evenly split and shuffle image blocks to generate different levels of granularity. In network training, starting from a more stable fine-grained granularity, gradually transitioning to a coarser granularity avoids the confusion caused by large intra-class differences that appear in large areas. It is equivalent to the network focusing on a local area and then shrinking the larger patches around this area until the entire image is completed. Specifically, at the end of each training step, the parameters of the current training step are passed to the next step as parameter initialization. This transfer enables the network to mine larger granularity information based on the areas learned in the previous training steps, and finally connect the features extracted in all stages. The feature maps processed in the first stage are divided into three granularities, namely 9 blocks, 4 blocks, and 1 block. After the fine granularity is fused with dla feature extraction, it is subjected to 1×1 and 3×3 convolutions, followed by a maximum pooling layer with a pooling window of 15×15 and a step size of 15. After the medium granularity is fused with dla feature extraction, it is subjected to 1×1 and 3×3 convolutions, followed by a maximum pooling layer with a maximum pooling window of 8×8 and a step size of 8. After the coarse granularity is fused with dla feature extraction, it is subjected to two 1*1 convolutions, followed by a fully connected operation, and then the outputs of the three granularities are spliced, and the final coefficients are returned after the maximum pooling and fully connected operations. At the same time, in order to enable the convolutional neural network to find the most distinguishing parts more effectively, cross entropy loss is used for training.
[0025] Obviously, the described embodiment is only one possible embodiment of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of the present invention.
Claims
1. A multi-granularity face feature aggregation method based on attention mechanism. The key point is to perform multi-granularity feature aggregation on image information based on an adjusted backbone network. The network part includes designing an efficient upsampling encoder to extract features for preliminary feature information processing. The backbone network performs deep aggregation of specific granularities to achieve semantic and spatial fusion. During the network training process, multi-granularity progressive training is performed to learn the complementary information between different image granularities. The shallow features of face images mainly include contours, edges, colors, textures, and shapes that can reflect the features of the image content. The shallow features have less semantic information, but the target position is accurate; the deep features of the image are mainly semantic features that are understandable by computers, but the target position is relatively rough. For face images, they are mainly divided into coarse-grained, medium-grained, and fine-grained for training, starting from the more stable fine-grained and gradually transitioning to coarser granularity, avoiding the confusion caused by differences within large categories in large areas. It is equivalent to the network focusing on a local area, and then shrinking the larger patch around this area until the entire image is completed. Specifically, at the end of each training step, the parameters of the current training step are passed to the next step as parameter initialization. This transfer enables the network to mine larger granularity information based on the areas learned in the previous training steps, and finally connect the features extracted in all stages. The specific steps include: First, the face dataset is processed initially, using sub-pixel convolution for simple upsampling to generate a feature map of a specific size; Then the neural network structure is optimized, and the backbone is modified to Deep Layer Aggregation, or DLA. Deep aggregation mainly has 6 stages, and the first two stages are composed of convolutional layers. The next four stages are hierarchical structures linked by tree structures, which can better propagate features and gradients. In each stage, the HDA structure is used to fuse the internal features of the stage, and the aggregation root node is shared between stages. Each stage uses max pooling with a kernel size of 2 and a step size of 2 for downsampling. Considering that facial feature information is relatively concentrated, the simam attention mechanism is introduced in the first stage. The output features of each encoder block are continuously corrected through the spatial domain suppression effect. Through multi-granular information propagation, the model parameters are reduced while maintaining good model learning ability. Next, we process the multi-granularity training of face images. We use a random jigsaw patch generator to evenly split and shuffle image blocks to generate different granularity levels. During network training, we start from a more stable fine granularity and gradually transition to a coarser granularity to avoid confusion caused by differences within large categories that appear in large areas. In order for the convolutional neural network to find the most distinguishable parts more effectively, we use the cross entropy loss to train the network. The last module is the 3DMM coefficient estimation module, which inputs the fused features into the fully connected layer and regresses the 62 coefficients of the 3DMM.
2. A multi-granularity face feature aggregation method based on attention mechanism according to claim 1, characterized in that: The features of convolutional layers of different granularities in the network are aggregated to avoid the loss of facial features. While ensuring the introduction of contextual information, richer features are extracted, and the accuracy and effectiveness of the extracted features are improved to a certain extent.