Lightweight face analysis method and device taking into account global and local feature relationships

Abstract

The present invention relates to the field of computer vision, and in particular to a lightweight face analysis method and device that takes into account global and local feature relationships. The method comprises the following steps: receiving an RGB image of a face, performing a convolution batch embedding operation on it to obtain an original feature map; performing a Transformer operation combined with convolution on the original feature map to obtain feature maps of different sizes; sending the feature maps of different sizes to FFN for feature fusion; and using the fused features to estimate the region category to which a single pixel belongs. The method of the present invention ensures that both long-distance pixel relationships and short-distance pixel relationships are taken into account by introducing a Transformer architecture and combining CNN, thereby greatly reducing the computational complexity.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a lightweight face analysis method and device that takes into account global and local feature relationships. Background Art

[0002] Facial parsing is a basic task to segment a face into different semantic regions, which provides a basis for many downstream applications such as facial analysis, facial recognition, and facial editing. In recent years, CNN has been widely used in the field of facial analysis, facial recognition, and facial editing.

[0003] It can extract the edge features of different face areas by extracting the relationship between adjacent pixels. Therefore, it has been widely used in various facial tasks, and many face parsing models based on convolutional neural networks (CNN) have emerged.

[0004] However, methods based on fully convolutional networks (FCNs) perform well in fully supervised face parsing tasks, such as the new operator proposed by J.Lin et al. to solve the RoI-related problem, but FCNs cannot capture global context information. Local-based methods train independent models for different facial regions to predict separately, but ignore the shared backbone network and joint optimization.

[0005] potential. Liu et al. and L. Zhou et al. incorporated CNN features into the CRF framework but did not consider regional correlation. A. S. Jackson et al. introduced a CNN cascade model to explore the relationship between pose estimation and segmentation, but it was difficult to use additional information. G. Te et al. introduced GCN into facial parsing, and Q. Zheng et al. introduced DML-CSR for facial parsing. The algorithm combined with graph neural network is good but the calculation is complex. Summary of the invention

[0006] In order to overcome the limitations of the prior art, the present invention aims to propose a lightweight end-to-end face parsing method that can effectively take into account the global feature relationship and local feature relationship of the image while greatly reducing the computational complexity. By introducing the Transformer architecture, the multi-head self-attention mechanism is used to extract the long-distance pixel relationship, improve the local information attention, and use the windowed self-attention mechanism to reduce the amount of calculation, and innovatively design a beam layer normalization method to solve the problem of inconsistent normalization of different parts of the same image.

[0007] In order to achieve the above objectives, the following technical solutions are proposed:

[0008] A lightweight face parsing method taking into account the relationship between global and local features comprises the following steps:

[0009] S1. Receive the RGB image of the face and perform a convolution batch embedding operation on it to obtain the original feature map;

[0010] S2. The original feature map is subjected to a Transformer operation combined with convolution to obtain feature maps of different sizes;

[0011] S3. Sending the feature maps of different sizes to FFN for feature fusion;

[0012] S4. Using the fused features to estimate the region category to which a single pixel belongs;

[0013] The Transformer operation combined with convolution uses beam layer normalization operations multiple times to ensure consistent normalization of the entire image;

[0014] Among them, convolutional neural networks are used for block embedding in the convolution batch embedding operation and the Transformer operation combined with convolution, which is used to solve the problem of the block embedding module of the original Transformer destroying the connectivity between adjacent pixels and preventing the loss of the relationship between the edges of image blocks and other image blocks.

[0015] Furthermore, the calculation formula for beam layer normalization is as follows:

[0016] ;

[0017] Among them, x is the feature input to the module, xi is the i-th sample in x, represents the field of real numbers, Indicates that the dimension of x is b, ,c, , Corresponding to Mean and variance in the channel dimension, m is the total number of samples, i is the sample number, , are learnable parameters, Is the minimum value to prevent the variance from being 0, X is the input face image, b is the batch size, and c is the number of channels of the image.

[0018] Furthermore, the formula for block embedding using convolutional neural network is as follows:

[0019] The convolution operation is: ;

[0020] in, is the input face image, H is the height of the image, W is the width of the image, and C is the number of channels of the image;

[0021] The size of the image after convolution is: ;

[0022] Among them, f( ) is the convolution operation, s is the size of the convolution kernel, is the step length, is the number of pixels filled around the image.

[0023] , , are the height, width, and number of channels of the image after convolution.

[0024] Furthermore, step S1 specifically includes the following steps:

[0025] S11. Receive an RGB image of a face, convert the RGB image into a feature map in the (B, H, W, C) format as input; where B represents the batch size, H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image;

[0026] S12. Perform two convolution operations on the feature map in the (B, H, W, C) format;

[0027] S13. Reshape the convolved feature map and output the original feature map.

[0028] Furthermore, the Transformer operation combined with convolution in step S2 specifically includes the following steps:

[0029] S21. Reduce the original feature map by passing it through a combination of long-short distance Transformer modules and convolution-based downsampling layers until the size is reduced to the required size;

[0030] S22. The feature map of the required size is based on the upsampling layer of deconvolution and the long-short distance Transformer module of the corresponding size to restore the features of the original size, and obtain feature maps of different sizes in the restoration process, in which the features extracted by downsampling are fused during upsampling.

[0031] Furthermore, the calculation method of the long-short distance Transformer module in step S21 is:

[0032] ;

[0033] Among them, x is the input (B, L, C) feature map, BLN () is the beam layer normalization module, LSAttention () is the long-short distance

[0034] From the attention module, MLP () is composed of two fully connected layers, x output is the output of the long-short distance Transformer module

[0035] Parameters.

[0036] Furthermore, the long-short distance attention module is mainly divided into two branches, one branch is the window multi-head self-attention module, and the other is the convolution branch;

[0037] ;

[0038] in, , , Refers to the operation of directly assigning feature x to Q, K, V. Q, K, V are the names of three features defined for the convenience of expression.

[0039] The process of the window multi-head self-attention module is:

[0040] ;

[0041] ;

[0042] Among them, d k is the number of channels of Q, K, V, and the head is composed of Composition,Concat represents the concatenation operation in the depth dimension; , , and They are the corresponding learnable parameters, The feature depth of the input long and short distance attention module. For each size of feature map, the Attention operation only calculates self-attention within non-overlapping windows;

[0043] The flow of the structure of the convolution branch is as follows:

[0044] ;

[0045] in, Refers to the size transformation operation, which converts the feature from a spatial vector to a sequence vector. Represents the reverse of the above transformation

[0046] operation, x' is the output of the convolution branch structure; bottleneck represents the output after the hourglass convolution layer processes the features, which is composed of two reshape layers, three convolution layers and a layer normalization layer. The number of convolution kernels in the three layers is 1, 3, and 1 respectively; BLN represents the output of the features after the beam layer normalization, and GELU is the Gaussian error linear unit;

[0047] The overall process of the long and short distance attention module is as follows:

[0048] ;

[0049] Among them, x' is the output of the convolution branch, and x is the feature of the input long and short distance attention module.

[0050] Furthermore, the steps of the convolution downsampling layer in step S21 are:

[0051] S31. Input (B, L, C) feature map to perform reshape operation, and output (B, H, W, C) feature map;

[0052] S32. Perform a convolution operation on the above (B, H, W, C) feature map;

[0053] S33. Performing beam-layer normalization processing on the above convolution feature map;

[0054] S34. Reshape the result after the above normalization processing and output (B, L / 4, 2C).

[0055] Furthermore, step S3 specifically includes the following steps:

[0056] ;

[0057] ;

[0058] Among them, x1 and x2 are two features that need to be fused; CNN modules are composed of two layers of convolution with a convolution kernel of 1, which represents the output of the CNN module after adding the two features. represents the output of global average pooling after adding two vectors, represents the multiplication of corresponding elements of the matrix, ⊕ represents the addition of corresponding elements, and the final output Z is the result of the fusion of the two features.

[0059] Based on the same concept, a lightweight face parsing device that takes into account both global and local feature relationships is also proposed, comprising at least one processor and a memory that is communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned lightweight face parsing methods that take into account both global and local feature relationships.

[0060] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a lightweight face parsing method that takes into account both global and local feature relationships, which cleverly combines the local feature extraction advantages of convolutional neural networks (CNNs) with the long-distance dependency processing capabilities of Transformer models, and achieves comprehensive capture of local details and overall structures of face images; by introducing convolution operations in block embedding and downsampling layers, the position information of adjacent areas after block segmentation is effectively retained, and the model's understanding of the relationship between local pixels is enhanced; by utilizing the convolution characteristics, the present invention innovatively reduces the need for moving windows and reduces the number of attention modules, thereby simplifying the model architecture and improving computational efficiency; the beam layer normalization strategy ensures that all blocks in the same image receive consistent normalization calculations, maintains the original relative relationship between blocks, and enhances the robustness and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is the overall framework flow chart of the method of the present invention;

[0062] FIG. 2( a) is a flow chart of a specific embodiment of the method of the present invention;

[0063] FIG2( b ) is a flow chart of the long-short distance Transformer module in the method of the present invention;

[0064] FIG2( c ) is a flow chart of the long-short distance attention module in the long-short distance Transformer module in the method of the present invention;

[0065] FIG2( d ) is a flow chart of the convolutional batch embedding module in the method of the present invention;

[0066] FIG2( e ) is a flow chart of a convolution-based downsampling layer in the method of the present invention;

[0067] Figure 3 It is a flow chart of the FFN module in the method of the present invention;

[0068] Figure 4 This is a diagram showing the effect of normalization of the beam layer in the present invention;

[0069] Figure 5 The difference between the processing effect of the present invention on the existing face analysis failure samples and the original method is compared;

[0070] Figure 6 The present invention is compared with the existing advanced model DML-CSR;

[0071] Figure 7 The experimental results of the present invention on the LaPa dataset;

[0072] Figure 8The experimental results of the present invention on the CelebAMask-HQ dataset;

[0073] Fig. 9 Represents the input image size, parameter quantity Params and computational quantity MACs of different models. DETAILED DESCRIPTION

[0074] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0075] Example 1

[0076] A lightweight face parsing method that takes into account both global and local feature relationships. The overall framework flow chart is as follows: Figure 1 The following steps are shown:

[0077] S1. Receive the RGB image of the face and perform a convolution batch embedding operation on it to obtain the original feature map;

[0078] S2. The original feature map is subjected to a Transformer operation combined with convolution to obtain feature maps of different sizes;

[0079] S3. Send the feature maps of different sizes to FFN for feature fusion;

[0080] S4. Estimate the region category to which a single pixel belongs using the fused features;

[0081] The Transformer operation combined with convolution uses beam layer normalization operations multiple times to ensure consistent normalization of the entire image;

[0082] The convolutional batch embedding operation and the Transformer operation combined with convolution use convolutional neural networks for block embedding to solve the problem of the original Transformer's block embedding module destroying the connectivity between adjacent pixels and preventing the loss of the relationship between the edges of image blocks and other image blocks.

[0083] The flowchart of the specific embodiment is shown in FIG2(a), which includes the following steps:

[0084] Step 1: Send the three-channel face image of size H*W into the convolutional batch embedding module to become The feature map of the convolutional batch embedding module is shown in Figure 2(d).

[0085] Step 2: The feature map is then passed through the long-short distance Transformer module, which can take into account both the long-distance and short-distance pixel relationships, and the convolution-based downsampling layer is used to reduce the size of the feature map to , and then goes through a combination of long and short distance Transformer modules and convolution-based downsampling layers until the size is reduced to , where the long-short distance Transformer module is shown in Figure 2(b), and the convolution-based downsampling layer is shown in Figure 2(e);

[0086] Step 3: Set the size to The feature image is sent to the upsampling layer based on deconvolution and the long-short distance Transformer module of the corresponding size to restore the features of the original size. The features extracted by downsampling will be fused during upsampling;

[0087] Step 4: The feature maps of three different sizes in the size recovery process ( ) is sent to the feature fusion module to estimate the region category to which a single pixel belongs based on the fused features. The feature fusion module is as follows: Figure 3 As shown;

[0088] Furthermore, in this embodiment, in order to solve the problem that the block embedding module of the original Transformer destroys the connectivity between adjacent pixels and loses the relationship between the edge of the image block and other image blocks, the convolutional neural network is used for block embedding in the convolutional batch embedding module and the convolution-based downsampling layer in steps 1 and 2, which can extract the positional relationship between the image block and other image blocks to improve the segmentation accuracy. Specifically, for an input face image, , f() is a convolution operation, where the convolution kernel is s*s and the step length is , the surrounding pixels are filled with After the convolution operation, the feature map is obtained , the dimensions are:

[0089] ;

[0090] Among them, f() is the convolution operation, where the convolution kernel is s*s, is the step length, Surrounding pixel padding. is the input face image, H, W are the image sizes, and C is the number of channels.

[0091] As shown in Figure 2 (d), the convolutional batch embedding module consists of two convolutions. The first convolution kernel size is 3, the stride is 1, and the padding is 1; the second layer kernel size is 4, the stride is 4, and no padding is performed.

[0092] As shown in Figure 2(e), for the convolution-based downsampling layer, the convolution kernel size is set to 7, the stride is set to 2, and no padding is performed.

[0093] Furthermore, in this embodiment, the long-short distance Transformer module and the convolution-based downsampling layer in step 2 use bundle normalization. The bundle normalization method is to normalize multiple blocks belonging to the same face image together. The formula is as follows:

[0094] ;

[0095] Among them, x is the feature input to the module, x i is the i-th sample in x, represents the field of real numbers, Indicates that the dimension of x is b, ,c, , Corresponding to Mean and variance in the channel dimension, m is the total number of samples, i is the sample number, , are learnable parameters, is a minimum value to prevent the variance from being zero, set to 1e-6; X is the input face image, b is the batch size, and c is the number of channels of the image; beam layer normalization can ensure that the blocks belonging to the same image can be uniformly normalized, so as not to destroy the relative relationship between the blocks belonging to the same image, such as Figure 4 The figure shows the effect of beam layer normalization.

[0096] Furthermore, in this embodiment, the calculation method of the long-short distance Transformer module in step 3 is:

[0097] ;

[0098] Where x is the input (B, L, C) feature map, BLN is the beam layer normalization module, LSAttention() is the long-short distance attention module, MLP() is composed of two fully connected layers, and x output are the parameters output by the long and short distance Transformer module.

[0099] The long- and short-distance attention module is divided into two branches, one is the window multi-head self-attention module, and the other is the convolution branch, as shown in Figure 2 (c).

[0100] The overall process of the long and short distance attention module is as follows:

[0101] ;

[0102] in, , , Refers to the operation of directly assigning feature x to Q, K, V. Q, K, V are the names of three features defined for the convenience of description. At the same time, they are divided into 7*7 windows respectively. Attention() refers to the window multi-head self-attention module.

[0103] The process of the window multi-head self-attention module is as follows:

[0104] ;

[0105] ;

[0106] Among them, d k is the number of channels of Q, K, V, and the head is composed of Composition,Concat represents the concatenation operation in the depth dimension; , , and They are the corresponding learnable parameters, The feature depth of the input long and short distance attention module. For each size of feature map, the Attention operation only calculates self-attention within non-overlapping windows;

[0107] The structure of the convolution branch is as follows:

[0108] ;

[0109] in, Refers to the size transformation operation, which converts the feature from a spatial vector to a sequence vector. represents the reverse operation of the above transformation; x' is the output of the convolution branch structure; bottleneck represents the output after the hourglass convolution layer processes the features, which is composed of two reshape layers, three convolutions and one layer normalization layer. The number of convolution kernels in the three layers is 1, 3, and 1 respectively; BLN represents the output of the features after beam layer normalization, and GELU is the Gaussian error linear unit.

[0110] Furthermore, in this embodiment, in step 4, the feature fusion module is as follows: Figure 3 As shown in the figure, both CNN modules are composed of two layers of convolution with a convolution kernel of 1. The calculation is as follows:

[0111] ;

[0112] ;

[0113] Among them, CNN (x1 + x2) represents the output of the CNN module after adding two features, and GAP (x1 + x2) represents the output of global average pooling after adding two vectors. Represents the matrix corresponding element multiplication. The final output Z is the result of the fusion of the two features.

[0114] Example 2

[0115] Experiments on CelebAMask-HQ and LaPa datasets Figure 7 and Figure 8 This shows that the method has a better effect. The present invention has the highest average F1 value and shows good performance in multiple subcategories. Figure 5 Figure 6 The visual segmentation result comparison is shown, and it is obvious that the present invention can better cope with the face segmentation task under complex backgrounds, occlusions, large angles and the like. Fig. 9 It is explained that the parameter amount and calculation amount of the present invention are smaller than those of the existing methods.

[0116] Finally, it should be noted that the embodiments described in detail above are only the best practices of the invention and cannot be used to limit the scope of rights of the invention. Equivalent replacement of the technical solutions recorded in the aforementioned embodiments does not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the invention, and they should all be included in the scope of the claims and description of the invention.

Claims

1. A lightweight face parsing method that takes into account the relationship between global and local features, characterized in that: The following steps are involved: S1. Receive the RGB image of the face and perform a convolution batch embedding operation on it to obtain the original feature map; S2. The original feature map is subjected to a Transformer operation combined with convolution to obtain feature maps of different sizes; S3. Sending the feature maps of different sizes to FFN for feature fusion; S4. Using the fused features to estimate the region category to which a single pixel belongs; The Transformer operation combined with convolution uses beam layer normalization operations multiple times to ensure consistent normalization of the entire image; The convolutional batch embedding operation and the Transformer operation combined with convolution use convolutional neural networks for block embedding to solve the problem of the original Transformer's block embedding module destroying the connectivity between adjacent pixels and preventing the loss of the relationship between the edge of the image block and other image blocks; The calculation formula of beam layer normalization is as follows: Among them, p is the feature input to the module, p i is the i-th sample in p, represents the field of real numbers, It means that the dimension of p is b, ,c, , Corresponding to Mean and variance in the channel dimension, m is the total number of samples, i is the sample number, , are learnable parameters, is the minimum value that prevents the variance from being 0, X is the input face image, b is the batch size, and c is the number of channels of the image; The Transformer operation combined with convolution includes a long- and short-distance attention module. The long- and short-distance attention module is divided into two branches, one branch is a window multi-head self-attention module, and the other is a convolution branch. The process of the window multi-head self-attention module is: ； ； where d k is the number of channels of Q, K, V, , , Refers to the operation of directly assigning feature x to Q, K, V. Q, K, V are the names of three features defined for the convenience of expression. The head is composed of Composition,Concat represents the concatenation operation in the depth dimension; , , and They are the corresponding learnable parameters, The feature depth of the input long and short distance attention module; the Attention operation only calculates self-attention within non-overlapping windows for feature maps of each size; The flow of the structure of the convolution branch is as follows: ； in, Refers to the size transformation operation, which converts the feature from a spatial vector to a sequence vector. Represents the reverse operation of the above transformation, bottleneck represents the output of the hourglass convolution layer after processing the features, BLN represents the output of the features after beam layer normalization, and GELU is the Gaussian error linear unit; The overall process of the long and short distance attention module is as follows: ； Among them, x' is the output of the convolution branch, and x is the feature of the input long and short distance attention module.

2. A lightweight face parsing method taking into account the relationship between global and local features according to claim 1, characterized in that: The formula for block embedding using convolutional neural networks is as follows: The convolution operation is: ; in, is the input face image, H is the height of the image, W is the width of the image, and C is the number of channels of the image; The size of the image after convolution is: ; Among them, f( ) is the convolution operation, s is the size of the convolution kernel, is the step length, is the number of pixels filled around the image. , , are the height, width, and number of channels of the image after convolution.

3. A lightweight face parsing method taking into account the relationship between global and local features according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Receive an RGB image of a face, convert the RGB image into a feature map in the (B, C, H, W) format as input; where B represents the batch size, H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image; S12. Perform two convolution operations on the feature map in the (B, C, H, W) format; S13. Reshape the convolved feature map and output the original feature map.

4. The lightweight face parsing method taking into account the relationship between global and local features according to claim 1, characterized in that: The Transformer operation combined with convolution in step S2 specifically includes the following steps: S21. Reduce the original feature map by passing it through a combination of long-short distance Transformer modules and convolution-based downsampling layers until the size is reduced to the required size; S22. The feature map of the required size is based on the upsampling layer of deconvolution and the long-short distance Transformer module of the corresponding size to restore the features of the original size, and obtain feature maps of different sizes in the restoration process, in which the features extracted by downsampling are fused during upsampling.

5. A lightweight face parsing method taking into account the relationship between global and local features according to claim 4, characterized in that: The calculation method of the long-short distance Transformer module in step S21 is: ； Among them, x is the input (B, L, C) feature map, BLN () is the beam layer normalization module, LSAttention () is the long-short distance From the attention module, MLP () is composed of two fully connected layers, x output is the output of the long-short distance Transformer module Parameters.

6. A lightweight face parsing method taking into account global and local feature relationships according to claim 4, characterized in that: The steps of the convolution downsampling layer in step S21 are: S31. Input (B, L, C) feature map to perform reshape operation, and output (B, C, H, W) feature map; S32. Perform a convolution operation on the above (B, C, H, W) feature map; S33. Performing beam-layer normalization processing on the above convolution feature map; S34. Reshape the result after the above normalization processing and output (B, L / 4, 2C).

7. The lightweight face parsing method taking into account the relationship between global and local features according to claim 1, characterized in that: Step S3 specifically includes the following steps: ； ； Among them, x1 and x2 are two features that need to be fused; the CNN module is composed of two layers of convolution with a convolution kernel of 1. represents the output of the CNN module after adding two features. represents the output of global average pooling after adding two vectors, represents the multiplication of corresponding elements of the matrix, ⊕ represents the addition of corresponding elements, and the final output Z is the result of the fusion of the two features.

8. A lightweight face analysis device that takes into account both global and local feature relationships, characterized in that: It includes at least one processor and a memory that is communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lightweight face parsing method that takes into account the relationship between global and local features as described in any one of claims 1 to 7.

Images