Face key point detection method based on feature enhancement

Abstract

This invention provides a facial keypoint detection method based on feature enhancement. First, convolutional features are extracted from facial images and mapped into graph node features. A keypoint adjacency matrix is generated based on a dataset. The graph node features and the keypoint adjacency matrix are then fed into a feature enhancement module. The feature enhancement module consists of four cascaded hourglass networks with a message passing layer inserted between adjacent hourglass networks. Keypoints are grouped according to facial structure, with each group of keypoints connected to form a boundary, resulting in a total of 13 boundaries. Boundaries are treated as tree nodes, and a bidirectional tree structure is constructed. Message passing is performed through the bidirectional tree structure to obtain enhanced node features. The enhanced node features are then subjected to graph convolutional inference in the keypoint generation module to predict a keypoint coordinate matrix, completing facial keypoint detection. This method utilizes a message passing mechanism to enable visible boundaries to provide information about occlusion boundaries based on facial structure, thereby improving the accuracy of predicted keypoint coordinates.

Description

Technical Field

[0001] The present invention belongs to the technical field of face recognition and detection, and in particular is a method for detecting key points of a face based on feature enhancement. Background Art

[0002] Facial landmark detection, also known as face alignment, is an important research area in computer vision. It plays a crucial role in connecting face recognition, facial expression analysis, and face reconstruction. The accuracy of facial landmark detection plays a crucial role in subsequent face-related tasks.

[0003] With the successful application of deep learning in computer vision tasks such as image classification and object detection, many facial landmark detection methods based on convolutional neural networks (CNNs) have been proposed. Existing CNN-based facial landmark detection algorithms can be divided into two main categories: heatmap regression-based methods and coordinate regression-based methods. Coordinate regression-based methods are relatively simple and can be built on lightweight convolutional networks. Heatmap regression-based methods typically use stacked hourglass networks (SHNs). Because they maintain both feature maps and heatmaps, they generally have higher detection accuracy than coordinate regression-based methods.

[0004] The above method has achieved good results in conventional facial key point detection, but the detection accuracy is low in extreme cases such as severe occlusion and large-angle side faces. Therefore, the present invention proposes a facial key point detection method based on feature enhancement, which also has good detection performance in the aforementioned extreme cases. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention aims to provide a method for detecting key points of a face based on feature enhancement.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A method for detecting key points of a face based on feature enhancement, characterized in that the method comprises the following steps:

[0008] Step 1: Obtain a dataset consisting of multiple face images. Each face image contains N key points, which are distributed in the facial contour, eyebrow contour, lip contour, and nose bridge position.

[0009] Step 2: Construct a facial key point detection model, including a feature extraction module, a node embedding module, an adjacency matrix generation module, a feature enhancement module, and a key point generation module;

[0010] 2-1. The face image is processed through the feature extraction module to obtain convolution features, and the node embedding module maps the convolution features into graph node features;

[0011] 2-2. Obtain the key point adjacency matrix through the adjacency matrix generation module;

[0012] First, construct the dataset into a third-order data tensor before training Represents the real number space, M represents the number of face images, and (x, y) represents the coordinates of the key points; then, the third-order data tensor T is divided into data tensors along the x and y directions and All face images form a data tensor T s and T y The horizontal and vertical coordinates of all key points of each face image constitute the data tensor T x and T y Column; according to the data tensor T x , calculate the Pearson correlation coefficient between key points along the x direction and obtain the x-direction correlation coefficient matrix Similarly, the y-direction correlation coefficient matrix is obtained Then calculate the Pearson correlation coefficient between the key points in the face image through formula (1) to obtain the relationship matrix C;

[0013]

[0014] Where abs(·) represents the key value of the returned matrix element;

[0015] Select the n largest values in each row of the relationship matrix C to construct a sparse matrix Q, and select the n most relevant key points for each key point; normalize the sparse matrix Q using formula (2) to obtain the key point adjacency matrix;

[0016]

[0017] Where Q ij 、C ij Represent the elements in row i and column j in the sparse matrix Q and the relational matrix C, respectively. Indicates the largest n values in each row;

[0018] 2-3. The feature enhancement module is composed of four cascaded hourglass networks, with a message passing layer inserted between two adjacent hourglass networks. The graph node features and key point adjacency matrix are input into the feature enhancement module to obtain enhanced node features.

[0019] All key points are grouped according to the facial structure, and each group of key points is connected into a boundary, resulting in a total of 13 boundaries, namely, the outer contour of the face, the left eyebrow, the right eyebrow, the nose bridge, the lower boundary of the nose, the upper boundary of the left eye, the lower boundary of the left eye, the upper boundary of the right eye, the lower boundary of the right eye, the upper boundary of the upper lip, the lower boundary of the upper lip, the upper boundary of the lower lip and the lower boundary of the lower lip; Definition: The outer contour of the face is connected with the left and right eyebrows, the lower boundaries of the left and right eyes and the lower boundary of the lower lip, and the upper boundary of the left eye is connected with the left eyebrow and the left eyebrow respectively. There is a connection between the lower boundaries of the eyes, the upper boundary of the right eye is respectively related to the right eyebrow and the lower boundary of the right eye, the bridge of the nose is respectively related to the lower boundaries of the left and right eyes and the lower boundary of the nose, the upper boundary of the upper lip is respectively related to the lower boundary of the nose and the lower boundary of the upper lip, and the upper boundary of the lower lip is respectively related to the lower boundary of the upper lip and the lower boundary of the lower lip; the boundaries are equivalent to tree nodes, and the tree nodes corresponding to the related boundaries are connected to construct a bidirectional tree structure composed of tree structures A and B. The tree nodes A1~A1 in the tree structures A and B are 13 and tree nodes B1 to B 13 They represent the outer contour of the face, left eyebrow, right eyebrow, nose bridge, lower boundary of the nose, upper boundary of the left eye, lower boundary of the left eye, upper boundary of the right eye, lower boundary of the right eye, upper boundary of the upper lip, lower boundary of the upper lip, upper boundary of the lower lip and lower boundary of the lower lip respectively;

[0020] Message passing is divided into intra-layer message passing and inter-layer message passing. Intra-layer messages are passed according to a bidirectional tree structure. For the message passing layer between the first hourglass network and the second hourglass network, when passing according to the tree structure A, a loop operation consisting of a convolutional layer, a batch normalization layer, an activation layer, an inter-layer message passing layer, and an intra-layer message passing layer is performed on each tree node. First, the first loop operation starts from the tree node A1. The graph node feature output by the first hourglass network passes through the convolutional layer to obtain feature F. After feature F passes through the batch normalization layer and the activation layer, feature R1 is obtained. Feature R1 passes through three divisions. The first operation is to put the feature R1 into the feature matrix result_A and update the feature matrix result_A. The feature matrix result_A is initially empty. The second operation is to get the inter-layer message passing feature MsgA1 after the feature R1 passes through the inter-layer message passing layer, and put the inter-layer message passing feature MsgA1 into the inter-layer feature matrix and update the inter-layer feature matrix. The third operation is to get the intra-layer message passing feature Inner1_out after the feature R1 passes through the intra-layer message passing layer. Since the tree node A1 will affect the tree nodes A2, A3, A7, A9 and A 13 Therefore, the intra-layer message passing feature Inner1_out is placed in the adjacent tree node feature matrix tree nodes A2, A3, A7, A9 and A 13The corresponding positions are recorded as the adjacent tree node features Msg_a2, Msg_a3, Msg_a7, Msg_a9 and Msg_a13, respectively. The adjacent tree node feature matrix is initially empty. Then, the second loop operation is performed on the tree node A. 13 The first hourglass network outputs the node feature through the convolution layer, and the dimension is feature F. Feature F and the number of nodes A 13 The adjacent tree node features Msg_a13 are added and then passed through the batch normalization layer and the activation layer to obtain the feature R2; the feature R2 undergoes three branch operations. First, the feature R2 is placed in the feature matrix result_A to update the feature matrix result_A; second, the feature R2 passes through the inter-layer message passing layer to obtain the inter-layer message passing feature MsgA13, and the inter-layer message passing feature MsgA13 is placed in the inter-layer feature matrix to update the inter-layer feature matrix; third, the feature R2 passes through the intra-layer message passing layer to obtain the intra-layer message passing feature Inner13_out; since the tree node A 13 Will affect tree node A 12 , so the intra-layer message passing feature Inner13_out is used as tree node A 12 The adjacent tree node features are placed in the adjacent tree node feature matrix tree node A 12 At the corresponding position, the feature matrix of the adjacent tree node is updated; similarly, the loop operation is performed on the remaining tree nodes to obtain the feature matrix result_A and the inter-layer feature matrix; when the loop operation is performed on the remaining tree nodes, the feature F is added to the adjacent tree node features of the tree node and then passed through the batch normalization layer and the activation layer; the same loop operation is performed on each tree node of the tree structure B to obtain the feature matrix result_B and the updated inter-layer feature matrix;

[0021] Finally, the feature matrices result_A and result_B are concatenated along the 0th dimension to obtain 13 features. These 13 features are subjected to convolution, batch normalization, and activation operations to obtain 13 node features. The 13 node features are then concatenated along the 0th dimension to obtain node features. This node feature is subjected to convolution, batch normalization, and activation operations once more to obtain the output features of the message passing layer.

[0022] For inter-layer message passing, in each loop operation of the next message passing layer, the graph node features output by the hourglass network are convolved, and then added to the inter-layer message passing features corresponding to the tree nodes in the inter-layer feature matrix obtained by the previous message passing layer. After that, the features are passed through the batch normalization layer and the activation layer to realize inter-layer message passing.

[0023] 2-3. The enhanced node features are subjected to graph convolution inference in the key point generation module to predict the key point coordinate matrix;

[0024] First, a trainable parameter matrix with uniform distribution is constructed, and the enhanced node features are multiplied by the 0th and 1st dimensions of the parameter matrix respectively to obtain the key point coordinate matrices h0 and h1; secondly, weights are set for all non-zero positions of the key point adjacency matrix to obtain the key point self-weight matrix and the adjacent key point weight matrix respectively; finally, the key point self-weight matrix is multiplied by the key point coordinate matrix h0, and the adjacent key point weight matrix is multiplied by the key point coordinate matrix h1, and the multiplied results are added together. The added matrix is then passed through the batch normalization layer and the activation layer to obtain the key point coordinate matrix, that is, the coordinates of the face key points;

[0025] Step 3: Use the dataset to train the facial key point detection model, and use the trained facial key point detection model for facial key point detection.

[0026] Furthermore, each hourglass network of the feature enhancement module has a depth of 4 layers, and the operation of each layer is the same; for the graph node features input into the hourglass network, the graph node features are input into the residual module to obtain feature up1; at the same time, the graph node features are subjected to a maximum pooling operation to obtain feature low1, feature low1 is input into the residual module to obtain feature low2, feature low2 is then passed through the residual module to obtain feature low3, feature low3 is subjected to an upsampling operation to obtain feature up2, and finally features up1 and up2 are added to obtain the output of the first layer depth of the hourglass network; the above operation is repeated four times to obtain the graph node features output by the hourglass network;

[0027] The residual module consists of two graph convolution modules, each of which includes a graph convolution layer, a batch normalization layer and an activation layer.

[0028] Furthermore, during the training of the face key point detection model, the total loss function is defined as:

[0029]

[0030] in, represents the wing loss function, Represents the key point p i The coordinate prediction value of represents the relative position loss function, δ i Represents the key point p i The position error between it and its adjacent key points, Represents δ i The predicted value of

[0031] Given a key point p i and his neighborhood set K i ={j:j≠i,Q ij=1}, define the key point p i The position function is:

[0032]

[0033] Among them, δ i Represents the key point p i The position error between it and its adjacent key points, ω ij Represents the key point p i Its adjacent key point p j The position weight between

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. In the feature enhancement module, all key points are grouped according to facial structure, resulting in multiple boundaries. A bidirectional tree structure is then constructed, and message passing is performed within this bidirectional tree structure. Furthermore, by combining the multi-scale feature recognition capabilities of the hourglass network with a message passing mechanism, feature enhancement is performed. Visible boundaries can provide information about occlusion boundaries based on the facial structure, improving the accuracy of predicted key point coordinates. This method is particularly suitable for facial key point detection in extreme situations such as severe occlusion, wide-angle profiles, and blur.

[0036] 2. This method fully considers the correlation between key points. Based on the correlation coefficient between key points, it selects multiple most relevant key points for each key point and constructs a key point adjacency matrix. The enhanced node features and the key point adjacency matrix are used as input to the key point generation module. In this way, each key point is no longer independent, but is connected to its adjacent key points. This connection can help predict occluded key points.

[0037] 3. During the training process of the facial key point detection model, the training loss is measured by combining the relative position loss function and the wing loss function, making the loss function sensitive to the errors at each stage, thereby further improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0039] Figure 2 It is a structural diagram of the face key point detection model of the present invention;

[0040] Figure 3(a) is a schematic diagram of the key point distribution of a face image;

[0041] Figure 3(b) is a schematic diagram of the boundary distribution of a face image;

[0042] FIG4( a ) is a schematic diagram of a tree structure A of a bidirectional tree structure;

[0043] FIG4( b ) is a schematic diagram of a tree structure B of a bidirectional tree structure;

[0044] FIG5( a) is a flow chart of the first cycle operation of the first message delivery layer of the present invention;

[0045] FIG5( b ) is a flow chart of the second cycle operation of the first message delivery layer of the present invention;

[0046] FIG5( c ) is a flow chart of the first cycle operation of the second message delivery layer of the present invention;

[0047] FIG5( d ) is a flow chart of the second cycle operation of the second message delivery layer of the present invention;

[0048] Figure 6 It is a visualization diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present application is not limited thereto.

[0050] The present invention provides a method for detecting key points of human face based on feature enhancement (hereinafter referred to as method, see Figures 1 to 6 ), including the following steps:

[0051] Step 1: Get the dataset.

[0052] This example uses the WFLW dataset as experimental data. The training set of this dataset contains 7,500 face images, and the test set contains 2,500 face images. Each face image contains N key points, which are distributed in the facial contour, eyebrow contour, lip contour, and nose bridge, as shown in Figure 3(a). The input dimension of each face image is 256×256×3, that is, the width and height of the face image are both 256 pixels, and the depth is 3 channels.

[0053] Step 2: Construct a facial key point detection model, including a feature extraction module, a node embedding module, an adjacency matrix generation module, a feature enhancement module, and a key point generation module;

[0054] 2-1. The face image is processed through the feature extraction module to obtain convolution features, and the node embedding module maps the convolution features into graph node features;

[0055] The feature extraction module uses the ResNet34 residual network. The face image is input into the feature extraction module, which outputs convolutional features with a dimension of 8×8×512. The convolutional features are used as input to the node embedding module, which maps the convolutional features into graph node features with an output dimension of 256×N. The node embedding module consists of two convolutional layers. The convolution kernel size of each convolution layer is 3×3, the stride is 1, and the activation function is the ReLu function.

[0056] 2-2. Obtain the key point adjacency matrix through the adjacency matrix generation module;

[0057] Since there is no predefined adjacency matrix in the face key point detection task, the key point adjacency matrix is obtained through the adjacency matrix generation module; first, the dataset is constructed into a third-order data tensor before training. M represents the number of face images, (x, y) represents the coordinates of key points; then, the third-order data tensor T is divided into data tensors along the x and y directions and All face images form a data tensor T x and T y The horizontal and vertical coordinates of all key points of each face image constitute the data tensor T x and T y Column; according to the data tensor T x , calculate the Pearson correlation coefficient between key points along the x direction and obtain the x-direction correlation coefficient matrix Similarly, the y-direction correlation coefficient matrix is obtained Then, the Pearson correlation coefficient between the key points in the face image is calculated by formula (1) to obtain the relationship matrix C with dimension N×N;

[0058]

[0059] Where abs(·) represents the key value of the returned matrix element;

[0060] Considering the influence of computational complexity and noise, we select the n largest values in each row of the relationship matrix C to construct a sparse matrix Q with a dimension of N×N, that is, we select the n most relevant key points for each key point; we normalize the sparse matrix Q by formula (2) to obtain the key point adjacency matrix;

[0061]

[0062] Where Q ij 、C ij Represent the elements in row i and column j in the sparse matrix Q and the relational matrix C, respectively. Indicates the largest n values in each row;

[0063] 2-3. Input the graph node features and key point adjacency matrix into the feature enhancement module to obtain enhanced node features;

[0064] The feature enhancement module is composed of four cascaded hourglass networks, with a message passing layer inserted between two adjacent hourglass networks. Each hourglass network has a depth of 4 layers, and the operation of each layer is the same. With each additional layer of depth, the size of the input graph node feature becomes half of the original, but the dimension of the graph node feature output by the hourglass network remains unchanged, and the information represented by the feature is increased compared to the original. For the graph node features input into the hourglass network, the graph node features are input into the residual module to obtain feature up1 with a size of N×256. At the same time, the graph node features are subjected to a maximum pooling operation to obtain feature low1 with a size of N×128. Feature low1 is input into the residual module to obtain feature low2. Feature low2 is then passed through the residual module to obtain feature low3. Feature low3 is upsampled to obtain feature up2 with a size of N×256. Finally, features up1 and up2 are added to obtain the output of the first layer of depth. The above operation is repeated four times to obtain the graph node features output by the hourglass network.

[0065] The residual module consists of two graph convolution modules, each of which includes a graph convolution layer, a batch normalization layer, and an activation layer. Unlike the standard convolution operation on Euclidean structured data, the purpose of graph convolution is to learn a function f(·), given that the input of the l+1th graph convolution layer is the node feature and the adjacency matrix Output is node features d l d l+1 Represents the node feature H l and H l+1 The dimension of the node features H output by the l+1 and l+2 graph convolutional layers is l+1 、H l+2 Respectively expressed as:

[0066] H l+1 =f(H l ,A) (3)

[0067] H l+2 =f(H l+1 ,A)+H l (4)

[0068] For convolutional networks, as the number of network layers increases, the features extracted become increasingly abstract. The bottom convolutional network can extract simple features such as facial contours; the middle convolutional network can extract more abstract features such as eyes and noses; and the higher-level convolutional network can extract even more abstract features, such as complete faces. The multi-scale feature recognition characteristics of the hourglass network can effectively solve the technical problems of difficult facial key point detection in cases of severe occlusion, large-angle profile, and image blur.

[0069] In order to further alleviate the technical problem of difficult facial key point detection in cases of severe occlusion, large-angle side face and image blur, the present invention makes full use of the connection between the human facial structure, groups all key points according to the facial structure, and connects each group of key points into a boundary, obtaining 13 boundaries as shown in Figure 3(b), namely, the outer contour of the face, left eyebrow, right eyebrow, nose bridge, lower boundary of the nose, upper boundary of the left eye, lower boundary of the left eye, upper boundary of the right eye, lower boundary of the right eye, upper boundary of the upper lip, lower boundary of the upper lip, upper boundary of the lower lip and lower boundary of the lower lip; Definition: The outer contour of the face is respectively connected with the left and right eyebrows, the lower boundaries of the left and right eyes and the lower boundary of the lower lip. In terms of connections, the upper boundary of the left eye is connected to the left eyebrow and the lower boundary of the left eye respectively, the upper boundary of the right eye is connected to the right eyebrow and the lower boundary of the right eye respectively, the bridge of the nose is connected to the lower boundary of the left and right eyes and the lower boundary of the nose respectively, the upper boundary of the upper lip is connected to the lower boundary of the nose and the lower boundary of the upper lip respectively, and the upper boundary of the lower lip is connected to the lower boundary of the upper lip and the lower boundary of the lower lip respectively; the boundaries are equivalent to tree nodes, and the tree nodes corresponding to the connected boundaries are connected to construct a bidirectional tree structure composed of tree structures A and B. Figures 4(a) and (b) are the tree structure A from bottom to top and the tree structure B from top to bottom respectively; the tree nodes A1~A1 in the tree structures A and B are 13 and tree nodes B1 to B 13 They represent the outer contour of the face, left eyebrow, right eyebrow, nose bridge, lower boundary of the nose, upper boundary of the left eye, lower boundary of the left eye, upper boundary of the right eye, lower boundary of the right eye, upper boundary of the upper lip, lower boundary of the upper lip, upper boundary of the lower lip and lower boundary of the lower lip respectively;

[0070] For occluded faces, visible boundaries can provide information about occluded boundaries based on the facial structure. Therefore, a message passing layer is introduced to transmit information between boundaries. Message passing is divided into intra-layer message passing and inter-layer message passing. Intra-layer message passing is used to transmit information between different boundaries, allowing information to be transferred from visible boundaries to occluded boundaries. At the same time, since different stacks of the hourglass network focus on different aspects of facial information, in order to ensure the quality of boundaries when multiple hourglass networks are cascaded, inter-layer message passing is used to pass information from lower stacks to higher stacks.

[0071] Taking the first message passing layer, that is, the message passing layer between the first hourglass network and the second hourglass network as an example, the intra-layer message passing is transmitted according to the tree structures A and B respectively; taking the tree structure A as an example, a loop operation consisting of a convolution layer, a batch normalization layer, an activation layer, an inter-layer message passing layer, and an intra-layer message passing layer is performed on each tree node; first, the first loop operation starts from the tree node A1. Since the tree node A1 is not affected by other tree nodes, the graph node feature output by the first hourglass network passes through the convolution layer to obtain a feature F with a dimension of 16×256. After the feature F passes through the batch normalization layer and the activation layer, the feature R1 is obtained. ; Feature R1 undergoes three branch operations. First, feature R1 is placed in the feature matrix result_A and the feature matrix result_A is updated. The feature matrix result_A is initially empty. Second, after feature R1 passes through the inter-layer message passing layer, the inter-layer message passing feature MsgA1 is obtained, and the inter-layer message passing feature MsgA1 is placed in the inter-layer feature matrix and the inter-layer feature matrix is updated. Third, after feature R1 passes through the intra-layer message passing layer, the intra-layer message passing feature Inner1_out is obtained. Since tree node A1 affects tree nodes A2, A3, A7, A9 and A 13 Therefore, the intra-layer message passing feature Inner1_out is placed in the adjacent tree node feature matrix tree nodes A2, A3, A7, A9 and A 13 The corresponding positions are recorded as the adjacent tree node features Msg_a2, Msg_a3, Msg_a7, Msg_a9 and Msg_a13, respectively. See Figure 5(a). The adjacent tree node feature matrix is initially empty. Then, the second loop operation is performed on the tree node A. 13 The first hourglass network outputs the node features through the convolution layer to obtain the feature F with a dimension of 16×256. The feature F is combined with the node A. 13 The adjacent tree node features Msg_a13 are added and then passed through the batch normalization layer and the activation layer to obtain the feature R2; the feature R2 undergoes three branch operations. First, the feature R2 is placed in the feature matrix result_A to update the feature matrix result_A; second, the feature R2 passes through the inter-layer message passing layer to obtain the inter-layer message passing feature MsgA13, and the inter-layer message passing feature MsgA13 is placed in the inter-layer feature matrix to update the inter-layer feature matrix; third, the feature R2 passes through the intra-layer message passing layer to obtain the intra-layer message passing feature Inner13_out; since the tree node A 13 Will affect tree node A 12 , so the intra-layer message passing feature Inner13_out is used as tree node A 12 The adjacent tree node features are placed in the adjacent tree node feature matrix tree node A 12At the corresponding position, the feature matrix of the adjacent tree node is updated, see Figure 5(b); similarly, the loop operation is performed on the remaining tree nodes to obtain the feature matrix result_A and the inter-layer feature matrix; when the loop operation is performed on the remaining tree nodes, the feature F is added to the features of the adjacent tree nodes of the tree node and then passed through the batch normalization layer and the activation layer; the same loop operation is performed on each tree node of the tree structure B to obtain the feature matrix result_B and the updated inter-layer feature matrix;

[0072] Finally, the feature matrices result_A and result_B are concatenated along the 0th dimension to obtain 13 features with a dimension of 32×256. These 13 features are respectively subjected to convolution, batch normalization, and activation operations to obtain 13 node features. The 13 node features are then concatenated along the 0th dimension to obtain a node feature with a dimension of 13×256. This node feature is further subjected to convolution, batch normalization, and activation operations to obtain an enhanced feature with a dimension of 98×256, which is the output feature of the message passing layer and serves as the input of the second hourglass network.

[0073] For inter-layer message passing, in each loop operation of the next message passing layer, the graph node features output by the hourglass network are convolved and then added to the inter-layer message passing features corresponding to the tree nodes in the inter-layer feature matrix obtained by the previous message passing layer. After that, the features are passed through the batch normalization layer and the activation layer to realize inter-layer message passing, see Figure 5(c) and (d).

[0074] Both the intra-layer message passing layer and the inter-layer message passing layer include two graph convolution operations;

[0075] 2-3. The enhanced node features are subjected to graph convolution inference in the key point generation module to predict the key point coordinate matrix;

[0076] First, a trainable parameter matrix with uniform distribution is constructed, and the dimension of the parameter matrix is 2×2×256. The enhanced node features are multiplied by the 0th and 1st dimensions of the parameter matrix respectively to obtain the key point coordinate matrices h0 and h1 with dimensions of n×2. Secondly, weights are set for all non-zero positions of the key point adjacency matrix to obtain the key point weight matrix and the adjacent key point weight matrix respectively. Finally, the key point weight matrix is multiplied by the key point coordinate matrix h0, and the adjacent key point weight matrix is multiplied by the key point coordinate matrix h1. The multiplied results are then added together. The added matrices are then passed through the batch normalization layer and the activation layer to obtain the key point coordinate matrix with dimension n×2, that is, the coordinates of the facial key points are obtained.

[0077] Step 3: Use the data set in step 1 to train the face key point detection model, and calculate the training loss through the loss function to obtain the trained face key point detection model; use the trained face key point detection model for face key point detection;

[0078] The loss function includes the relative position loss function and the wing loss function. The total loss function is defined as:

[0079]

[0080] in, represents the wing loss function, Represents the key point p i The coordinate prediction value of represents the relative position loss function, δ i Represents the key point p i The position error between it and its adjacent key points, Represents δ i The predicted value of

[0081] Generally speaking, given the key point prediction coordinates And the real coordinates P of the key points, the training goal is to minimize the error of the absolute position of the key points, that is However, by simply minimizing the absolute position error, this loss function ignores the relative positions between key points. In the face key point detection task, this relative position clue is crucial for maintaining the facial structure. In order to obtain this information during training, this paper proposes a new loss function, namely the relative position loss function; specifically, given a key point p i and his neighborhood set K i ={j:j≠i,Q ij =1}, define the key point p i The position function is:

[0082]

[0083] Among them, δ i Represents the key point p i The position error between it and its adjacent key points, ω ij Represents the key point p i Its adjacent key point p j The position weight between

[0084] The relative position loss function is to minimize the position error between the predicted value and the true value.

[0085] In order to verify the effectiveness of the proposed method, key point detection is performed on face images with different labels on the public face dataset WFLW, and compared with the existing face key point detection algorithm. The experimental results are statistically shown in Table 1. The visualization of the experimental results is shown in Table 1. Figure 6 shown.

[0086] Table 1 Experimental results of different models on the WFLW dataset

[0087]

[0088] Note: NME stands for normalized mean error, and AUC stands for accuracy.

[0089] As can be seen from the results in the table, the detection effect of the method of the present invention is better than that of the existing facial key point detection algorithm. This is because the method considers the adjacent key points of each key point according to the facial structure of the face, that is, considers the correlation between key points, selects the n most relevant key points for each key point, and generates a key point adjacency matrix; at the same time, in the feature enhancement module, the hourglass network and message passing mechanism are combined to enhance the graph node features, so that the visible boundary can provide information for the occlusion boundary according to the facial structure of the face, thereby ensuring the accuracy of detection.

[0090] The sources of various algorithms in Table 1 are:

[0091] [1]W.Wu and S.Yang, “Leveraging intra and inter-dataset variations for robust face align ment,” in Proc.IEEE Conf.Comput.Vis.PatternRecognit.Workshop, July 2017.

[0092] [2] B. Browatzki and C. Wallraven, "3fabrec: Fast few-shot face alignment by reconstructio n," in Proc.IEEE Conf.Comput.Vis.Pattern Recognit., 2020, pp.6110–6120.

[0093] [3]W.Wu,C.Qian,S.Yang,Q.Wang,Y.Cai,and Q.Zhou,“Look at boundary:Aboun dary-aware face alignment algorithm,”in Proc.IEEEConf.Comput.Vis.Pattern Recognit.,2018.

[0094] [4]Z.-H.Feng,J.Kittler,M.Awais,P.Huber,and X.Wu,“Wing loss for robustfacial lan dmark localisation with convolutional neural networks,”inProc.IEEE Conf.Comput.Vis.Pattern Recognit.,2017,pp.2235–2245.

[0095] [5]X.Liu,H.Wang,J.Zhou,and L.Tao,“Attention-guided coarse-to-finenetwork for 2d face alignment in the wild,”IEEE Access,vol.7,pp.97 196–97207,2019.

[0096] [6]S.Qian,K.Sun,W.Wu,C.Qian,and J.Jia,“Aggregation via separation:Boosting fac ial landmark detector with semi-supervised style translation,”inProc.IEEE Int.Conf.Comp ut.Vis.,2019.

[0097] [7]A.Dapogny,K.Bailly,and M.Cord,“Decafa:Deep convolutional cascadefor face alig nment in the wild,”in Proc.IEEE Int.Conf.Comput.Vis.,2019.

[0098] [8] K. Sun, Y. Zhao, B. Jiang, T. Cheng, B. Xiao, D. Liu, Y. Mu, X. Wang, W. Liu, and J. Wang, "High-resolution representations for labeling pixels and regions," CoRR, vol.abs / 1904.04514, 2019.

[0099] In the above method, the ResNet34 residual network, WFLW dataset, TensorFlow deep learning framework and Relu activation function are well-known technologies in the field.

[0100] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A facial key point detection method based on feature enhancement, characterized in that: The method comprises the following steps: Step 1: Obtain a dataset consisting of multiple face images. Each face image contains N key points, which are distributed in the facial contour, eyebrow contour, lip contour, and nose bridge position. Step 2: Construct a facial key point detection model, including a feature extraction module, a node embedding module, an adjacency matrix generation module, a feature enhancement module, and a key point generation module; 2-1. The face image is processed through the feature extraction module to obtain convolution features, and the node embedding module maps the convolution features into graph node features; 2-2. Obtain the key point adjacency matrix through the adjacency matrix generation module; First, construct the dataset into a third-order data tensor before training Represents the real number space, M represents the number of face images, and (x, y) represents the coordinates of the key points; then, the third-order data tensor T is divided into data tensors along the x and y directions and All face images form a data tensor T x and T y The horizontal and vertical coordinates of all key points of each face image constitute the data tensor T x and T y Column; according to the data tensor T x , calculate the Pearson correlation coefficient between key points along the x direction and obtain the x-direction correlation coefficient matrix Similarly, the y-direction correlation coefficient matrix is obtained Then calculate the Pearson correlation coefficient between the key points in the face image through formula (1) to obtain the relationship matrix C; Where abs(·) represents the key value of the returned matrix element; Select the n largest values in each row of the relationship matrix C to construct a sparse matrix Q, and select the n most relevant key points for each key point; normalize the sparse matrix Q using formula (2) to obtain the key point adjacency matrix; Where Q ij 、C ij Represent the elements in row i and column j in the sparse matrix Q and the relational matrix C, respectively. Indicates the largest n values in each row; 2-3. The feature enhancement module is composed of four cascaded hourglass networks, with a message passing layer inserted between two adjacent hourglass networks. The graph node features and key point adjacency matrix are input into the feature enhancement module to obtain enhanced node features. All key points are grouped according to the facial structure, and each group of key points is connected into a boundary, resulting in a total of 13 boundaries, namely, the outer contour of the face, the left eyebrow, the right eyebrow, the nose bridge, the lower boundary of the nose, the upper boundary of the left eye, the lower boundary of the left eye, the upper boundary of the right eye, the lower boundary of the right eye, the upper boundary of the upper lip, the lower boundary of the upper lip, the upper boundary of the lower lip and the lower boundary of the lower lip; Definition: The outer contour of the face is connected with the left and right eyebrows, the lower boundaries of the left and right eyes and the lower boundary of the lower lip, and the upper boundary of the left eye is connected with the left eyebrow and the left eyebrow respectively. There is a connection between the lower boundaries of the eyes, the upper boundary of the right eye is respectively related to the right eyebrow and the lower boundary of the right eye, the bridge of the nose is respectively related to the lower boundaries of the left and right eyes and the lower boundary of the nose, the upper boundary of the upper lip is respectively related to the lower boundary of the nose and the lower boundary of the upper lip, and the upper boundary of the lower lip is respectively related to the lower boundary of the upper lip and the lower boundary of the lower lip; the boundaries are equivalent to tree nodes, and the tree nodes corresponding to the related boundaries are connected to construct a bidirectional tree structure composed of tree structures A and B. The tree nodes A1~A1 in the tree structures A and B are 13 and tree nodes B1 to B 13 They represent the outer contour of the face, left eyebrow, right eyebrow, nose bridge, lower boundary of the nose, upper boundary of the left eye, lower boundary of the left eye, upper boundary of the right eye, lower boundary of the right eye, upper boundary of the upper lip, lower boundary of the upper lip, upper boundary of the lower lip and lower boundary of the lower lip respectively; Message passing is divided into intra-layer message passing and inter-layer message passing. Intra-layer messages are passed according to a bidirectional tree structure. For the message passing layer between the first hourglass network and the second hourglass network, when passing according to the tree structure A, a loop operation consisting of a convolutional layer, a batch normalization layer, an activation layer, an inter-layer message passing layer, and an intra-layer message passing layer is performed on each tree node. First, the first loop operation starts from the tree node A1. The graph node feature output by the first hourglass network passes through the convolutional layer to obtain feature F. After feature F passes through the batch normalization layer and the activation layer, feature R1 is obtained. Feature R1 passes through three divisions. The first operation is to put the feature R1 into the feature matrix result_A and update the feature matrix result_A. The feature matrix result_A is initially empty. The second operation is to get the inter-layer message passing feature MsgA1 after the feature R1 passes through the inter-layer message passing layer, and put the inter-layer message passing feature MsgA1 into the inter-layer feature matrix and update the inter-layer feature matrix. The third operation is to get the intra-layer message passing feature Inner1_out after the feature R1 passes through the intra-layer message passing layer. Since the tree node A1 will affect the tree nodes A2, A3, A7, A9 and A 13 Therefore, the intra-layer message passing feature Inner1_out is placed in the adjacent tree node feature matrix tree nodes A2, A3, A7, A9 and A 13 The corresponding positions are recorded as the adjacent tree node features Msg_a2, Msg_a3, Msg_a7, Msg_a9 and Msg_a13, respectively. The adjacent tree node feature matrix is initially empty. Then, the second loop operation is performed on the tree node A. 13 The first hourglass network outputs the node feature through the convolution layer, and the dimension is feature F. Feature F and the number of nodes A 13 The adjacent tree node features Msg_a13 are added and then passed through the batch normalization layer and the activation layer to obtain the feature R2; the feature R2 undergoes three branch operations. First, the feature R2 is placed in the feature matrix result_A to update the feature matrix result_A; second, the feature R2 passes through the inter-layer message passing layer to obtain the inter-layer message passing feature MsgA13, and the inter-layer message passing feature MsgA13 is placed in the inter-layer feature matrix to update the inter-layer feature matrix; third, the feature R2 passes through the intra-layer message passing layer to obtain the intra-layer message passing feature Inner13_out; since the tree node A 13 Will affect tree node A 12 , so the intra-layer message passing feature Innerl3_out is used as the tree node A 12 The adjacent tree node features are placed in the adjacent tree node feature matrix tree node A 12 At the corresponding position, the feature matrix of the adjacent tree node is updated; similarly, the loop operation is performed on the remaining tree nodes to obtain the feature matrix result_A and the inter-layer feature matrix; when the loop operation is performed on the remaining tree nodes, the feature F is added to the adjacent tree node features of the tree node and then passed through the batch normalization layer and the activation layer; the same loop operation is performed on each tree node of the tree structure B to obtain the feature matrix result_B and the updated inter-layer feature matrix; Finally, the feature matrices result_A and result_B are concatenated along the 0th dimension to obtain 13 features. These 13 features are subjected to convolution, batch normalization, and activation operations to obtain 13 node features. The 13 node features are then concatenated along the 0th dimension to obtain node features. This node feature is subjected to convolution, batch normalization, and activation operations once more to obtain the output features of the message passing layer. For inter-layer message passing, in each loop operation of the next message passing layer, the graph node features output by the hourglass network are convolved, and then added to the inter-layer message passing features corresponding to the tree nodes in the inter-layer feature matrix obtained by the previous message passing layer. After that, the features are passed through the batch normalization layer and the activation layer to realize inter-layer message passing. 2-3. The enhanced node features are subjected to graph convolution inference in the key point generation module to predict the key point coordinate matrix; First, a trainable parameter matrix with uniform distribution is constructed, and the enhanced node features are multiplied by the 0th and 1st dimensions of the parameter matrix respectively to obtain the key point coordinate matrices h0 and h1; secondly, weights are set for all non-zero positions of the key point adjacency matrix to obtain the key point self-weight matrix and the adjacent key point weight matrix respectively; finally, the key point self-weight matrix is multiplied by the key point coordinate matrix h0, and the adjacent key point weight matrix is multiplied by the key point coordinate matrix h1, and the multiplied results are added together. The added matrix is then passed through the batch normalization layer and the activation layer to obtain the key point coordinate matrix, that is, the coordinates of the face key points; Step 3: Use the dataset to train the facial key point detection model, and use the trained facial key point detection model for facial key point detection.

2. The method for detecting facial key points based on feature enhancement according to claim 1, wherein: Each hourglass network in the feature enhancement module has a depth of 4 layers, and the operation of each layer is the same; for the graph node features input into the hourglass network, the graph node features are input into the residual module to obtain feature up1; at the same time, the graph node features are subjected to a maximum pooling operation to obtain feature low1, feature low1 is input into the residual module to obtain feature low2, feature low2 is further passed through the residual module to obtain feature low3, feature low3 is subjected to an upsampling operation to obtain feature up2, and finally features up1 and up2 are added to obtain the output of the first layer depth of the hourglass network; the above operation is repeated four times to obtain the graph node features output by the hourglass network; The residual module consists of two graph convolution modules, each of which includes a graph convolution layer, a batch normalization layer and an activation layer.

3. The method for detecting facial key points based on feature enhancement according to claim 1, wherein: During the training process of the face key point detection model, the total loss function is defined as: in, represents the wing loss function, Represents the key point p i The coordinate prediction value of represents the relative position loss function, δ i Represents the key point p i The position error between it and its adjacent key points, Represents δ i The predicted value of Given a key point p i and his neighborhood set K i ={j:j≠i, Q ij =1}, define the key point p i The position function is: Among them, δ i Represents the key point p i The position error between it and its adjacent key points, ω ij Represents the key point p i Its adjacent key point p j The position weight between

Images