Face image recognition method and device, storage medium and electronic equipment

Abstract

The application discloses a face image recognition method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring a target face image and a first key point heat map of the target face image; determining a first image feature matrix corresponding to the target face image by using a trained first learning model; acquiring a first key point relationship matrix, and determining a first target image feature matrix corresponding to the target face image according to the first image feature matrix, the first key point heat map and the first key point relationship matrix; acquiring a second key point relationship matrix, and recognizing the target face image according to the first key point heat map, the first target image feature matrix and the second key point relationship matrix, so that the face image which is blocked or abnormal can be effectively recognized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a face image recognition method, apparatus, storage medium, and electronic device. Background Technology

[0002] Facial recognition is a biometric technology that identifies individuals based on their facial features. It involves using cameras or webcams to capture images or videos containing faces, automatically detecting and tracking faces within those images or videos, and then performing facial recognition on the detected faces. This technology is also commonly referred to as image recognition or face identification.

[0003] However, existing facial recognition technology performs poorly when the facial image is incomplete, such as when the face is severely obscured or parts of the face are not visible in the image. Summary of the Invention

[0004] This invention provides a facial image recognition method, apparatus, storage medium, and electronic device, which can effectively recognize occluded or abnormal facial images and improve the accuracy of facial recognition.

[0005] This application provides a facial image recognition method, including:

[0006] Obtain the target face image to be identified, and multiple first key point heatmaps corresponding to the target face image;

[0007] The first image feature matrix corresponding to the target face image is determined using the trained first learning model;

[0008] Obtain the first key point relationship matrix, and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap, and the first key point relationship matrix;

[0009] Obtain the second key point relationship matrix, and identify the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix.

[0010] This application also provides a facial image recognition device, including:

[0011] The acquisition module is used to acquire the target face image to be identified, and multiple first key point heat maps corresponding to the target face image;

[0012] The first determining module is used to determine the first image feature matrix corresponding to the target face image using a trained first learning model;

[0013] The second determining module is used to obtain the first key point relationship matrix and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap and the first key point relationship matrix.

[0014] The recognition module is used to obtain a second key point relationship matrix and to recognize the target face image based on the first key point heatmap, the first target image feature matrix and the second key point relationship matrix.

[0015] This application also provides a computer-readable storage medium storing a plurality of instructions adapted to be loaded by a processor to execute any of the above-described face image recognition methods.

[0016] This application also provides an electronic device, including a coupled memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to perform any of the above-described facial image recognition methods.

[0017] The facial image recognition method, apparatus, storage medium, and electronic device provided in this application acquire a target facial image to be recognized and a first keypoint heatmap corresponding to the target facial image; determine a first image feature matrix corresponding to the target facial image using a trained first learning model; acquire a first keypoint relationship matrix, and determine a first target image feature matrix corresponding to the target facial image based on the first image feature matrix, the first keypoint heatmap, and the first keypoint relationship matrix; acquire a second keypoint relationship matrix, and recognize the target facial image based on the first keypoint heatmap, the first target image feature matrix, and the second keypoint relationship matrix, thereby effectively recognizing occluded or abnormal facial images and improving the accuracy of facial recognition. Attached Figure Description

[0018] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0019] Figure 1 This is a schematic flowchart of the face image recognition method provided in the embodiments of this application.

[0020] Figure 2 This is another schematic flowchart of the face image recognition method provided in the embodiments of this application.

[0021] Figure 3 This is a schematic diagram of the structure of the facial image recognition device provided in the embodiments of this application.

[0022] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0023] Figure 5 Another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a facial image recognition method, apparatus, storage medium, and electronic device.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating the facial image recognition method provided in this application embodiment. The facial image recognition method can be applied to electronic devices such as terminal devices or servers. Terminal devices may include mobile phones, tablets, laptops, etc., or wearable devices, smart TVs, or other smart terminals. Servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud services and basic cloud computing services such as big data and artificial intelligence platforms, but are not limited to these.

[0027] In some embodiments, the specific process of the face image recognition method may include the following steps S101-S104, wherein:

[0028] S101. Obtain the target face image to be identified, and the first key point heatmap corresponding to the target face image.

[0029] The target face image can be a normal face image or a face image with severe occlusion or abnormalities. The face in the face image is mainly a human face, but it can also be other types of faces, such as the face of a certain animal; there are no restrictions here. The first keypoint heatmap corresponds to multiple facial keypoints, such as K facial keypoints. It is an image that represents the intensity (probability). The position of the pixels on it corresponds one-to-one with the position of the pixels on the target face image. In the first keypoint heatmap, the pixel value of each pixel ranges from 0 to 1. The larger the pixel value, the higher the probability value. The closer the pixel value of a pixel is to a facial keypoint, the higher the probability value of that pixel being the location of the facial keypoint, and the closer the pixel is to the facial keypoint. Conversely, the closer the pixel value of a pixel is to a facial keypoint, the lower the probability value of that pixel being the location of the facial keypoint, and the farther the pixel is from the facial keypoint.

[0030] Facial keypoints mainly refer to the keypoints that a facial keypoint detection model can identify. This model can be any existing model capable of facial keypoint recognition; no restrictions are placed here. Specifically, an existing facial keypoint detection model can be used to detect the target face image and identify the distribution of each facial keypoint (e.g., K facial keypoints) within the target face image. Typically, the facial keypoint detection model identifies the probability value of each pixel in the target face image potentially being a certain facial keypoint. These probability values ​​are then processed using softmax (normalization) to prevent noise and outliers. Finally, the normalized data is expressed as an image, resulting in a heatmap, also known as the first keypoint heatmap.

[0031] S102. Using the trained first learning model, determine the first image feature matrix corresponding to the target face image.

[0032] The first image feature matrix is ​​a matrix composed of image features of each point in the target face image, including image features of facial key points and non-facial key points. The first learning model may include a convolutional neural network, which should be pre-trained with a large number of image samples. Its training process mainly involves learning the position and category of facial key points. Here, the category mainly refers to the facial part where the facial key point is located, such as the mouth, eyes, and nose. Different facial parts may include different numbers and positions of facial key points. For example, the mouth may include 3 facial key points, and the nose may include 5 facial key points, etc.

[0033] In some implementations, please refer to Figure 2 Before step S102 above, the face image recognition method may further include the following steps S1051-S1055, wherein:

[0034] S1051. Obtain a first face image sample set, the first face image sample set including multiple first face image samples.

[0035] The first face image sample set can be obtained from a local image library or from other devices via a network. Each first face image sample is a normal image sample with an unobstructed face. Specifically, the first face image sample set can include multiple first face image samples taken of the same object, as well as multiple first face image samples taken of different objects. Generally, any two first face image samples taken of the same object are called a pair of positive sample images, and any two first face image samples taken of different objects are called a pair of negative sample images.

[0036] S1052. Using the trained facial landmark detection model, determine the first sample landmark heatmap corresponding to each first face image sample.

[0037] Similar to the method of determining the first key point heatmap in step S101 above, the face detection model can also identify the distribution of each facial key point in the first face image sample, and generate the corresponding first sample key point heatmap based on the distribution. The specific steps will not be repeated here.

[0038] S1053. Use a preset first learning model to extract features from the first face image sample to obtain the original sample image feature matrix.

[0039] The original sample image feature matrix is ​​a matrix composed of image features of each point in the first face image sample, including image features of facial key points and non-facial key points. Since the original sample image feature matrix is ​​extracted by the untrained first learning model, it can only simply convert two-dimensional images into multi-dimensional image features and cannot effectively combine facial characteristics for feature extraction, which is not conducive to feature extraction of images with occluded faces.

[0040] S1054. Based on the first sample key point heatmap and the original sample image feature matrix, determine multiple original sample key point feature vectors corresponding to multiple facial key points.

[0041] Each heatmap (e.g., the first keypoint heatmap) is equivalent to a matrix, where each value can be seen as the pixel value of a pixel on the heatmap. The original sample keypoint feature vector is the feature of a single facial keypoint, and its number is the same as the number of facial keypoints. Each facial keypoint corresponds to one original sample keypoint feature vector.

[0042] Assuming the feature matrix of the original sample image is F0, the heat map of any first sample key point is H0, and the feature vector of the original sample key point is L0, then F0 and H0 can be multiplied to extract the features of each facial key point (such as the m-th facial key point) (that is, the feature vector of the original sample key point).

[0043] S1055. Train the first learning model based on the feature vector of the original sample key points and the heat map of the first sample key points.

[0044] In this approach, considering the need for precise alignment of local features when performing face recognition with occlusion, we can first find the image location of the maximum probability value of each facial keypoint from the heatmap of the first sample keypoints. Then, we use the pixel in the first face image sample corresponding to this image location as the facial keypoint. This allows us to find each facial keypoint in the first face image sample and obtain the maximum probability value of that keypoint from the heatmap. Finally, based on this maximum probability value and the original sample keypoint feature vectors corresponding to each facial keypoint, we train the first learning model.

[0045] Specifically, the loss function Loss1 of the first learning model can be defined as follows:

[0046]

[0047] Where, β h =max(H0)∈[0,1],β h β is the maximum probability value of the h-th keypoint (facial keypoint) in the heatmap of the first sample keypoints, where h∈[1,K], K is the number of keypoints, and β k+1 =1, || + This indicates that the value is greater than 0.

[0048] Indicates feature l h and features l p The distance between them Indicates feature l h and features l n The distance between them, where l h l p and l n Let h, P, and n represent the original sample keypoint feature vectors corresponding to the facial keypoints, respectively, and let l represent the feature vectors. h and features l p Let l represent a positive sample feature pair. h and features l nThis represents a negative sample feature pair. Features of two facial keypoints belonging to the same category (i.e., the original sample keypoint feature vectors) belong to a positive sample feature pair, while features of two facial keypoints belonging to different categories (i.e., the original sample keypoint feature vectors) belong to a negative sample feature pair. That is, h and p belong to the same category of facial keypoints, while h and n belong to different categories of facial keypoints. For example, the features of any two facial keypoints in the mouth area belong to a positive sample feature pair, while the features of any one facial keypoint in the eye area and any one facial keypoint in the mouth area belong to a negative sample feature pair. lh This indicates that the first learning model applies to feature l h The resulting predicted value (i.e., the prediction value obtained by the first learning model based on feature l) h Predicting l h α is a hyperparameter representing the probability of features of facial keypoint h.

[0049] It should be noted that during the training of the first learning model, facial key points can be classified as correctly as possible, and the feature distance between related facial key points (i.e., those belonging to the same category) should be closer, while the feature distance between unrelated facial key points (i.e., those belonging to different categories) should be further apart, thereby improving the accuracy of facial key point recognition. Since the first learning model is learned based on the position and category of facial key points, it can effectively combine facial characteristics for feature extraction, accurately extracting the features of facial key points, thus improving the accuracy of subsequent face recognition.

[0050] S103. Obtain the first key point relationship matrix, and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap and the first key point relationship matrix.

[0051] The first keypoint relationship matrix represents the association between different facial keypoints in the same image. The first target image feature matrix is ​​a matrix composed of the features of all points in the target face image.

[0052] In some implementations, please refer to [link / reference]. Figure 2 The above step S103 may specifically include:

[0053] Based on the first key point heatmap and the first image feature matrix, determine multiple first key point feature vectors corresponding to multiple facial key points;

[0054] Multiply the first key point relationship matrix and each first key point feature vector to obtain the corresponding first product;

[0055] The first image feature matrix is ​​added to all the first products to obtain the first target image feature matrix corresponding to the target face image.

[0056] Here, the first keypoint feature vector is the feature of a single facial keypoint, and each facial keypoint corresponds to one first keypoint feature vector. Assuming H is the heatmap of the first keypoint and F is the first image feature matrix, the first keypoint feature vector corresponding to each keypoint can be obtained by multiplying F and H. Furthermore, assuming the first keypoint relation matrix is ​​N, and the first product corresponding to the i-th facial keypoint is R... i L i Let X be the feature vector of the first keypoint corresponding to the i-th facial keypoint, and let X be the feature matrix of the first target image that we need to obtain. Then there is a relationship between them: R i =NL i X = ΣR i +F, i∈[1,K].

[0057] It should be noted that the first keypoint relationship matrix is ​​obtained by using the second learning model to learn the correlations between different facial keypoints in the same facial image sample, and the training step of the second learning model needs to be executed after the training of the first learning model is completed. For details, please refer to [link to documentation]. Figure 2 The facial image recognition method may further include the following steps S1061-S1064, wherein:

[0058] S1061. Using the trained first learning model and the first sample key point heatmap, determine the first sample image feature matrix corresponding to each first face image sample, and the first sample key point feature vector corresponding to each face key point in the first face image sample.

[0059] The first sample image feature matrix is ​​a matrix of image features of all points in the first face image sample, and the first sample keypoint feature vector is the feature of a single face keypoint. The method for determining the first sample image feature matrix is ​​similar to the method for determining the first image feature matrix in step S102 above, and the method for determining the first sample keypoint feature vector is similar to the method for determining the original sample keypoint feature vector in step S1054 above, and will not be repeated here.

[0060] S1062. Determine the corresponding overall key point feature vector based on the first sample key point feature vector corresponding to each first face image sample.

[0061] For each first face image sample, the overall keypoint feature vector is a feature vector derived from the features of each individual facial keypoint. Each first face image sample corresponds to one overall keypoint feature vector. First, a pooling operation, such as average pooling, is performed on all the first sample keypoint feature vectors corresponding to each first face image sample to obtain a feature matrix L1. The purpose of pooling is to improve important feature information, compress features, and reduce the amount of data computation. Then, a global average pooling operation is performed on the feature matrix L1 to obtain the overall keypoint feature vector corresponding to the corresponding first face image sample, i.e., the overall keypoint feature vector. Among them, l ij ∈L1, K is the number of facial key points, l ij Let be the eigenvector of the i-th row and j-th column in L1.

[0062] S1063. Determine the difference feature matrix corresponding to each first face image sample based on the feature vector of the first sample key points and the feature vector of the overall key points.

[0063] Specifically, for each first facial image sample corresponding to all first sample keypoint feature vectors, the difference between each first sample keypoint feature vector and the overall keypoint feature vector can be calculated first, and its absolute value can be obtained. Then, these absolute values ​​are normalized and fully connected to obtain a difference characteristic matrix. That is, through the formula D=f(BN(|l 1g -G|)) calculate the difference feature matrix D, where, l 1g Let be the feature vector of the first sample keypoint of the g-th facial keypoint. BN() is used for normalization, and f() is used for fully connected processing. 1g The larger the value of -G|, the greater the difference between the feature information of a single keypoint and the feature information of the overall keypoint, the less referential it is, and the more likely it is abnormal information, which is more likely to be occluded local information. The smaller the value, the more similar the feature information of a single keypoint is to the feature information of the overall keypoint, that is, the more important it is.

[0064] S1064. Determine the first key point relationship matrix based on the first sample key point feature vector, the first sample image feature matrix, the difference feature matrix, and the first sample key point heatmap.

[0065] In some implementations, step S1064 may further include:

[0066] An initial node relationship matrix is ​​determined based on the heat map of the first sample key points, and the feature vectors of the first sample key points and the overall key point feature vectors are used as graph nodes. At the same time, a graph convolutional neural network is constructed based on the initial node relationship matrix to obtain a second learning model.

[0067] The second learning model is trained based on the difference feature matrix, the first sample image feature matrix, the first sample key point feature vector, and the first sample key point heatmap.

[0068] The node relationship matrix in the trained second learning model is used as the first key point relationship matrix.

[0069] The second learning model is a graph convolutional neural network. For any first face image sample, the feature vectors l of the K first sample key points corresponding to its K facial key points can be generated. 11 , l 12 …l 1k The first sample keypoint heatmap H0 is used as the matrix of the first sample keypoint heatmap H0, and the matrix of the first sample keypoint heatmap H0 is used as the initial node relationship matrix P between the graph nodes.

[0070] During the training of the second learning model, the difference feature matrix D is input into the second learning model, i.e., f(D,P)=σ(PDW), where W is the weight parameter matrix and σ(·) is the activation function. This allows the model to obtain information about different facial key points through information transmission between graph nodes and learn the correlation between different facial key points. Then, based on the node relationship matrix P, the feature matrix F1 of the first sample image, and the feature vector l of the first sample key points, the model learns the relationship between different facial key points. 11 , l 12 …l 1k To determine a feature matrix X0 of a predicted sample image, for example, the feature matrix X0 of the predicted sample image satisfies the formula: R 1i =Pl 1i X0 = ΣR 1i +F1, i∈[1,K],l 1i Let X0 be the feature vector of the first sample key point of the i-th facial key point. Then, the feature matrix X0 of the predicted sample image and the heat map H0 of the first sample key point are substituted into the loss function of the second learning model. The node relationship matrix P of the second learning model is adjusted in reverse through iteration, thereby realizing model training.

[0071] The loss function of the second learning model can be similar to the loss function Loss1 of the first learning model. In the actual calculation process, the input of the loss function of the second learning model is changed to the feature vector corresponding to each facial key point in the feature matrix X0 of the predicted sample image and the maximum probability value corresponding to each facial key point in the heat map H0 of the first sample key point. The specific calculation method can refer to the calculation method of the loss function Loss1 mentioned above, which will not be repeated here.

[0072] S104. Obtain the second key point relationship matrix, and identify the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix.

[0073] The second keypoint relationship matrix represents the association between the same facial keypoints in different images.

[0074] The recognition of target facial images is generally based on a preset facial image library. The preset facial image library can store a large number of unoccluded facial images with known identities. The identity of the target facial image is recognized by matching the target facial image with the facial images in the preset facial image library.

[0075] In some implementations, the above step "identifying the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix" may specifically include the following steps S1041-S1044, wherein:

[0076] S1041. Determine the second target image feature matrix and the second key point heatmap corresponding to the preset face image.

[0077] The preset face image is any face image from the aforementioned preset face image library. The method for determining the feature matrix of the second target image is similar to the method for determining the feature matrix of the first target image, as detailed in steps S101-S103 above. The method for determining the heatmap of the second key point is similar to the method for determining the heatmap of the first key point, as detailed in step S101 above, and will not be repeated here.

[0078] S1042. Based on the first target image feature matrix and the second key point relationship matrix, determine the first comparison feature matrix corresponding to the target face image, and based on the second target image feature matrix and the second key point relationship matrix, determine the second comparison feature matrix corresponding to the preset face image.

[0079] The first comparison feature matrix and the second comparison feature matrix are matrices composed of features corresponding to all points in the target face image and the preset face image, respectively. The determination methods for both are similar; here, the determination method for the first comparison feature matrix is ​​used as an example. Specifically, "determining the first comparison feature matrix corresponding to the target face image based on the first target image feature matrix and the second key point relationship matrix" may include:

[0080] Multiply the second key point relationship matrix and the first target image feature matrix to obtain the second product;

[0081] The second product is added to the first target image feature matrix to obtain the first comparison feature matrix corresponding to the target face image.

[0082] Here, assuming the second keypoint relationship matrix is ​​U, the first target image feature matrix is ​​X1, and the first comparison feature matrix is ​​V1, then...

[0083] S1043. Determine the similarity between the target face image and the preset face image based on the first comparison feature matrix, the second comparison feature matrix, the first key point heatmap, and the second key point heatmap.

[0084] The similarity S between the target face image and the preset face image can be calculated using the following similarity calculation formula.

[0085]

[0086] Where, β 1m ,β 2m Let m represent the maximum probability value of facial keypoint m in the first keypoint heatmap and the second keypoint heatmap, respectively, where K is the number of facial keypoints, and v is the maximum probability value of facial keypoint m in the first keypoint heatmap and the second keypoint heatmap, respectively. 1m v 2m The table represents the feature vector of facial key point m in the first comparison feature matrix V1 and the second comparison feature matrix V2. 1m ∈V1, v 2m ∈V2.

[0087] S1044. Identify the target face image based on the similarity.

[0088] When the similarity is greater than or equal to a certain threshold, such as 0.8, it can be considered that the target face image and the preset face image were taken of the same object and have the same identity. When the similarity is less than the threshold, it can be considered that the target face image and the preset face image have different identities. In this case, other preset face images can be obtained from the preset face image library to re-identify the target face image.

[0089] It is easy to understand that the second keypoint relationship matrix is ​​obtained by using the third learning model to learn the correlation between the same facial keypoints in different image samples. Similar to the first and second learning models, the third learning model is also trained in advance using a large number of image samples. That is, before step S104 above, the third learning model needs to be trained.

[0090] The training samples for the third learning model can be the same as or different from those for the first two learning models. Furthermore, the training steps for the third learning model must be performed after the second learning model has been trained. For details, please refer to [link to relevant documentation]. Figure 2 Before obtaining the second keypoint relationship matrix, the face image recognition method may further include the following steps S1071-S1075, wherein:

[0091] S1071. Obtain multiple pairs of second face image samples, and the second sample key point heatmap corresponding to each second face image sample.

[0092] Each pair of second face image samples includes two second face image samples, which can be taken from the same object or from different objects. The second sample keypoint heatmap can also be obtained using the facial keypoint detection model described above.

[0093] S1072. Based on the trained first learning model, the second sample key point heatmap, and the first key point relationship matrix, determine the target sample image feature matrix corresponding to each second face image sample.

[0094] In some implementations, step S1072 may specifically include:

[0095] The second sample image feature matrix corresponding to each second face image sample is determined based on the trained first learning model;

[0096] Based on the second sample image feature matrix, the second sample key point heatmap, and the first key point relationship matrix, the target sample image feature matrix corresponding to each second face image sample is determined.

[0097] Each second face image sample can be input into the trained first learning model to obtain the corresponding second sample image feature matrix. Then, the target sample image feature matrix can be determined in a manner similar to that used in step S103 above. This will not be elaborated further here.

[0098] S1073. Based on the preset objective function and the target sample image feature matrix, determine the prediction comparison feature matrix corresponding to each second face image sample, wherein the objective function includes a matching relationship matrix variable.

[0099] The objective function is defined as: U * =argmax U U T U is a matching relation matrix variable with an initial value.

[0100] Where st||U||=1 (that is, the norm of U is restricted to 1). U∈[0,1] K×K U represents the relationship between facial keypoints in two facial image samples, where K is the number of facial keypoints. For any pair of second facial image samples, U i，j X represents 1i and X 2j The degree of matching between them, where X 1i X represents the feature of the i-th facial key point in one of the second facial image samples. 1i ∈X1, where X1 is the feature matrix of the target sample image of the corresponding second face image sample, X 2j X represents the feature of the j-th facial keypoint in another second facial image sample. 2j ∈X2, where X2 is the feature matrix of the target sample image of the corresponding second face image sample.

[0101] The target sample image feature matrix X1X2 can be multiplied by the initial value of U to extract the related features in X1 and X2. Then, the extracted features are added to X1 and X2 respectively to obtain feature matrices V1 and V2 containing the related features in the two second face image samples. V1 and V2 are the prediction comparison feature matrices.

[0102] S1074. Based on the predicted comparison feature matrix and the second sample key point heat map, determine the predicted similarity between each pair of second face image samples.

[0103] The method for calculating similarity during the training of the third model is the same as the method for calculating similarity between the target face image and the preset face image in step S1043 above, and will not be repeated here.

[0104] S1075. The third learning model and the objective function are jointly trained based on the predicted similarity, and the values ​​of the matching relationship matrix variables after joint training are used as the second key point relationship matrix.

[0105] The loss function of the third learning model can be:

[0106] Loss3=ylogs+(1-y)log(1-s),

[0107] Where y is the label value for whether any pair of second face image samples belong to the same object, and S is the predicted similarity mentioned above.

[0108] By jointly training the objective function and the third learning model, the objective function can be optimized simultaneously as the loss function converges, so that the matching relation matrix variable U has a final optimized value, which is then used as the second key point relation matrix.

[0109] As described above, the face image recognition method provided in this embodiment acquires a target face image to be recognized and a first keypoint heatmap corresponding to the target face image; determines a first image feature matrix corresponding to the target face image based on a trained first learning model; acquires a first keypoint relationship matrix, and determines a first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first keypoint feature matrix, and the first keypoint relationship matrix; acquires a second keypoint relationship matrix, and recognizes the target face image based on the first keypoint heatmap, the first target image feature matrix, and the second keypoint relationship matrix, thereby effectively recognizing occluded or abnormal face images and improving the accuracy of face recognition.

[0110] Based on the methods described in the above embodiments, this embodiment will further describe the face image recognition device from the perspective of the face image recognition device. The face image recognition device can be implemented as an independent entity and can be applied to electronic devices such as servers or terminal devices. The mobile terminal can include mobile phones, tablets, wearable devices, etc., and the server can be a server that can provide face recognition function.

[0111] Please see Figure 3 , Figure 3 This application provides a specific description of a facial image recognition device, which may include: an acquisition module 10, a first determination module 20, a second determination module 30, and a recognition module 40, wherein:

[0112] (1) Obtain module 10

[0113] The acquisition module 10 is used to acquire the target face image to be identified, and multiple first key point heat maps corresponding to the target face image.

[0114] (2) First Determination Module 20

[0115] The first determining module 20 is used to determine the first image feature matrix corresponding to the target face image using a trained first learning model.

[0116] In some embodiments, the face image recognition device further includes a training module for:

[0117] Before the first determining module 20 uses the trained first learning model to determine the first image feature matrix corresponding to the target face image, a first face image sample set is obtained, which includes multiple first face image samples.

[0118] Using a trained facial landmark detection model, determine the first sample landmark heatmap corresponding to each of the first facial image samples;

[0119] The first face image sample is subjected to feature extraction using a preset first learning model to obtain the original sample image feature matrix;

[0120] Based on the heat map of the first sample key points and the feature matrix of the original sample image, determine the feature vectors of multiple original sample key points corresponding to multiple facial key points;

[0121] The first learning model is trained based on the original sample key point feature vector and the first sample key point heatmap.

[0122] (3) Second determination module 30

[0123] The second determining module 30 is used to obtain the first key point relationship matrix and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap and the first key point relationship matrix.

[0124] In some implementations, the second determining module 30 is specifically used for:

[0125] Based on the first key point heatmap and the first image feature matrix, determine multiple first key point feature vectors corresponding to multiple facial key points;

[0126] Multiply the first key point relationship matrix and each first key point feature vector to obtain the corresponding first product;

[0127] The first image feature matrix is ​​added to all the first products to obtain the first target image feature matrix corresponding to the target face image.

[0128] In some implementations, the training module can also be used for:

[0129] Before the second determining module 30 obtains the first key point relationship matrix, the first sample image feature matrix corresponding to each first face image sample and the first sample key point feature vector corresponding to each face key point in the first face image sample are determined by using the trained first learning model and the first sample key point heat map.

[0130] Based on the first sample key point feature vector corresponding to each first face image sample, determine the corresponding overall key point feature vector.

[0131] The difference feature matrix corresponding to each of the first face image samples is determined based on the feature vector of the first sample key points and the feature vector of the overall key points.

[0132] The first key point relationship matrix is ​​determined based on the first sample key point feature vector, the first sample image feature matrix, the difference feature matrix, and the first sample key point heatmap.

[0133] In some implementations, the training module can further be used to:

[0134] The initial node relationship matrix is ​​determined based on the heat map of key points of the first sample;

[0135] Using the feature vectors of the first sample key points and the overall key point feature vectors as graph nodes, and constructing a graph convolutional neural network based on the initial node relationship matrix, a second learning model is obtained.

[0136] The second learning model is trained based on the difference feature matrix, the first sample image feature matrix, the first sample key point feature vector, and the first sample key point heatmap.

[0137] The node relationship matrix in the trained second learning model is used as the first key point relationship matrix. (4) Recognition module 40

[0138] The recognition module 40 is used to obtain the second key point relationship matrix and to recognize the target face image based on the first key point heat map, the first target image feature matrix and the second key point relationship matrix.

[0139] In some embodiments, the identification module 40 is specifically used for:

[0140] Determine the feature matrix of the second target image corresponding to the preset face image, and the heat map of the second key points;

[0141] Based on the first target image feature matrix and the second key point relationship matrix, a first comparison feature matrix corresponding to the target face image is determined, and based on the second target image feature matrix and the second key point relationship matrix, a second comparison feature matrix corresponding to the preset face image is determined.

[0142] Based on the first comparison feature matrix, the second comparison feature matrix, the first key point heatmap, and the second key point heatmap, the similarity between the target face image and the preset face image is determined;

[0143] The target face image is identified based on the similarity.

[0144] In some embodiments, the identification module 40 described above can further be used for:

[0145] Multiply the second key point relationship matrix and the first target image feature matrix to obtain the second product;

[0146] The second product is added to the first target image feature matrix to obtain the first comparison feature matrix corresponding to the target face image.

[0147] In some implementations, the training module can also be used for:

[0148] Before the recognition module 40 obtains the second key point relationship matrix, it obtains multiple pairs of second face image samples and the second sample key point heat map corresponding to each second face image sample.

[0149] Based on the trained first learning model, the second sample key point heatmap, and the first key point relationship matrix, determine the target sample image feature matrix corresponding to each second face image sample;

[0150] Based on the preset objective function and the target sample image feature matrix, a prediction comparison feature matrix is ​​determined for each second face image sample, wherein the objective function includes a matching relationship matrix variable;

[0151] Based on the predicted comparison feature matrix and the heat map of the key points of the second sample, the predicted similarity between each pair of the second face image samples is determined;

[0152] The third learning model and the objective function are jointly trained based on the predicted similarity.

[0153] The numerical values ​​of the matching relationship matrix variables after joint training are used as the second keypoint relationship matrix.

[0154] In some implementations, the training module can further be used to:

[0155] The second sample image feature matrix corresponding to each second face image sample is determined based on the trained first learning model;

[0156] Based on the second sample image feature matrix, the second sample keypoint heatmap, and the first keypoint relationship matrix, the target sample image feature matrix corresponding to each second face image sample is determined. In specific implementation, the above modules can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0157] In addition, this application also provides an electronic device, which may be a smartphone, tablet computer, or other similar device. Figure 4 As shown, the electronic device 200 includes a processor 201 and a memory 202. The processor 201 and the memory 202 are electrically connected.

[0158] The processor 201 is the control center of the electronic device 200. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 202 and calling the data stored in the memory 202, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0159] In this embodiment, the processor 201 in the electronic device 200 loads the instructions corresponding to the processes of one or more application programs into the memory 202 according to the following steps, and the processor 201 runs the application programs stored in the memory 202 to realize various functions:

[0160] Obtain the target face image to be identified, and the first key point heatmap corresponding to the target face image;

[0161] The first image feature matrix corresponding to the target face image is determined using the trained first learning model;

[0162] Obtain the first key point relationship matrix, and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap, and the first key point relationship matrix;

[0163] Obtain the second key point relationship matrix, and identify the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix.

[0164] Figure 5 Another schematic diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the facial image recognition method provided in the above embodiments. The electronic device may include a smartphone or a server.

[0165] The electronic device may include a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a radio frequency (RF) circuit 303, a power supply 304, an input unit 305, and a display unit 306, etc. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0166] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor may include one or more processing cores; preferably, the processor may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor.

[0167] The memory 302 can be used to store software programs (computer programs) and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0168] RF circuit 303 can be used for signal reception and transmission during information transmission and reception. Specifically, it receives downlink information from the base station and hands it over to one or more processors 301 for processing; additionally, it transmits uplink data to the base station. Typically, RF circuit 303 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 303 can also communicate wirelessly with networks and other devices. This wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile Communications), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, and SMS (Short Messaging Service).

[0169] The electronic device also includes a power supply 304 (such as a battery) that supplies power to various components. Preferably, the power supply 304 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 304 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0170] The electronic device may also include an input unit 305, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, the input unit 305 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 301, and can receive and execute commands from the processor 301. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touch-sensitive surface. In addition to the touch-sensitive surface, the input unit 305 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0171] The electronic device may further include a display unit 306, which can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit 306 includes multiple hardware display processing units, a video frame processing module, a display screen, etc. The multiple hardware display processing units and the video frame processing module can be integrated into a processing chip. The display screen may include a display panel, optionally configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event. Subsequently, the processor 301 provides corresponding visual output on the display panel according to the type of touch event. Although in the figures, the touch-sensitive surface and the display panel are implemented as two separate components to achieve input and output functions, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve input and output functions.

[0172] Although not shown, the electronic device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows:

[0173] Obtain the target face image to be identified, and the first key point heatmap corresponding to the target face image;

[0174] The first image feature matrix corresponding to the target face image is determined using the trained first learning model;

[0175] Obtain the first key point relationship matrix, and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap, and the first key point relationship matrix;

[0176] Obtain the second key point relationship matrix, and identify the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix.

[0177] This electronic device can implement the steps of any embodiment of the face image recognition method provided in this application. Therefore, it can achieve the beneficial effects that any face image recognition method provided in this application can achieve. For details, please refer to the previous embodiments, which will not be repeated here.

[0178] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions (computer programs), or by instructions (computer programs) controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps of any embodiment of the facial image recognition method provided by the present invention.

[0179] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0180] Since the computer program stored in the storage medium can execute the steps in any of the face image recognition method embodiments provided in the present invention, the beneficial effects that any face image recognition method provided in the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0181] The foregoing has provided a detailed description of a facial image recognition method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A facial image recognition method, characterized in that, include: Obtain the target face image to be identified, and the first key point heatmap corresponding to the target face image; The first image feature matrix corresponding to the target face image is determined using the trained first learning model; Obtain the first key point relationship matrix, and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap and the first key point relationship matrix, wherein the first key point relationship matrix represents the association between different facial key points in the same image; A second key point relationship matrix is ​​obtained, and the target face image is identified based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix. The second key point relationship matrix represents the association between the same facial key points in different images.

2. The facial image recognition method according to claim 1, characterized in that, Before determining the first image feature matrix corresponding to the target face image using the trained first learning model, the method further includes: Obtain a first face image sample set, which includes multiple first face image samples; Using a trained facial landmark detection model, determine the first sample landmark heatmap corresponding to each of the first facial image samples; The first face image sample is subjected to feature extraction using a preset first learning model to obtain the original sample image feature matrix; Based on the heat map of the first sample key points and the feature matrix of the original sample image, determine the feature vectors of multiple original sample key points corresponding to multiple facial key points; The first learning model is trained based on the feature vector of the key points of the original sample and the heat map of the key points of the first sample.

3. The facial image recognition method according to claim 2, characterized in that, Before obtaining the first keypoint relationship matrix, the following steps are also included: Using the trained first learning model and the first sample key point heatmap, determine the first sample image feature matrix corresponding to each first face image sample, and the first sample key point feature vector corresponding to each face key point in the first face image sample. Based on the first sample key point feature vector corresponding to each first face image sample, determine the corresponding overall key point feature vector. Based on the first sample key point feature vector and the overall key point feature vector, determine the difference feature matrix corresponding to each first face image sample; The first key point relationship matrix is ​​determined based on the first sample key point feature vector, the first sample image feature matrix, the difference feature matrix, and the first sample key point heatmap.

4. The facial image recognition method according to claim 3, characterized in that, The step of determining the first keypoint relationship matrix based on the first sample keypoint feature vector, the first sample image feature matrix, the difference feature matrix, and the first sample keypoint heatmap includes: The initial node relationship matrix is ​​determined based on the heat map of key points of the first sample; Using the feature vectors of the first sample key points and the overall key point feature vectors as graph nodes, and constructing a graph convolutional neural network based on the initial node relationship matrix, a second learning model is obtained. The second learning model is trained based on the difference feature matrix, the first sample image feature matrix, the first sample key point feature vector, and the first sample key point heatmap. The node relationship matrix in the trained second learning model is used as the first key point relationship matrix.

5. The facial image recognition method according to claim 1, characterized in that, Before obtaining the second keypoint relationship matrix, the following steps are also included: Obtain multiple pairs of second face image samples, and a heatmap of key points of the second sample corresponding to each second face image sample; Based on the trained first learning model, the second sample key point heatmap, and the first key point relationship matrix, determine the target sample image feature matrix corresponding to each second face image sample; Based on the preset objective function and the target sample image feature matrix, a prediction comparison feature matrix is ​​determined for each second face image sample, wherein the objective function includes a matching relationship matrix variable; Based on the predicted comparison feature matrix and the heat map of the key points of the second sample, the predicted similarity between each pair of the second face image samples is determined; The third learning model and the objective function are jointly trained based on the predicted similarity. The numerical values ​​of the matching relationship matrix variables after joint training are used as the second keypoint relationship matrix.

6. The facial image recognition method according to claim 5, characterized in that, The step of determining the target sample image feature matrix corresponding to each second face image sample based on the trained first learning model, the second sample key point heatmap, and the first key point relationship matrix includes: The second sample image feature matrix corresponding to each second face image sample is determined based on the trained first learning model; Based on the second sample image feature matrix, the second sample key point heatmap, and the first key point relationship matrix, the target sample image feature matrix corresponding to each second face image sample is determined.

7. The face image recognition method according to claim 1, characterized in that, The step of determining the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap, and the first key point relationship matrix includes: Based on the first key point heatmap and the first image feature matrix, determine multiple first key point feature vectors corresponding to multiple facial key points; Multiply the first key point relationship matrix and each first key point feature vector to obtain the corresponding first product; The first image feature matrix is ​​added to all the first products to obtain the first target image feature matrix corresponding to the target face image.

8. The facial image recognition method according to claim 1, characterized in that, The step of identifying the target face image based on the first key point heatmap, the first target image feature matrix, and the second key point relationship matrix includes: Determine the feature matrix of the second target image corresponding to the preset face image, and the heat map of the second key points; Based on the first target image feature matrix and the second key point relationship matrix, a first comparison feature matrix corresponding to the target face image is determined, and based on the second target image feature matrix and the second key point relationship matrix, a second comparison feature matrix corresponding to the preset face image is determined. Based on the first comparison feature matrix, the second comparison feature matrix, the first key point heatmap, and the second key point heatmap, the similarity between the target face image and the preset face image is determined; The target face image is identified based on the similarity.

9. The facial image recognition method according to claim 8, characterized in that, The step of determining the first comparison feature matrix corresponding to the target face image based on the first target image feature matrix and the second key point relationship matrix includes: Multiply the second key point relationship matrix and the first target image feature matrix to obtain the second product; The second product is added to the first target image feature matrix to obtain the first comparison feature matrix corresponding to the target face image.

10. A facial image recognition device, characterized in that, include: The acquisition module is used to acquire the target face image to be identified and the first key point heat map corresponding to the target face image; The first determining module is used to determine the first image feature matrix corresponding to the target face image using a trained first learning model; The second determining module is used to obtain the first key point relationship matrix and determine the first target image feature matrix corresponding to the target face image based on the first image feature matrix, the first key point heatmap and the first key point relationship matrix, wherein the first key point relationship matrix represents the association between different facial key points in the same image. The recognition module is used to obtain a second key point relationship matrix and to recognize the target face image based on the first key point heatmap, the first target image feature matrix and the second key point relationship matrix, wherein the second key point relationship matrix represents the association between the same facial key points in different images.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor to perform the face image recognition method of any one of claims 1 to 9.

12. An electronic device, characterized in that, The method includes a coupled memory and a processor, the memory storing a computer program, and the processor running the computer program in the memory to perform the steps of the face image recognition method according to any one of claims 1 to 9.

Images