Cross-domain key point detection method compatible with real faces and comic faces

By establishing a qualitative face shape model and training related detectors, the problem that real face detection methods cannot adapt to comic faces is solved, and cross-domain key point detection of real faces and comic faces is realized under the single source domain. It has stronger generalization ability and robustness.

CN115147886BActive Publication Date: 2024-12-06CENT SOUTH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110343531.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-30
Publication Date
2024-12-06
Estimated Expiration
2041-03-30

AI Technical Summary

Technical Problem

The existing real face key point detection methods cannot effectively adapt to the comic face domain and cannot effectively generalize to the entire comic face domain. The domain adaptive method requires a large number of target domain images, which is difficult to obtain.

Method used

Establish a qualitative face shape model (QSM model), train five sense detectors, local key point detectors and overall key point fine-tuners, perform initial detection through five sense detectors, use the QSM model to screen candidate five sense boxes, and local key point detectors to detect Key points and adjust the position through the overall key point spinner.

Benefits of technology

It realizes cross-domain key point detection of real faces and comic faces in the single source domain, has stronger generalization ability, and can robustly detect comic faces of different exaggerated styles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115147886B_ABST
    Figure CN115147886B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-domain key point detection method compatible with real human faces and comic human faces. By establishing a qualitative shape model of human faces, and training a facial feature detector, a local key point detector and an overall key point fine-tuner, the facial feature detector is used to perform initial detection of the facial feature positions on the facial image to obtain candidate facial feature frames, and then the QSM model is used for screening to determine the correct facial feature frames, and then the local key point detector is used to detect the key points of the facial features respectively, and the overall key point fine-tuner is used to adjust the key point positions to obtain the key points. The technical effect of the present invention is that the present method belongs to a single source domain generalization method, and only requires the real human face domain as the source domain. Compared with other detection methods, the present method has stronger generalization ability, can simultaneously perform key point detection on real human faces and comic human faces, and still ensures strong robustness for comic human faces of different exaggerated styles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and in particular to a cross-domain key point detection method compatible with real faces and cartoon faces. Background Art

[0002] Facial key point detection is a key step in many face-related applications, such as facial expression recognition, head posture estimation, etc., which usually rely on the facial shape information provided by the key point position. Cartoons are an artistic expression of portraits. Usually, some unique features of characters are exaggerated in cartoons to create humorous or sarcastic effects. Most of the local facial features of cartoon faces have a strong sense of reality, but their scale, position, aspect ratio, etc. are exaggerated and varied. Cartoon face key point detection has a wide range of applications in many fields, such as cartoon detection and cartoon editing.

[0003] At present, there are three types of facial key point detection methods, namely the traditional methods based on ASM (Active Shape Model) and AAM (Active Appearance Model); the methods based on cascade shape regression, such as ERT (Ensemble of Regression Trees); and the methods based on deep learning, such as DAN (Deep Alignment Network) and FAN (Face Alignment Network). If divided by whether they are parameterized or not, they are divided into parametric methods and non-parametric methods. ASM, AAM and CLM (Constrained Local Models) are parametric methods, while cascade regression and deep learning methods are non-parametric methods. The popular method now is to use deep CNN to learn facial features and regressors, and gradually correct the estimated key points, such as the ECT (Estimation-Correction-Tuning) method; or to take advantage of the end-to-end advantages of the deep CNN model to learn robust facial key point heat maps, or to improve the accuracy of feature detection through multi-task learning, such as the MTCNN (Multi-task Cascaded Convolutional Networks) method. Although existing real face key point detection methods can achieve high detection accuracy for real faces, they cannot adapt to the comic face domain because there are large differences between the scales, positions and aspect ratios of the facial features in these two domains. At the same time, comic faces are more diverse in these aspects, so existing real face key point detection methods cannot effectively generalize from real faces to comic faces.

[0004] Yaniv et al. studied the facial key point detection tasks in different domains and proposed a method based on ECT called ECpTp+A. This method augmented the real face dataset by texture augmentation and geometric augmentation, and then used the ECT method for supervised training on the augmented dataset. Texture augmentation was completed using the neural-style-transfer method using the large-scale art painting dataset WikiArt; geometric augmentation randomly transformed the position, scale, and aspect ratio of local facial features. Since the ECpTp+A method uses the image information of the target domain for data augmentation without using annotation information, this method belongs to the unsupervised domain adaptation method for facial key point detection tasks. Domain adaptation methods can improve the generalization ability of the model in the target domain. These methods can solve the differences in shape, scale, and position between the real face domain and the comic face domain, but they need to be re-adapted when facing comics of different styles, and cannot be effectively generalized to the entire comic face domain; in addition, domain adaptation methods require a large number of target domain images, which are difficult to obtain under conditions such as privacy and copyright.

[0005] Therefore, in this context, studying a cross-domain key point detection method that is compatible with real faces and cartoon faces can effectively improve the generalization ability of the key point detection model in different domains. Summary of the invention

[0006] In order to achieve the above technical purpose, the technical solution of the present invention is:

[0007] A cross-domain key point detection method compatible with real faces and cartoon faces includes the following steps:

[0008] Step A, building a qualitative face shape model, namely the QSM model, and training the facial features detector, local key point detector and global key point fine-tuner;

[0009] Step B, using a facial feature detector to perform an initial detection of facial feature positions on the face image to obtain candidate facial feature frames, and then using a QSM model to screen and determine the correct facial feature frames;

[0010] Step C, using a local key point detector to detect the facial features respectively to obtain key points;

[0011] In step D, use the global key point fine-tuning tool to adjust the key point position.

[0012] In the method, in step A, establishing a qualitative face shape model comprises the following steps:

[0013] Step A1, dividing the facial features into four independent features, namely the left eye including the left eyebrow, the right eye including the right eyebrow, the nose and the mouth; calculating the coordinates of the center points of each facial feature in the facial image, and determining the up-down-left-right qualitative spatial relationship attributes between the facial features;

[0014] Step A2: Generate facial features samples according to the up, down, left, and right qualitative spatial relationship attributes, and use the current best hypothesis algorithm to perform inductive learning on all generated samples to obtain the initial first-order logic definition of facial features;

[0015] Step A3, use the three primary colors similarity method and combine the up, down, left, and right qualitative spatial relationships to perform symmetry detection; convert the symmetry detection results into symmetry qualitative spatial relationship attributes; add the symmetry qualitative spatial relationship attributes to the first-order logic definition of the five senses learned in step A2 in a disjunctive manner, thereby forming a qualitative face shape model, namely the QSM model.

[0016] The method, step A1 comprises:

[0017] Step A11, for each sample in the real face dataset, based on the key point labels provided by the sample, find the minimum horizontal coordinate x1, the minimum vertical coordinate y1, the maximum horizontal coordinate x2 and the maximum vertical coordinate y2 of the key points contained in each facial feature; calculate the center point (x0, y0) of each facial feature:

[0018] x0=(x1+x2) / 2

[0019] y0=(y1+y2) / 2

[0020] Step A12, according to the coordinates of the center points of the facial features, determine the up-down-left-right qualitative spatial relationship attributes of all facial features with other facial features, including four attributes for describing the horizontal qualitative spatial relationship, namely horiLE, horiRE, horiN and horiM, where horiLE represents the horizontal relationship between the current facial feature and the left eye, horiRE represents the horizontal relationship between the current facial feature and the right eye, horiN represents the horizontal relationship between the current facial feature and the nose, and horiM represents the horizontal relationship between the current facial feature and the mouth. The attribute value is left, which means the current facial feature is on the left side of the corresponding facial feature; equal, which means the current facial feature is at the same horizontal position as the corresponding facial feature; or right, which means the current facial feature is on the right side of the corresponding facial feature.

[0021] It also includes four attributes, vertLE, vertRE, vertN and vertM, which are used to describe vertical qualitative spatial relationships. vertLE represents the vertical relationship between the current facial feature and the left eye, vertRE represents the vertical relationship between the current facial feature and the right eye, vertN represents the vertical relationship between the current facial feature and the nose, and vertM represents the vertical relationship between the current facial feature and the mouth. The attribute values ​​are up, indicating that the current facial feature is above the corresponding facial feature, equal, indicating that the current facial feature is at the same vertical position as the corresponding facial feature, or down, indicating that the current facial feature is below the corresponding facial feature.

[0022] The method, step A2 comprises:

[0023] Step A21, for each sample of the real face data set, generate positive examples and negative examples for inductive learning for each facial feature x: the set of facial features other than facial feature x in the face is called set, and the facial feature x is conjuncted with the up-down-left-right qualitative spatial relationship attributes of each facial feature in the set to obtain positive examples of facial feature x; for each facial feature in the set, the up-down-left-right qualitative spatial relationship attributes of each facial feature in the set are also conjuncted to obtain negative examples of facial feature x;

[0024] Step A22, for each of the five sense organs, use the Current-best-hypothesis algorithm to perform inductive learning on all the positive and negative examples of the five sense organs, and the initial hypothesis is randomly set to one of the positive examples of the five sense organs. If a false negative example is encountered during learning, that is, the current hypothesis is judged to be a negative example, but it is actually a positive example, then the current hypothesis is generalized; if a false positive example is encountered, that is, the current hypothesis is judged to be a positive example, but it is actually a negative example, then the current hypothesis is narrowed; if a true positive example and a true negative example are encountered, the current hypothesis is maintained unchanged; after the inductive learning is completed, the initial first-order logic definition of each of the five sense organs is obtained.

[0025] The method described, step A3 comprises:

[0026] Step A31, using the three-primary-color similarity method to calculate the average apparent similarity of each two facial features on the real face dataset, and statistically analyzing the up-down-left-right qualitative spatial relationship of the two facial features on the real face dataset;

[0027] Step A32: if the average similarity of the two facial features is greater than a threshold value S, and there is a left relationship or a right relationship that is always established in the horizontal direction, and the number of occurrences of the upper relationship and the lower relationship of the two facial features in the vertical direction is equal, then it is determined that the two facial features form horizontal symmetry; if the average similarity of the two facial features is greater than a threshold value S, and there is an upper relationship or a lower relationship that is always established in the vertical direction, and the number of occurrences of the left relationship and the right relationship of the two facial features in the horizontal direction is equal, then it is determined that the two facial features form vertical symmetry;

[0028] Step A33, converting the symmetry detection result into a symmetry qualitative spatial relationship attribute. If the facial feature x forms horizontal symmetry with the left eye, the right eye, the nose or the mouth, then the corresponding attributes horiSYM(x, LE), horiSYM(x, RE), horiSYM(x, N), and horiSYM(x, M) respectively hold. If the facial feature x does not form horizontal symmetry with any other facial feature, then the attribute horiSYM(x, None) holds. If the facial feature x forms vertical symmetry with the left eye, the right eye, the nose or the mouth, then the corresponding attributes vertSYM(x, LE), vertSYM(x, RE), vertSYM(x, N), and vertSYM(x, M) respectively hold. If the facial feature x does not form vertical symmetry with any other facial feature, then the attribute vertSYM(x, None) holds.

[0029] In step A34, the relaxed symmetry qualitative spatial relationship attribute is added to the first-order logic definition of the facial features learned in step A22 in a disjunctive manner, thereby forming a qualitative shape model of the face, namely, a QSM model.

[0030] The method, in step A, training the five sense organ detector comprises the following steps:

[0031] Step A41, for each sample in the real face dataset, based on the key point labels provided by the sample, find the minimum horizontal coordinate x1, the minimum vertical coordinate y1, the maximum horizontal coordinate x2 and the maximum vertical coordinate y2 of the key points included in each facial feature, and obtain a labeled facial feature frame with the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2);

[0032] Step A42, for each sample in the real face dataset, the facial feature frame is expanded outward using the ratio r according to the following formula:

[0033] x′1=x1-(x2-x1)r / 2

[0034] x′2=x2+(x2-x1)r / 2

[0035] y′1=y1-(y2-y1)r / 2

[0036] y′2=y2+(y2-y1)r / 2

[0037] Get a new facial feature box with the coordinates of the upper left corner (x′1, y′1) and the coordinates of the lower right corner (x′2, y′2);

[0038] In step A43, the face image is used as the input of the facial features detector, and the enlarged annotated facial features frames are used as the output of the facial features detector. After training to convergence, a trained facial features detector is obtained.

[0039] The method, in step A, training the local key point detector comprises the following steps:

[0040] Step A51, for each sample in the real face data set, crop the original image according to the annotated facial feature frames generated in step A42 to obtain a partial image of each facial feature;

[0041] Step A52, for each facial feature of each sample in the real face dataset, calculate the facial feature key point label: from all the key points (x, y) belonging to the facial feature in the labeled key points of the sample, subtract the coordinates of the upper left corner of the labeled facial feature box (x′1, y′1), and add them to the facial feature key point label of the sample;

[0042] Step A53, for each facial feature, all local images of the facial feature are used as input of the key point detector, and all key point labels of the facial feature are used as output of the key point detector. After training to convergence, a trained local key point detector for detecting the key points of the facial feature is obtained; the corresponding local key point detectors are trained for each of the four facial features of the face, so as to detect the key points of different facial features respectively.

[0043] The method, in step A, training the overall key point fine-tuner comprises the following steps:

[0044] Step A61, for all samples in the real face dataset, remove the key points on the face contour in their key point labels, and then use Procrustes Analysis to perform rigid alignment on the remaining key points of all samples;

[0045] In step A62, all aligned key points are input into the point distribution model PDM for principal component analysis to obtain an overall key point fine-tuner.

[0046] The method, step B comprises the following steps:

[0047] Step B1, using the trained facial feature detector to perform initial facial feature position detection on the input real face or cartoon face image, and output a large number of candidate facial feature frames, each of which has a confidence value;

[0048] Step B2: Use constrained optimization combined with the QSM model to screen candidate facial features:

[0049] The candidate boxes of each facial feature are x1, x2, x3, x4, and the facial feature detector is used to identify the candidate box x k The confidence level is Appearance(x k ), candidate box x i With the candidate box x j The qualitative spatial relationship between them is Relation(x i , x j ), the first-order logic of the five senses learned by the QSM model is defined as FORs, then all candidate boxes are screened according to the following formula to determine a unique five sense box for each five sense:

[0050]

[0051] stRelation(x i , x j )∈FORs

[0052] Appearance(x i )≥α, i=1,...,4

[0053] Appearance(x j )≥α, j=1,...,4; j≠i

[0054] The method, step C comprises the following steps:

[0055] Step C1, cropping the input image according to the detection frames of the facial features to obtain partial images of the facial features;

[0056] Step C2: for each local image of the facial features, input them into the trained local key point detector of the facial features to obtain the predicted key points of each facial feature.

[0057] The method, step D comprises the following steps:

[0058] Step D1, resetting the key points of each facial feature prediction according to the distribution of the facial features on the face;

[0059] In step D2, the reset key points are input into the trained overall key point fine-tuner, and the adjusted key points are output.

[0060] The technical effect of the present invention is that the method is a single-source domain generalization method and only requires a real face training set as the source domain. Compared with other detection methods, the method has stronger generalization ability and can perform key point detection on real faces and comic faces at the same time, and still ensures strong robustness for comic faces of different exaggerated styles.

[0061] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0063] Figure 2 This is a flowchart for implementing step A of an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of initial detection and screening of facial feature frames according to an embodiment of the present invention;

[0065] Figure 4 This is the final key point detection result of the real face in the embodiment of the present invention.

[0066] Figure 5 This is the final key point detection result of the comic face according to the embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following embodiments are for detecting key points of real faces or cartoon faces. The complete process is as follows: Figure 1 As shown. The training samples in the embodiment consist of 3148 samples, each of which contains a face image and 68 key point annotations. The training samples come from the 300W real face dataset released by Imperial College London. The test samples are divided into real face test samples and cartoon face test samples. The real face test samples also come from the 300W dataset, and the cartoon face test samples come from the WebCaricature cartoon face dataset released by Nanjing University.

[0068] This embodiment provides a cross-domain key point detection method compatible with real faces and cartoon faces, and the method includes the following steps:

[0069] Step A, establishing a qualitative face shape model (QSM model);

[0070] Step B, training the facial features detector;

[0071] Step C: training a local key point detector;

[0072] Step D: training the overall key point fine tuner;

[0073] Step E: Use the QSM model and the facial features detector to complete facial features detection;

[0074] Step F, using a local key point detector to detect key points of the facial features respectively;

[0075] Step G: Use the global key point fine-tuner to adjust the key point position.

[0076] The implementation process of step A is as follows: Figure 2 As shown, the specific processing process is as follows:

[0077] Step A1, dividing the facial features into four features, namely the left eye (including the left eyebrow), the right eye (including the right eyebrow), the nose and the mouth; calculating the coordinates of the center points of each feature in the facial image, and determining the up-down-left-right qualitative spatial relationship attributes between the features;

[0078] Step A2: Generate facial features samples according to the up, down, left, and right qualitative spatial relationship attributes, and use the current best hypothesis algorithm to perform inductive learning on all generated samples to obtain the initial first-order logic definition of facial features;

[0079] Step A3, using the three-primary color similarity method and combining the up-down-left-right qualitative spatial relationship, perform relaxed symmetry detection. The reason why it is called relaxed symmetry here is that the conditions for determining whether it is established are not very strict. It is to determine whether symmetry is formed under relatively relaxed conditions, that is, only similarity and qualitative spatial relationship are used for determination. Then the detection result of relaxed symmetry is converted into relaxed symmetry qualitative spatial relationship attribute; the symmetry qualitative spatial relationship attribute is added to the first-order logic definition of the facial features learned in step A2 in a disjunctive manner, thereby forming a qualitative shape model of the face (QSM model).

[0080] The specific processing process of step A1 is as follows:

[0081] Step A11: For each sample in the real face data set, calculate the minimum horizontal coordinate x1, minimum vertical coordinate y1, maximum horizontal coordinate x2, and maximum vertical coordinate y2 of the key points contained in each facial feature according to its key point label; calculate the center point (x0, y0) of each facial feature:

[0082] x0=(x1+x2) / 2

[0083] y0=(y1+y2) / 2

[0084] Step A12, for each sample of the real face data set, according to the coordinates of the center points of the facial features, determine the up-down-left-right qualitative spatial relationship attributes between all facial features and other facial features: if the center point of facial feature x is to the left of the center point of the left eye, then the attribute horiLE(x, left) holds, if it is to the right, then the attribute horiLE(x, right) holds, if they are at the same horizontal position, then the attribute horiLE(x, equal) holds; if it is above, then the attribute vertLE(x, up) holds, if it is below, then the attribute vertLE(x, down) holds, if they are at the same vertical position, then the attribute vertLE(x, equal) holds; the up-down-left-right qualitative spatial relationship attributes horiRE, vertRE, horiN, vertN, horiM, vertM between facial feature x and the right eye, nose, and mouth can be obtained in the same way. The value ranges of each up-down-left-right qualitative spatial relationship attribute are shown in Table 1 below.

[0085] Table 1 Value ranges of up, down, left, and right qualitative spatial relationship attributes

[0086] Qualitative spatial relationship attributes of up, down, left, and right Value range horiLE (left, equal, right) horiRE (left, equal, right) horiN (left, equal, right) horiM (left, equal, right) vertLE (up, equal, down) vertRE (up, equal, down) vertN (up, equal, down) vertM (up, equal, down)

[0087] Furthermore, the specific processing process of step A2 is as follows:

[0088] Step A21, for each sample of the real face data set, generate positive and negative examples for inductive learning for each facial feature x, and the set of facial features remaining after removing feature x is called set. Conjunction is performed between feature x and the up, down, left, and right qualitative spatial relationship attributes of each facial feature in the set to obtain positive examples of feature x; for each facial feature in the set, conjunction is performed between them and the up, down, left, and right qualitative spatial relationship attributes of each facial feature in the set to obtain negative examples of feature x. The positive and negative examples generated for the left eye of one of the face samples are shown in Table 2 below:

[0089] Table 2 Positive and negative examples generated by face samples for the left eye

[0090]

[0091] Step A22: For each facial feature, use the current best hypothesis algorithm to perform inductive learning on all positive and negative examples of the facial feature. If a false negative example is encountered during learning, a generalization operation is performed; if a false positive example is encountered, a narrowing operation is performed. After the inductive learning is completed, the initial first-order logic definition of each facial feature is obtained. In this embodiment, the learning results of the target predicates LeftEye(x), RightEye(x), Nose(x), and Mouth(x) for the left eye, right eye, nose, and mouth are shown in Table 3 below.

[0092] Table 3 Learning results of the current best hypothesis algorithm

[0093] LeftEye(x)←horiRE(x, left)∧vertN(x, up) RightEye(x)←horiLE(x,right)∧vertN(x,up) Nose(x)←vertLE(x, down)∧vertRE(x, down)∧vertM(x, up) Mouth(x)←vertN(x,down)

[0094] Furthermore, the specific processing process of step A3 is as follows:

[0095] Step A31: Use the three-primary-color similarity method to calculate the average apparent similarity of each two facial features on the real face dataset, and count the up-down, left-right qualitative spatial relationship of the two facial features on the real dataset.

[0096] Step A32: If the average similarity of the two facial features is greater than the threshold value S, and there is a constant left relationship or right relationship in the horizontal direction, and the number of occurrences of the upper relationship and the lower relationship of the two facial features in the vertical direction is equal, then it is determined that the two facial features form horizontal symmetry. The threshold value S is specified by experience and is set to 0.6 in this embodiment. If the average similarity of the two facial features is greater than the threshold value S, and there is a constant upper relationship or lower relationship in the vertical direction, and the number of occurrences of the left relationship and the right relationship of the two facial features in the horizontal direction is equal, then it is determined that the two facial features form vertical symmetry; in this embodiment, the left eye and the right eye form horizontal symmetry.

[0097] Step A33, convert the detection result of relaxed symmetry into relaxed symmetry qualitative spatial relationship attributes. If the facial feature x forms horizontal symmetry with the left eye, right eye, nose or mouth, then the attributes horiSYM(x, LE), horiSYM(x, RE), horiSYM(x, N), and horiSYM(x, M) respectively hold. If the facial feature x does not form horizontal symmetry with any other facial feature, then the attribute horiSYM(x, None) holds. If the facial feature x forms vertical symmetry with the left eye, right eye, nose or mouth, then the attributes vertSYM(x, LE), vertSYM(x, RE), vertSYM(x, N), and vertSYM(x, M) respectively hold. If the facial feature x does not form vertical symmetry with any other facial feature, then the attribute vertSYM(x, None) holds. The value ranges of each relaxed symmetry qualitative spatial relationship attribute are shown in Table 4 below.

[0098] Table 4 Value ranges of relaxed symmetry qualitative spatial relationship attributes

[0099] Relaxed symmetry qualitative spatial relationship properties Value range horiSYM (LE, RE, N, M, None) vertSYM (LE, RE, N, M, None)

[0100] Step A34, add the relaxed symmetry qualitative spatial relationship attribute to the first-order logic definition of the five sense organs learned in step A22 in a disjunctive manner, thereby forming a qualitative shape model of the face (QSM model); in this embodiment, horiSYM(x, RE) and horiSYM(x, LE) are added to the first-order logic definitions of the left eye and the right eye respectively. The final first-order logic definition of the five sense organs of the QSM model is shown in Table 5 below.

[0101] Table 5 Final five-sense first-order logic definition of the QSM model

[0102] LeftEye(x)←[horiRE(x, left)∧vertN(x, up)]VhoriSYM(x, RE) RightEye(x)←[horiLE(x,right)∧vertN(x,up)]VhoriSYM(x,LE) Nose(x)←vertLE(x, down)∧vertRE(x, down)∧vertM(x, up) Mouth(x)←vertN(x,down)

[0103] Furthermore, the specific processing process of step B is as follows:

[0104] Step B1, for each sample in the real face dataset, calculate the minimum horizontal coordinate x1, minimum vertical coordinate y1, maximum horizontal coordinate x2, and maximum vertical coordinate y2 of the key points contained in each facial feature according to its key point labels, and obtain a facial feature frame with the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2);

[0105] Step B2: For each sample in the real face dataset, the facial feature frame is expanded outward using the ratio r according to the following formula:

[0106] x′1=x1-(x2-x1)r / 2

[0107] x′2=x2+(x2-x1)r / 2

[0108] y′1=y1-(y2-y1)r / 2

[0109] y′2=y2+(y2-y1)r / 2

[0110] Get the newly annotated facial feature frame with the coordinates of the upper left corner (x′1, y′1) and the lower right corner (x′2, y′2);

[0111] Step B3, using the face image as the input of the facial features detector, and the expanded annotated facial features frame as the output of the facial features detector, and training until convergence to obtain a facial features detector. In this embodiment, the facial features detector, the local key point detector, and the global key point fine-tuner are all implemented using existing technologies, and only need to be trained according to the requirements of this embodiment. The facial features detector is implemented using the FasterR-CNN model.

[0112] The specific processing process of step C is as follows:

[0113] Step C1: for each sample in the real face dataset, crop the original image according to the facial feature frames generated in step B2 to obtain a partial image of each facial feature;

[0114] Step C2, for each facial feature of each sample in the real face dataset, calculate the facial feature key point label; the specific calculation method is: from all the key points (x, y) belonging to the facial feature in the labeled key points of the sample, subtract the coordinates of the upper left corner of the labeled facial feature box (x′1, y′1), and add them to the facial feature key point label of the sample;

[0115] Step C3: for each facial feature, all local images of the facial feature are used as inputs of the key point detector, all key point labels of the facial feature are used as outputs of the key point detector, and a local key point detector for detecting the key points of the facial feature is trained. The present invention divides the face into four facial features, and a total of four local key point detectors are trained, which are respectively used to detect the key points of different facial features. In this embodiment, the ERT model is used as the local key point detector.

[0116] Furthermore, the specific processing process of step D is as follows:

[0117] Step D1: For all samples in the real face dataset, remove the key points on the face contour in their key point labels, and then use Procrustes Analysis to perform rigid alignment on the remaining key points of all samples;

[0118] Step D2: Input all aligned key points into the point distribution model PDM for principal component analysis to obtain an overall key point fine-tuner.

[0119] Furthermore, the specific processing process of step E is as follows:

[0120] Step E1: Use the facial features detector trained in step B3 to perform initial facial features detection on the input real face or cartoon face image. The initial detection result is as follows: Figure 3 As shown in (a), a large number of candidate facial feature boxes are output, each of which has a confidence value;

[0121] Step E2: Use constrained optimization combined with the QSM model to screen candidate facial feature frames. The specific screening method is:

[0122] Suppose the candidate boxes of each facial feature are x1, x2, x3, x4, and the facial feature detector is used to identify the candidate box x k The confidence level is Appearance(x k ), candidate box x i With the candidate box x j The qualitative spatial relationship between them is Relation(xi , x j ), the first-order logic of the five senses learned by the QSM model is defined as FORs, then all candidate boxes are screened according to the following formula to determine a unique five sense box for each five sense. The screening results are as follows Figure 3 (b) as shown.

[0123]

[0124] stRelation(x i , x j )∈FORs

[0125] Appearance(x i )≥α, i=1,...,4

[0126] Appearance(x j )≥α, j=1,...,4; j≠i

[0127] Furthermore, the specific processing process of step F is as follows:

[0128] Step F1, cropping the input image according to the detection frames of the facial features determined in step E2 to obtain partial images of the facial features;

[0129] Step F2: For each local image of the facial features, input them into the local key point detector of the facial features trained in step C3 to obtain the predicted key points of each facial feature.

[0130] Finally, the specific processing process of step G is as follows:

[0131] Step G1: Since the key points predicted for the facial features in step F are respectively performed for each facial feature by an independent local key point detector, it is necessary to reset these predicted key points according to the distribution of the facial features before inputting them into the PDM model, that is, reset the key points predicted for each facial feature according to the distribution of the facial features on the face;

[0132] Step G2: Input the reset key points into the PDM model trained in step D2, and output the adjusted key points. The key point positions output in this step are the final key point positions predicted by this method, where the final key point detection result of the real face is as follows: Figure 4 As shown in the figure, the final key point detection result of the comic face is as follows Figure 5 shown.

Claims

1. A cross-domain key point detection method compatible with real faces and cartoon faces, characterized in that: The following steps are involved: Step A, building a qualitative face shape model, namely the QSM model, and training the facial features detector, local key point detector and global key point fine-tuner; Step B, using a facial feature detector to perform an initial detection of facial feature positions on the face image to obtain candidate facial feature frames, and then using a QSM model to screen and determine the correct facial feature frames; Step C, using a local key point detector to detect the facial features respectively to obtain key points; Step D, use the global key point fine-tuner to adjust the key point position; In the step A, establishing a qualitative face shape model comprises the following steps: Step A1, dividing the facial features into four independent features, namely the left eye including the left eyebrow, the right eye including the right eyebrow, the nose and the mouth; calculating the coordinates of the center points of each facial feature in the facial image, and determining the up-down-left-right qualitative spatial relationship attributes between the facial features; Step A2: Generate facial features samples according to the up, down, left, and right qualitative spatial relationship attributes, and use the current best hypothesis algorithm to perform inductive learning on all generated samples to obtain the initial first-order logic definition of facial features; Step A3, using the three primary colors similarity method and combining the up-down-left-right qualitative spatial relationship to perform symmetry detection; converting the symmetry detection result into a symmetry qualitative spatial relationship attribute; adding the symmetry qualitative spatial relationship attribute to the first-order logic definition of the facial features learned in step A2 in a disjunctive manner, thereby forming a qualitative face shape model, namely, a QSM model; The step A2 comprises: Step A21, for each sample of the real face data set, generate positive examples and negative examples for inductive learning for each facial feature x: the set of facial features other than facial feature x in the face is called set, and the facial feature x is conjuncted with the up-down-left-right qualitative spatial relationship attributes of each facial feature in the set to obtain positive examples of facial feature x; for each facial feature in the set, the up-down-left-right qualitative spatial relationship attributes of each facial feature in the set are also conjuncted to obtain negative examples of facial feature x; Step A22, for each facial feature, use the Current-best-hypothesis algorithm to perform inductive learning on all positive and negative examples of the facial feature, and the initial hypothesis is randomly set to one of the positive examples of the facial feature. If a false negative example is encountered during learning, that is, the current hypothesis is judged as a negative example, but it is actually a positive example, then the current hypothesis is generalized; if a false positive example is encountered, that is, the current hypothesis is judged as a positive example, but it is actually a negative example, then the current hypothesis is narrowed; if a true positive example and a true negative example are encountered, the current hypothesis is maintained unchanged; after the inductive learning is completed, the initial first-order logic definition of each facial feature is obtained; The step A3 comprises: Step A31, using the three-primary-color similarity method to calculate the average apparent similarity of each two facial features on the real face dataset, and statistically analyzing the up-down-left-right qualitative spatial relationship of the two facial features on the real face dataset; Step A32: if the average similarity of the two facial features is greater than a threshold value S, and there is a left relationship or a right relationship that is always established in the horizontal direction, and the number of occurrences of the upper relationship and the lower relationship of the two facial features in the vertical direction is equal, then it is determined that the two facial features form horizontal symmetry; if the average similarity of the two facial features is greater than a threshold value S, and there is an upper relationship or a lower relationship that is always established in the vertical direction, and the number of occurrences of the left relationship and the right relationship of the two facial features in the horizontal direction is equal, then it is determined that the two facial features form vertical symmetry; Step A33, converting the symmetry detection result into a symmetry qualitative spatial relationship attribute. If the facial feature x forms horizontal symmetry with the left eye, the right eye, the nose or the mouth, then the corresponding attributes horiSYM(x,LE), horiSYM(x,RE), horiSYM(x,N), and horiSYM(x,M) respectively hold. If the facial feature x does not form horizontal symmetry with any other facial feature, then the attribute horiSYM(x,None) holds. If the facial feature x forms vertical symmetry with the left eye, the right eye, the nose or the mouth, then the corresponding attributes vertSYM(x,LE), vertSYM(x,RE), vertSYM(x,N), and vertSYM(x,M) respectively hold. If the facial feature x does not form vertical symmetry with any other facial feature, then the attribute vertSYM(x,None) holds. In step A34, the relaxed symmetry qualitative spatial relationship attribute is added to the first-order logic definition of the facial features learned in step A22 in a disjunctive manner, thereby forming a qualitative shape model of the face, namely, a QSM model.

2. The method according to claim 1, characterized in that: The step A1 comprises: Step A11, for each sample in the real face dataset, based on the key point labels provided by the sample, find the minimum horizontal coordinate x1, the minimum vertical coordinate y1, the maximum horizontal coordinate x2 and the maximum vertical coordinate y2 of the key points contained in each facial feature; calculate the center point (x0, y0) of each facial feature: x0=(x1+x2) / 2 y0=(y1+y2) / 2 Step A12, according to the coordinates of the center points of the facial features, determine the up-down-left-right qualitative spatial relationship attributes of all facial features with other facial features, including four attributes for describing the horizontal qualitative spatial relationship, namely horiLE, horiRE, horiN and horiM, where horiLE represents the horizontal relationship between the current facial feature and the left eye, horiRE represents the horizontal relationship between the current facial feature and the right eye, horiN represents the horizontal relationship between the current facial feature and the nose, and horiM represents the horizontal relationship between the current facial feature and the mouth. The attribute value is left, which means the current facial feature is on the left side of the corresponding facial feature; equal, which means the current facial feature is at the same horizontal position as the corresponding facial feature; or right, which means the current facial feature is on the right side of the corresponding facial feature. It also includes four attributes, vertLE, vertRE, vertN and vertM, which are used to describe vertical qualitative spatial relationships. vertLE represents the vertical relationship between the current facial feature and the left eye, vertRE represents the vertical relationship between the current facial feature and the right eye, vertN represents the vertical relationship between the current facial feature and the nose, and vertM represents the vertical relationship between the current facial feature and the mouth. The attribute values ​​are up, indicating that the current facial feature is above the corresponding facial feature, equal, indicating that the current facial feature is at the same vertical position as the corresponding facial feature, or down, indicating that the current facial feature is below the corresponding facial feature.

3. The method according to claim 1, characterized in that In step A, training the five sense organ detector comprises the following steps: Step A41, for each sample in the real face dataset, based on the key point labels provided by the sample, find the minimum horizontal coordinate x1, the minimum vertical coordinate y1, the maximum horizontal coordinate x2 and the maximum vertical coordinate y2 of the key points included in each facial feature, and obtain a labeled facial feature frame with the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2); Step A42, for each sample in the real face dataset, the facial feature frame is expanded outward using the ratio r according to the following formula: x ′ 1=x1-(x2-x1)r / 2 x ′ 2=x2+(x2-x1)r / 2 y1 ′ =y1-(y2-y1)r / 2 y2 ′ =y2+(y2-y1)r / 2 The coordinates of the upper left corner are (x ′ 1,y1 ′ ), the coordinate of the lower right corner is (x ′ 2,y2 ′ )'s new facial features frame; In step A43, the face image is used as the input of the facial features detector, and the enlarged annotated facial features frames are used as the output of the facial features detector. After training to convergence, a trained facial features detector is obtained.

4. The method according to claim 3, characterized in that In step A, training the local key point detector comprises the following steps: Step A51, for each sample in the real face data set, crop the original image according to the annotated facial feature frames generated in step A42 to obtain a partial image of each facial feature; Step A52, for each facial feature of each sample in the real face dataset, calculate the facial feature key point label: for each key point (x, y) belonging to the facial feature in the sample's labeled key points, subtract the coordinates of the upper left corner of the labeled facial feature box (x ′ 1,y1 ′ ), and then added to the facial feature key point label of the sample; Step A53, for each facial feature, all local images of the facial feature are used as input of the key point detector, and all key point labels of the facial feature are used as output of the key point detector. After training to convergence, a trained local key point detector for detecting the key points of the facial feature is obtained; the corresponding local key point detectors are trained for each of the four facial features of the face, so as to detect the key points of different facial features respectively.

5. The method according to claim 1, characterized in that In step A, training the overall key point fine-tuner includes the following steps: Step A61, for all samples in the real face dataset, remove the key points on the face contour in their key point labels, and then use Procrustes Analysis to perform rigid alignment on the remaining key points of all samples; In step A62, all aligned key points are input into the point distribution model PDM for principal component analysis to obtain an overall key point fine-tuner.

6. The method according to claim 1, characterized in that Described step B comprises the following steps: Step B1, using the trained facial feature detector to perform initial facial feature position detection on the input real face or cartoon face image, and output a large number of candidate facial feature frames, each of which has a confidence value; Step B2: Use constrained optimization combined with the QSM model to screen candidate facial features: The candidate boxes of each facial feature are x1, x2, x3, x4, and the facial feature detector is used to identify the candidate box x k The confidence level is Appearance(x k ), candidate box x i With the candidate box x j The qualitative spatial relationship between them is Relation(x i ,x j ), the first-order logic of the facial features learned by the QSM model is defined as FORs, then all candidate boxes are screened according to the following formula to determine a unique facial feature box for each facial feature: s.t.Relation(x i ,x j )∈FORs Appearance(x i )≥α,i=1,…,4 Appearance(x j )≥α,j=1,…,4;j≠i。 7. The method according to claim 1, characterized in that The step C comprises the following steps: Step C1, cropping the input image according to the detection frames of the facial features to obtain partial images of the facial features; Step C2: for each local image of the facial features, input them into the trained local key point detector of the facial features to obtain the predicted key points of each facial feature.

8. The method according to claim 1, characterized in that The step D comprises the following steps: Step D1, resetting the key points of each facial feature prediction according to the distribution of the facial features on the face; Step D2, input the reset key points into the trained overall key point fine-tuner, and output the adjusted key points.