Face recognition method and device, equipment and storage medium
Through the multi-level integration of makeup recognition, bone structure characteristics and human ear characteristics, the problem of insufficient facial recognition accuracy under the influence of makeup in the prior art is solved, and a high accuracy recognition effect under different makeup is achieved.
Patent Information
- Application Number
- CN202411128442.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
Existing facial recognition technology has insufficient accuracy in dealing with nonlinear changes and subtle feature differences caused by makeup, especially in heavy or extreme makeup cases.
Using a multi-level and multi-module method of makeup recognition, bone structure feature extraction and human ear feature integration, facial area segmentation is performed through the UNet model, and key points are located using the pre-trained MTCNN model, combining the facial bone contour recognition model and ear contour features. Compute similarity for accurate identification.
It significantly improves the accuracy and robustness of facial recognition, can maintain high recognition accuracy under different makeup styles, effectively avoid the impact of makeup and lighting changes on recognition results, and enhances adaptability and recognition performance in complex environments. .
Smart Images

Figure CN119091488A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of face recognition technology, and in particular to a face recognition method, device, equipment and storage medium. Background Art
[0002] Existing face recognition technologies have shown a high level of maturity in many applications and are widely used in security, payment, and smart device unlocking. However, these technologies still face significant challenges in the face of changes before and after makeup. Makeup can significantly change the color, texture, and feature position of the face, resulting in a decrease in the accuracy of the recognition system. Despite advances in deep learning and multimodal data fusion technology, existing technologies still have much room for improvement in dealing with nonlinear changes and subtle feature differences caused by makeup.
[0003] In existing research, a Chinese patent with a publication number of CN109165546B proposes a face recognition method and device, which includes obtaining a first face image, determining the makeup degree parameter of the image, and determining the corresponding restoration parameter based on the parameter, and processing the image to generate a second face image. Then, the second face image is matched with a preset face template. If the match is successful, it is confirmed that the first face image is successfully recognized. The makeup degree parameter is determined by multi-scale decomposition and high-frequency component feature extraction of the first face image, and the makeup degree is determined according to the distribution density of the feature points. The corresponding restoration parameter is obtained by the mapping relationship between the makeup level and the restoration algorithm. The method relies on restoring the first face image after makeup through the makeup degree parameter and the restoration parameter.
[0004] Another Chinese patent with the publication number CN109858392B proposes a method for automatic recognition of facial images before and after makeup. First, the facial data is initially extracted for feature extraction; an algorithm based on facial key area replacement is used to expand the makeup face dataset and create a training dataset containing multiple examples. During the training process, an auxiliary learning network that uses key local area features is used to extract global and local features, and then these features are fused through the attention mechanism to obtain the final features. The network is continuously trained and fine-tuned until the network converges. In the testing phase, the facial data before and after makeup to be identified is input into the trained face recognition network, features are extracted and similarity is calculated, so as to achieve recognition.
[0005] Another Chinese patent with the publication number CN112800819B proposes a face recognition method, device and electronic device. First, the acquired face image to be processed is preprocessed to obtain a target face image; then, the makeup key points are determined according to the face features of the target face image, and a standard face image corresponding to the target face image is selected from a preset face database. Then, the standard face image is rendered based on the makeup key points to obtain a makeup face image; the similarity between the target face image and the makeup face image is calculated, and whether the target face image and the makeup face image are the same person is determined based on the similarity.
[0006] However, the above methods mainly rely on facial features and are suitable for regular makeup. When encountering heavy makeup or extreme makeup, the accuracy of face recognition is often not ideal. At present, there is an urgent need for a face recognition method that can maintain high recognition accuracy for different makeup styles.
[0007] To this end, a face recognition method, device, equipment and storage medium are proposed. Summary of the invention
[0008] The purpose of the present invention is to provide a face recognition method, device, equipment and storage medium. First, collect the frontal face image of the target object with exposed ears, obtain the image to be tested, and obtain the first face image set and the first ear image set pre-stored by the specified group. Makeup recognition is performed on the image to be tested. If it is identified as plain face, the first maximum similarity between the image to be tested and the first face image set is calculated. If it is greater than a threshold, the corresponding personnel information is output; if it is identified as makeup, the bone structure features are extracted and the second maximum similarity is calculated. If it is greater than a threshold, the personnel information is output. If none of the conditions are met, the second ear image of the target object is collected, the contour and position ratio features are extracted, and the integrated ear features to be tested are obtained by integration, and matched with the first ear image set, and the third maximum similarity is calculated. If it is greater than a threshold, the personnel information is output, otherwise it is identified as a strange object and an early warning is issued. The method realizes accurate face recognition and identity verification through makeup recognition, bone structure feature extraction and ear feature integration. To achieve the above purpose, the present invention provides the following technical solutions:
[0009] A face recognition method, comprising:
[0010] Collect a frontal facial image of the target object with ears exposed to obtain an image to be inspected;
[0011] Acquire a first face image set and a first ear image set pre-stored in a specified group;
[0012] Performing makeup recognition on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup;
[0013] If the makeup recognition result is no makeup, obtaining a first maximum similarity based on the image to be inspected and the first face image set; if the first maximum similarity is greater than a first threshold, outputting the person information corresponding to the first maximum similarity;
[0014] If the makeup recognition result is makeup, the bone structure features of the image to be tested and the first face image set are extracted respectively, and the second maximum similarity is calculated. If the second maximum similarity is greater than a second threshold, the personnel information corresponding to the second maximum similarity is output; otherwise, the second ear image of the target object is collected, and the features are extracted to obtain the contour features of the ear to be tested; the ear features of the image to be tested are extracted to obtain the position and proportion features of the ear to be tested; the contour features of the ear to be tested and the position and proportion features of the ear to be tested are integrated to obtain the integrated ear features to be tested; based on the first ear image set and the first face image set, an integrated ear feature set is obtained; based on the integrated ear features to be tested and the integrated ear feature set, a third maximum similarity is obtained; if the third maximum similarity is greater than a third threshold, the personnel information corresponding to the third maximum similarity is output; otherwise, the target object is identified as a stranger and an early warning is issued.
[0015] Furthermore, the first face image set and the first ear image set are specifically:
[0016] The first human ear image set consists of a number of first human ear images, wherein the first human ear images include a complete left ear image and a complete right ear image of the specified group, and the first human ear image contains the outline of the entire ear; the first human ear image is collected from the side of the members of the group by a high-resolution camera; the first human face image set includes the frontal facial images of the specified group without makeup and with ears exposed; the name of each member of the group has a one-to-one correspondence with the corresponding first human face image and first human ear image, and is stored in a database.
[0017] Furthermore, makeup recognition is performed on the image to be inspected, and the makeup recognition result obtained includes:
[0018] The pre-processed image to be inspected is segmented into several key ROI regions by using the UNet model, including pupil, upper eyelid, lower eyelid, lip, eyebrow, nose, forehead and cheek, and a segmented image of each key ROI region is obtained; the key points of the face are located by using the pre-trained MTCNN model, and the key ROI region is corrected to obtain a corrected ROI region and a corrected segmented image corresponding to each corrected ROI region; the first face image set is subjected to the same pre-processing, segmentation and key point positioning to obtain a second corrected ROI region, and the plain face reference color range of each second corrected ROI region is calculated to establish a plain face reference color feature database;
[0019] For each segmented image of the key ROI area, the color features of all pixels of the segmented image are extracted to obtain multiple ROI area color features; the ROI area color features of each image to be inspected are compared with the bare face reference color feature database; if all the ROI area color features of the image to be inspected are within the corresponding bare face reference color range, the makeup recognition result is bare face, otherwise the makeup recognition result is makeup.
[0020] Furthermore, obtaining the first maximum similarity according to the image to be detected and the first face image set includes the following steps:
[0021] Input each image in the preprocessed first face image set into a pretrained convolutional neural network model to extract facial key point features to obtain M N-dimensional first key feature vectors;
[0022] Input the preprocessed image to be inspected into the convolutional neural network model to extract facial key point features to obtain an N-dimensional second key feature vector;
[0023] The similarities between the second key feature vector and the M first key feature vectors are respectively calculated to obtain M first similarities; and the maximum value among the M first similarities is selected as the first maximum similarity.
[0024] Further, respectively extracting the bone structure features from the image to be inspected and the first face image set, and calculating the second maximum similarity includes:
[0025] Inputting the preprocessed first face image set into a trained facial skeleton contour recognition model to obtain M K-dimensional first contour feature vectors;
[0026] Inputting the preprocessed image to be inspected into the trained facial bone contour recognition model to obtain a K-dimensional second contour feature vector;
[0027] respectively calculating similarities between the second contour feature vector and the M first contour feature vectors to obtain M second similarities; selecting a maximum value among the M second similarities as the second maximum similarity;
[0028] The facial bone contour recognition model includes:
[0029] Input layer, used to receive input images;
[0030] A facial contour segmentation layer, used to segment the facial area of the input image using a UNet model to obtain a facial segmentation area;
[0031] A feature extraction layer, using a multi-scale convolutional neural network structure, is used to extract features from the segmented facial segmentation area;
[0032] The attention mechanism layer includes a channel attention module and a spatial attention module, which are used to guide the facial bone contour recognition model to focus on key contour areas, including cheekbones, mandibular angles, brow bones, and nose bridge;
[0033] The feature fusion layer is used to fuse features from different levels and scales using an adaptive weighting mechanism;
[0034] The dynamic adjustment layer includes an environment perception module and a dynamic adjustment module; the environment perception module is used to detect the lighting conditions and makeup conditions of the input image in real time; the dynamic adjustment module is used to dynamically adjust the parameters of the feature extraction layer and the feature fusion layer according to the output result of the environment perception module;
[0035] The output layer is used to output the bone structure features.
[0036] Further, collecting the second human ear image of the target object, extracting features, and obtaining the contour features of the human ear to be tested; extracting the human ear features of the image to be tested, and obtaining the position ratio features of the human ear to be tested includes:
[0037] Performing feature extraction on the second human ear image to obtain the human ear contour feature set, including auricle contour features, cochlear structure features, tragus features, antitragus features, earlobe features, and texture features;
[0038] Use the pre-trained MTCNN model to detect the image to be tested and locate the key points of the face; further refine the detection through the key point detection model to identify the precise key points of the ear, including the top of the auricle, the earlobe, the tragus and the antitragus; determine the pre-recognition area of the ear based on the detected precise key points; use image segmentation technology to finely segment the pre-recognition area of the ear to obtain ear contour features; calculate the relative distance and ratio between the precise key points of the ear to obtain the ear position ratio feature; integrate the ear contour feature and the ear position ratio feature to obtain the ear position ratio feature of the person to be tested.
[0039] Further, according to the integrated human ear feature to be tested and the integrated human ear feature set, the third maximum similarity is obtained as follows:
[0040] The dimension of the integrated human ear feature set is H; the integrated human ear feature set includes M H-dimensional integrated human ear feature vectors; the similarities between the integrated human ear feature set and the M integrated human ear feature vectors are calculated respectively to obtain M third similarities; the maximum value of the M third similarities is selected as the third maximum similarity.
[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a face recognition device, comprising:
[0042] An image acquisition module is used to acquire a frontal facial image of a target object with its ears exposed to obtain an image to be inspected;
[0043] An image acquisition module, used to acquire a first face image set and a first ear image set pre-stored by a designated group;
[0044] A makeup recognition module is used to perform makeup recognition on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup;
[0045] A first maximum similarity calculation module, configured to obtain a first maximum similarity according to the image to be inspected and the first face image set when the makeup recognition result is a natural face;
[0046] A second maximum similarity calculation module is used to extract bone structure features from the image to be tested and the first face image set respectively, and calculate a second maximum similarity when the makeup recognition result is makeup;
[0047] A third maximum similarity calculation module is used for, when the second maximum similarity is less than or equal to a second threshold, collecting a second human ear image of the target object, extracting features, and obtaining a human ear contour feature to be tested; extracting human ear features of the image to be tested, and obtaining a human ear position ratio feature to be tested; integrating the human ear contour feature to be tested and the human ear position ratio feature to be tested, and obtaining an integrated human ear feature to be tested; obtaining an integrated human ear feature set based on the first human ear image set and the first face image set; and obtaining a third maximum similarity based on the integrated human ear feature to be tested and the integrated human ear feature set;
[0048] A similarity judgment module, used to judge whether the first maximum similarity is greater than a first threshold; judge whether the second maximum similarity is greater than a second threshold; judge whether the third maximum similarity is greater than a third threshold;
[0049] The recognition result output module is used to output the personnel information corresponding to the first maximum similarity when the first maximum similarity is greater than the first threshold; output the personnel information corresponding to the second maximum similarity when the second maximum similarity is greater than the second threshold; output the personnel information corresponding to the third maximum similarity when the third maximum similarity is greater than the third threshold; and identify the target object as an unfamiliar object and issue a warning when the third maximum similarity is less than or equal to the third threshold.
[0050] In addition, to achieve the above-mentioned purpose, the present invention also proposes a face recognition device, which includes at least a processor and a memory, and the processor is used to implement the steps of the method described above when executing a computer program stored in the memory.
[0051] In addition, to achieve the above-mentioned purpose, the present invention also proposes a face recognition storage medium, on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the method described above.
[0052] Compared with the prior art, the present invention has the following beneficial effects:
[0053] 1. The present invention uses the UNet model to segment the facial regions of the image to be inspected, and uses the pre-trained MTCNN model to locate and correct key points, extract the color features of these facial regions, and compare them with the base color feature database of bare face. Since each segmented ROI area has unique color features, by extracting the color features of these areas separately and comparing them with the base color, it is possible to more effectively distinguish between makeup and bare face states, laying the foundation for subsequent recognition and helping to improve the precision and accuracy of face recognition.
[0054] 2. The present invention proposes a facial bone contour recognition model that can capture the structural features of the face, especially in the case of makeup, by focusing on the key areas such as the cheekbones, mandibular angles, brow bones and nose bridge, effectively avoiding the influence of makeup on the recognition results; the introduction of the attention mechanism layer increases the focus on the features of the key areas and enhances the robustness of the feature representation; the environment perception module detects the lighting conditions and makeup conditions of the input image in real time, and the dynamic adjustment module dynamically adjusts the parameters of the feature extraction layer and the feature fusion layer according to this information to ensure consistent recognition effects in different environments; the adaptive weighted feature fusion layer can integrate features of different levels and scales, and improve the accuracy and reliability of the overall feature representation. This method significantly improves the accuracy of face recognition through the collaborative work of multiple levels and modules, without being constrained by makeup styles.
[0055] 3. This face recognition method effectively improves the accuracy and robustness of recognition by introducing ear contour features and position ratio features. When the recognition of facial key points and contour features cannot achieve the expected results, the ear features, as independent auxiliary features, provide a reliable basis for recognition. By utilizing the structural and position ratio features of the ear, the interference of factors such as makeup and lighting changes on the recognition results is avoided, the recognition ability under complex conditions is enhanced, and the adaptability and overall recognition performance in a changing environment are significantly improved. Especially in the case of makeup, the detection and calculation of the precise key points and position ratio features of the ear effectively improves the accuracy and reliability of recognition, and the high recognition accuracy can be maintained when facing different makeup styles. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 A schematic diagram of a specific flow chart of an attendance system provided by an embodiment of the present invention;
[0057] Figure 2 A flow chart of a face recognition method provided by an embodiment of the present invention;
[0058] Figure 3 A structural diagram of a facial bone contour recognition model provided by an embodiment of the present invention;
[0059] Figure 4 A structural diagram of a face recognition device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0061] Embodiment 1
[0062] The face attendance system of Company A applies a face recognition method, such as Figure 1 As shown, it is used to recognize the face of employees, including: Figure 2 In S10, a frontal facial image of the target object with ears exposed is collected to obtain an image to be inspected;
[0063] See also Figure 2 In S20, obtaining a first face image set and a first ear image set pre-stored for a designated group;
[0064] Furthermore, the first face image set and the first ear image set are specifically:
[0065] The first human ear image set consists of a number of first human ear images, wherein the first human ear images include a complete left ear image and a complete right ear image of the specified group, and the first human ear image contains the outline of the entire ear; the first human ear image is collected from the side of the members of the group by a high-resolution camera; the first human face image set includes the frontal facial images of the specified group without makeup and with ears exposed; the name of each member of the group has a one-to-one correspondence with the corresponding first human face image and first human ear image, and is stored in a database.
[0066] By collecting complete left and right ear images and frontal facial images of the specified group without makeup and ears, and storing each member's name and corresponding image in the database, a solid data foundation is laid for subsequent face comparison. Frontal facial images without makeup and ears ensure high accuracy recognition in a natural state. High-resolution ear images provide detailed and stable ear features, which can maintain high recognition accuracy even under complex conditions such as makeup and lighting changes.
[0067] In this embodiment, the first face image set includes frontal plain-face images of all authorized users of Company A; authorized users include all employees of Company A. The first face image set is stored in a database of the attendance system.
[0068] See also Figure 2 In S30, makeup recognition is performed on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup;
[0069] Furthermore, makeup recognition is performed on the image to be inspected, and the makeup recognition result obtained includes:
[0070] The UNet model is used to perform facial region segmentation on the preprocessed image to be inspected, and the image is segmented into several key ROI areas, namely, key parts of the face, including pupil, upper eyelid, lower eyelid, lip, eyebrow, nose, forehead and cheek, to obtain a segmented image of each key ROI area; the pre-trained MTCNN model is used to locate the key points of the face, and the key ROI area is corrected to obtain the corrected ROI area and the corrected segmented image corresponding to each corrected ROI area; the first face image set is subjected to the same preprocessing, segmentation and key point positioning to obtain the second corrected ROI area, and the bare face benchmark color range of each second corrected ROI area is calculated, and a bare face benchmark color feature database is established, as shown in Table 1.
[0071] For each segmented image of the key ROI area, the color features of all pixels of the segmented image are extracted to obtain multiple ROI area color features; the ROI area color features of each image to be inspected are compared with the bare face reference color feature database; if all the ROI area color features of the image to be inspected are within the corresponding bare face reference color range, the makeup recognition result is bare face, otherwise the makeup recognition result is makeup.
[0072] Table 1. Basic color range for bare face
[0073] Location RGB color range pupil (0,0,0) to (50,50,50) Lips (150,50,50) to (230,150,160) Eyebrow (30,20,10) to (80,50,40) nose (180,140,120) to (240,190,160) Forehead (200,150,130) to (250,200,170) Cheek (180,120,100) to (230,180,150) Upper eyelid (160,110,100) to (210,160,150) Lower eyelid (140,90,80) to (190,140,130)
[0074] As a specific implementation, color feature analysis is performed on the corrected lips, and the RGB range is detected to be (240, 130, 140). Comparing with the base color range of (150, 50, 50) to (230, 150, 160), the result shows that there are traces of makeup in this area.
[0075] This makeup recognition method uses the UNet model and the pre-trained MTCNN model to accurately segment the facial region and locate key points of the image to be inspected, thereby ensuring the accuracy of the key ROI area. Then, by processing the first face image set and establishing a base color feature database for bare face, a reliable base color range can be provided. When performing makeup recognition, by comparing the ROI area color features of the image to be inspected with the base color feature database for bare face, it is possible to effectively determine whether the image is in a bare face state. The reason for color comparison by part is that the color change characteristics of different facial regions under bare face and makeup states are different, so a single color range cannot accurately reflect the bare face or makeup situation of the entire face. Different areas of the face, such as lips, eyebrows, eyes, and cheeks, have significant differences in color and texture changes before and after makeup. By performing color comparison by part, the makeup situation of each area can be detected more accurately and misjudgment can be reduced, especially lipstick, eye shadow, beauty contact lens, eyebrow pencil, blush and contouring have a greater impact on different areas. If only the color average value of the entire face is used for comparison, it is easy to cause misjudgment due to the unclear overall color change caused by a certain local makeup. In addition, the part-by-part color comparison method can better adapt to various makeup styles and types, improve the flexibility and accuracy of recognition, and thus provide more reliable makeup recognition results.
[0076] See also Figure 2 In S40, if the makeup recognition result is a bare face, obtaining a first maximum similarity according to the image to be inspected and the first face image set; if the first maximum similarity is greater than a first threshold, outputting the person information corresponding to the first maximum similarity;
[0077] Furthermore, obtaining the first maximum similarity according to the image to be detected and the first face image set includes the following steps:
[0078] Input each image in the preprocessed first face image set into the pretrained convolutional neural network model to extract facial key point features to obtain M N-dimensional first key feature vectors; M≥1, M is the total number of authorized users entered.
[0079] Input the preprocessed image to be inspected into the convolutional neural network model to extract facial key point features to obtain an N-dimensional second key feature vector;
[0080] The similarities between the second key feature vector and the M first key feature vectors are respectively calculated to obtain M first similarities; and the maximum value among the M first similarities is selected as the first maximum similarity.
[0081] By introducing a pre-trained convolutional neural network model to extract facial key point features and calculating the similarity between the image to be tested and each image in the first face image set, the accuracy and robustness of facial recognition can be significantly improved. Taking the maximum value of multiple first similarities as the first maximum similarity can ensure that the image to be tested is matched with the most similar face image in the database, thereby effectively reducing the misjudgment rate.
[0082] In this embodiment, as an optional implementation, if the first maximum similarity is less than or equal to a first threshold, the target object is identified as a strange object and an early warning is issued.
[0083] See also Figure 2 In S50, if the makeup recognition result is makeup, respectively extracting bone structure features from the image to be inspected and the first face image set, calculating a second maximum similarity, and if the second maximum similarity is greater than a second threshold, outputting the person information corresponding to the second maximum similarity;
[0084] Further, respectively extracting the bone structure features from the image to be inspected and the first face image set, and calculating the second maximum similarity includes:
[0085] Inputting the preprocessed first face image set into a trained facial skeleton contour recognition model to obtain M K-dimensional first contour feature vectors;
[0086] Inputting the preprocessed image to be inspected into the trained facial bone contour recognition model to obtain a K-dimensional second contour feature vector;
[0087] respectively calculating similarities between the second contour feature vector and the M first contour feature vectors to obtain M second similarities; selecting a maximum value among the M second similarities as the second maximum similarity;
[0088] The facial bone contour recognition model includes:
[0089] Input layer, used to receive input images;
[0090] A facial contour segmentation layer, used to segment the facial area of the input image using a UNet model to obtain a facial segmentation area;
[0091] A feature extraction layer, using a multi-scale convolutional neural network structure, is used to extract features from the segmented facial segmentation area;
[0092] The attention mechanism layer includes a channel attention module and a spatial attention module, which are used to guide the facial skeleton contour recognition model to focus on key contour areas, including cheekbones, mandibular angles, brow bones and nose bridge, to enhance feature representation;
[0093] The feature fusion layer is used to fuse features from different levels and scales using an adaptive weighting mechanism;
[0094] The dynamic adjustment layer includes an environment perception module and a dynamic adjustment module; the environment perception module is used to detect the lighting conditions and makeup conditions of the input image in real time; the dynamic adjustment module is used to dynamically adjust the parameters of the feature extraction layer and the feature fusion layer according to the output result of the environment perception module;
[0095] The output layer is used to output the bone structure features.
[0096] By extracting the bone structure features of the image to be tested and the first face image set and calculating their similarity, the accuracy and robustness of face recognition can be further improved. The facial bone contour recognition model uses the UNet model to segment the facial region, and through the multi-scale convolutional neural network structure and attention mechanism layer, it can extract the contour features of key facial areas, such as cheekbones, mandibular angles, brow bones and nose bridge. These features are of great significance for recognizing facial contours, because they are usually not easily changed by makeup, so they can provide more stable and unique facial features. In addition, the feature fusion layer adopts an adaptive weighting mechanism to fuse features of different levels and scales, thereby enhancing the model's ability to recognize facial contours. The environmental perception module and dynamic adjustment module in the dynamic adjustment layer can detect the lighting conditions and makeup conditions of the input image in real time, and dynamically adjust the model parameters to ensure that features can be accurately extracted in various environments. The beneficial effect of this method is that by extracting and comparing facial bone structure features, face recognition can be performed more accurately, faces with high similarity can be effectively distinguished, and the robustness of the model in complex environments is enhanced. The attention mechanism layer enables the model to focus on the key areas of the facial contour, improving the effectiveness of feature extraction. By combining the extraction and comparison of facial key point features and bone structure features, a high-precision and high-robust makeup recognition solution is provided.
[0097] See also Figure 2 S60, otherwise (if the second maximum similarity is less than or equal to the second threshold), collect the second human ear image of the target object, extract features, and obtain the contour features of the human ear to be tested; extract the human ear features of the image to be tested, and obtain the position ratio features of the human ear to be tested; integrate the human ear contour features to be tested and the human ear position ratio features to be tested to obtain the integrated human ear features to be tested; obtain an integrated human ear feature set based on the first human ear image set and the first face image set; obtain a third maximum similarity based on the integrated human ear features to be tested and the integrated human ear feature set; if the third maximum similarity is greater than the third threshold, output the personnel information corresponding to the third maximum similarity;
[0098] Further, collecting the second human ear image of the target object, extracting features, and obtaining the contour features of the human ear to be tested; extracting the human ear features of the image to be tested, and obtaining the position ratio features of the human ear to be tested includes:
[0099] Performing feature extraction on the second human ear image to obtain the human ear contour feature set, including auricle contour features, cochlear structure features, tragus features, antitragus features, earlobe features, and texture features;
[0100] Among them, the tragus and antitragus are both ear structures. The tragus refers to the protrusion in front of the external auditory canal, which is composed of cartilage and skin and can cover the external auditory canal. There is a protrusion under the antihelix at the back, which is called the antitragus.
[0101] Use the pre-trained MTCNN model to detect the image to be tested and locate the key points of the face; further refine the detection through the key point detection model to identify the precise key points of the ear, including the top of the auricle, the earlobe, the tragus and the antitragus; determine the pre-recognition area of the ear based on the detected precise key points; use image segmentation technology to finely segment the pre-recognition area of the ear to obtain ear contour features; calculate the relative distance and ratio between the precise key points of the ear to obtain the ear position ratio feature; integrate the ear contour feature and the ear position ratio feature to obtain the ear position ratio feature of the person to be tested.
[0102] In this embodiment, the measured ear position ratio features include the ratio of the length from the top of the auricle to the earlobe to the length of the face, the ratio of the width from the tragus to the antitragus to the width of the face, the ratio of the vertical distance from the tragus to the outer corner of the eye to the length of the face, the ratio of the horizontal distance from the tragus to the wing of the nose to the width of the face, the ratio of the vertical distance from the earlobe to the chin to the length of the face, the ratio of the horizontal distance from the midpoint of the ear to the outer corner of the eye to the width of the face, the ratio of the vertical distance from the midpoint of the ear to the nose to the length of the face, the ratio of the width from the tragus to the antitragus to the width of the mouth, and the ratio of the width of the auricle to the width of the forehead.
[0103] By extracting features from the second ear image, the detailed structural features of the ear are obtained, including the auricle contour features, cochlear structure features, tragus features, antitragus features, earlobe features and texture features, providing rich ear feature information; using the pre-trained MTCNN model for detection and key point positioning, the accurate identification of key points is ensured, and the key points of the ear can be accurately identified through further refinement of the key point detection model; based on the detected precise key points, the pre-identification area of the ear is determined, and the image segmentation technology is used to finely segment it to obtain accurate ear contour features; by calculating the relative distance and ratio between the precise key points of the ear, the position and ratio features of the ear are obtained. After integration, these features can provide a more comprehensive and accurate ear description, thereby improving the accuracy and robustness of recognition. Especially under makeup and different lighting conditions, ear features, as auxiliary features independent of the face, can effectively supplement and verify facial recognition results, significantly improving the reliability and adaptability of the recognition method.
[0104] Further, according to the integrated human ear feature to be tested and the integrated human ear feature set, the third maximum similarity is obtained as follows:
[0105] The dimension of the integrated human ear feature set is H; the integrated human ear feature set includes M H-dimensional integrated human ear feature vectors; the similarities between the integrated human ear feature set and the M integrated human ear feature vectors are calculated respectively to obtain M third similarities; the maximum value of the M third similarities is selected as the third maximum similarity.
[0106] The ear contour feature provides unique geometric information, while the ear position ratio feature adds spatial layout information. The integration of the two features can effectively resist interference and reduce misjudgment. Through multi-feature fusion and similarity maximum selection, the unique biological characteristics of individuals can be captured more accurately, improving the accuracy of recognition.
[0107] See also Figure 2 In S70, otherwise (if the third maximum similarity is less than or equal to the third threshold), the target object is identified as a strange object and an early warning is issued.
[0108] Table 2. Threshold range table
[0109] index scope First threshold [0.90—1.00) Second threshold [0.80—0.90) The third threshold [0.70—0.80)
[0110] In this embodiment, the first threshold, the second threshold and the third threshold are designed in a step-by-step manner and decrease in sequence, as shown in Table 2. As a possible implementation, the first threshold is specifically 0.9, the second threshold is 0.8, and the third threshold is 0.7.
[0111] This face recognition method achieves efficient and accurate identity recognition by comprehensively utilizing makeup recognition, bone structure feature extraction, and further ear feature extraction of frontal facial images with exposed ears. When the target object is not wearing makeup, it can be quickly recognized by calculating the face similarity, which improves the recognition speed and efficiency; when the target object is wearing makeup, multiple verifications of bone structure features and ear features ensure the recognition accuracy and robustness under makeup. Especially under makeup, by extracting ear contour features and position ratio features, the recognition accuracy and reliability are further improved, avoiding recognition errors caused by relying solely on facial features. At the same time, the method provides an early warning mechanism when a known identity cannot be matched, improving the security and practicality of the recognition process.
[0112] Embodiment 2
[0113] The property management system of residential area B applies a face recognition method to identify the faces of owners, wherein the plain-face images of all owners have been pre-stored in the system's database.
[0114] A face recognition device, such as Figure 4 As shown, including:
[0115] An image acquisition module is used to acquire a frontal facial image of a target object with its ears exposed to obtain an image to be inspected;
[0116] An image acquisition module, used to acquire a first face image set and a first ear image set pre-stored by a designated group;
[0117] A makeup recognition module is used to perform makeup recognition on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup;
[0118] A first maximum similarity calculation module, configured to obtain a first maximum similarity according to the image to be inspected and the first face image set when the makeup recognition result is a natural face;
[0119] A second maximum similarity calculation module is used to extract bone structure features from the image to be tested and the first face image set respectively, and calculate a second maximum similarity when the makeup recognition result is makeup;
[0120] A third maximum similarity calculation module is used for, when the second maximum similarity is less than or equal to a second threshold, collecting a second human ear image of the target object, extracting features, and obtaining a human ear contour feature to be tested; extracting human ear features of the image to be tested, and obtaining a human ear position ratio feature to be tested; integrating the human ear contour feature to be tested and the human ear position ratio feature to be tested, and obtaining an integrated human ear feature to be tested; obtaining an integrated human ear feature set based on the first human ear image set and the first face image set; and obtaining a third maximum similarity based on the integrated human ear feature to be tested and the integrated human ear feature set;
[0121] A similarity judgment module, used to judge whether the first maximum similarity is greater than a first threshold; judge whether the second maximum similarity is greater than a second threshold; judge whether the third maximum similarity is greater than a third threshold;
[0122] The recognition result output module is used to output the personnel information corresponding to the first maximum similarity when the first maximum similarity is greater than the first threshold; output the personnel information corresponding to the second maximum similarity when the second maximum similarity is greater than the second threshold; output the personnel information corresponding to the third maximum similarity when the third maximum similarity is greater than the third threshold; and identify the target object as an unfamiliar object and issue a warning when the third maximum similarity is less than or equal to the third threshold.
[0123] In this embodiment, it also includes: an identity authentication module, a door opening control module and a recording notification module;
[0124] After receiving the personnel information, the identity verification module verifies their door opening authority. If the identified person has the authority, it sends a door opening signal to the door opening control module. After receiving the signal, the door opening control module drives the access control system to open the door lock and allow the person to enter. At the same time, the recording and notification module records the door opening operation and triggers an early warning when an unfamiliar object is identified, notifying the security management personnel to ensure timely response.
[0125] In addition, to achieve the above-mentioned purpose, the present invention also proposes a face recognition device, which includes at least a processor and a memory, and the processor is used to implement the steps of the method described above when executing a computer program stored in the memory.
[0126] In addition, to achieve the above-mentioned purpose, the present invention also proposes a face recognition storage medium, on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the method described above.
[0127] Before applying the face recognition system proposed in the present invention, the property management system B applied a traditional face recognition method (HOG+SVM). HOG is an image feature descriptor that describes the shape and texture information of an image by counting the distribution of the local gradient direction of the image. SVM is a supervised learning model commonly used for classification tasks. The HOG and SVM face recognition method combines the HOG feature and the SVM classifier, which can be used for face recognition tasks.
[0128] Table 3. Comparison data before and after the application of face recognition system
[0129]
[0130] As shown in Table 3, in traditional face recognition methods (such as HOG+SVM), the overall recognition accuracy is 85%, the recognition accuracy is 95% without makeup, and the recognition accuracy with makeup drops significantly to only 60%. After applying the face recognition system proposed in the present invention, the overall recognition accuracy is increased to 95%, the recognition accuracy without makeup remains at 95%, and the recognition accuracy with makeup is greatly improved to 90%. By introducing ear contour features and position ratio features, the system effectively solves the interference of makeup on the recognition results, and significantly improves the recognition accuracy and robustness in various complex environments.
[0131] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A face recognition method, characterized in that: include: Collect a frontal facial image of the target object with ears exposed to obtain an image to be inspected; Acquire a first face image set and a first ear image set pre-stored in a specified group; Performing makeup recognition on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup; If the makeup recognition result is no makeup, obtaining a first maximum similarity based on the image to be inspected and the first face image set; if the first maximum similarity is greater than a first threshold, outputting the person information corresponding to the first maximum similarity; If the makeup recognition result is makeup, extract bone structure features from the image to be tested and the first face image set respectively, calculate a second maximum similarity, and if the second maximum similarity is greater than a second threshold, output the person information corresponding to the second maximum similarity; Otherwise, collecting a second human ear image of the target object, extracting features, and obtaining contour features of the human ear to be measured; Extracting the ear features of the image to be tested to obtain the ear position ratio features to be tested; integrating the ear contour features to be tested and the ear position ratio features to be tested to obtain the integrated ear features to be tested; obtaining an integrated ear feature set based on the first ear image set and the first face image set; obtaining a third maximum similarity based on the integrated ear features to be tested and the integrated ear feature set; if the third maximum similarity is greater than a third threshold, outputting the personnel information corresponding to the third maximum similarity; Otherwise, the target object is identified as a strange object and an early warning is issued.
2. A face recognition method according to claim 1, characterized in that: The first face image set and the first ear image set are specifically: The first human ear image set consists of a number of first human ear images, wherein the first human ear images include a complete left ear image and a complete right ear image of the specified group, and the first human ear image contains the outline of the entire ear; the first human ear image is collected from the side of the members of the group by a high-resolution camera; the first human face image set includes the frontal facial images of the specified group without makeup and with ears exposed; the name of each member of the group has a one-to-one correspondence with the corresponding first human face image and first human ear image, and is stored in a database.
3. A face recognition method according to claim 1, characterized in that: Performing makeup recognition on the image to be inspected to obtain the makeup recognition result includes: The pre-processed image to be inspected is segmented into several key ROI regions by using the UNet model, including pupil, upper eyelid, lower eyelid, lip, eyebrow, nose, forehead and cheek, and a segmented image of each key ROI region is obtained; the key points of the face are located by using the pre-trained MTCNN model, and the key ROI region is corrected to obtain a corrected ROI region and a corrected segmented image corresponding to each corrected ROI region; the first face image set is subjected to the same pre-processing, segmentation and key point positioning to obtain a second corrected ROI region, and the plain face reference color range of each second corrected ROI region is calculated to establish a plain face reference color feature database; For each segmented image of the key ROI area, the color features of all pixels of the segmented image are extracted to obtain multiple ROI area color features; the ROI area color features of each image to be inspected are compared with the bare face reference color feature database; if all the ROI area color features of the image to be inspected are within the corresponding bare face reference color range, the makeup recognition result is bare face, otherwise the makeup recognition result is makeup.
4. A face recognition method according to claim 1, characterized in that: Obtaining the first maximum similarity according to the image to be detected and the first face image set comprises the following steps: Input each image in the preprocessed first face image set into a pretrained convolutional neural network model to extract facial key point features to obtain M N-dimensional first key feature vectors; Input the preprocessed image to be inspected into the convolutional neural network model to extract facial key point features to obtain an N-dimensional second key feature vector; The similarities between the second key feature vector and the M first key feature vectors are respectively calculated to obtain M first similarities; and the maximum value among the M first similarities is selected as the first maximum similarity.
5. A face recognition method according to claim 1, characterized in that: Extracting the bone structure features from the image to be inspected and the first face image set respectively, and calculating the second maximum similarity comprises: Inputting the preprocessed first face image set into a trained facial skeleton contour recognition model to obtain M K-dimensional first contour feature vectors; Inputting the preprocessed image to be inspected into the trained facial bone contour recognition model to obtain a K-dimensional second contour feature vector; respectively calculating similarities between the second contour feature vector and the M first contour feature vectors to obtain M second similarities; selecting a maximum value among the M second similarities as the second maximum similarity; The facial bone contour recognition model includes: Input layer, used to receive input images; A facial contour segmentation layer, used to segment the facial area of the input image using a UNet model to obtain a facial segmentation area; A feature extraction layer, using a multi-scale convolutional neural network structure, is used to extract features from the segmented facial segmentation area; The attention mechanism layer includes a channel attention module and a spatial attention module, which are used to guide the facial bone contour recognition model to focus on key contour areas, including cheekbones, mandibular angles, brow bones, and nose bridge; The feature fusion layer is used to fuse features from different levels and scales using an adaptive weighting mechanism; The dynamic adjustment layer includes an environment perception module and a dynamic adjustment module; the environment perception module is used to detect the lighting conditions and makeup conditions of the input image in real time; the dynamic adjustment module is used to dynamically adjust the parameters of the feature extraction layer and the feature fusion layer according to the output result of the environment perception module; The output layer is used to output the bone structure features.
6. A face recognition method according to claim 1, characterized in that: Acquire the second human ear image of the target object, extract features, and obtain the contour features of the human ear to be measured; Extracting the human ear feature of the image to be tested to obtain the position ratio feature of the human ear to be tested includes: Performing feature extraction on the second human ear image to obtain the human ear contour feature set, including auricle contour features, cochlear structure features, tragus features, antitragus features, earlobe features, and texture features; The image to be inspected is detected using a pre-trained MTCNN model to locate key points of the face; the detection is further refined through a key point detection model to identify precise key points of the ear, including the top of the auricle, the earlobe, the tragus, and the antitragus; a pre-recognized area of the ear is determined based on the detected precise key points; the pre-recognized area of the ear is finely segmented using an image segmentation technique to obtain ear contour features; The relative distance and ratio between the precise key points of the ear are calculated to obtain the ear position ratio feature; the ear contour feature and the ear position ratio feature are integrated to obtain the ear position ratio feature of the person to be tested.
7. A face recognition method according to claim 1, characterized in that: According to the integrated human ear feature to be tested and the integrated human ear feature set, the third maximum similarity is obtained as follows: The dimension of the integrated human ear feature set is H; the integrated human ear feature set includes M H-dimensional integrated human ear feature vectors; the similarities between the integrated human ear feature set and the M integrated human ear feature vectors are calculated respectively to obtain M third similarities; the maximum value of the M third similarities is selected as the third maximum similarity.
8. A face recognition device, characterized in that: include: An image acquisition module is used to acquire a frontal facial image of a target object with its ears exposed to obtain an image to be inspected; An image acquisition module, used to acquire a first face image set and a first ear image set pre-stored by a designated group; A makeup recognition module is used to perform makeup recognition on the image to be inspected to obtain a makeup recognition result, including makeup and no makeup; A first maximum similarity calculation module, configured to obtain a first maximum similarity according to the image to be inspected and the first face image set when the makeup recognition result is a natural face; A second maximum similarity calculation module is used to extract bone structure features from the image to be tested and the first face image set respectively, and calculate a second maximum similarity when the makeup recognition result is makeup; A third maximum similarity calculation module is used to collect a second human ear image of the target object, extract features, and obtain a human ear contour feature to be measured when the second maximum similarity is less than or equal to a second threshold; Extracting the ear features of the image to be tested to obtain the ear position ratio features to be tested; integrating the ear contour features to be tested and the ear position ratio features to be tested to obtain the integrated ear features to be tested; obtaining an integrated ear feature set based on the first ear image set and the first face image set; obtaining a third maximum similarity based on the integrated ear features to be tested and the integrated ear feature set; A similarity judgment module, used to judge whether the first maximum similarity is greater than a first threshold; judge whether the second maximum similarity is greater than a second threshold; judge whether the third maximum similarity is greater than a third threshold; The recognition result output module is used to output the personnel information corresponding to the first maximum similarity when the first maximum similarity is greater than the first threshold; output the personnel information corresponding to the second maximum similarity when the second maximum similarity is greater than the second threshold; output the personnel information corresponding to the third maximum similarity when the third maximum similarity is greater than the third threshold; and identify the target object as an unfamiliar object and issue a warning when the third maximum similarity is less than or equal to the third threshold.
9. A face recognition device, characterized in that: The device comprises at least a processor and a memory, and the processor is configured to implement the steps of the method according to any one of claims 1 to 7 when executing a computer program stored in the memory.
10. A face recognition storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Facial recognition methods and devices
CN109165546B
A method for automatic facial image recognition before and after makeup
CN109858392B
Face recognition method, device and electronic equipment
CN112800819B
