Facial attribute recognition method, device and electronic device compatible with occlusion
The facial attribute recognition method enhances accuracy and robustness by differentiating between occluded and unoccluded facial images and training a model to handle both types effectively, maintaining high recognition performance across varying conditions.
Patent Information
- Application Number
- CN202011059698.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-30
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2040-09-30
AI Technical Summary
Existing facial attribute recognition models suffer from reduced accuracy due to occlusion, as they fail to effectively handle both occluded and unoccluded facial images, leading to erroneous information and low robustness.
A facial attribute recognition method that distinguishes between global and partial target images based on occlusion, using a pre-trained model to process complete or partial facial images, and employs a training process that minimizes loss values to enhance recognition accuracy.
The method improves the accuracy and robustness of facial attribute recognition by adapting to both occluded and unoccluded scenarios, ensuring high recognition accuracy without reducing performance in non-occluded conditions.
Smart Images
Figure CN112149601B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a facial attribute recognition method, device and electronic device compatible with occlusion. Background Art
[0002] The technology of recognizing facial attributes based on facial images has received widespread attention. When the face is occluded, the accuracy of facial attribute recognition will be seriously affected. Occlusion destroys the inherent structure and geometric features of the face, and seriously reduces the richness of facial feature information.
[0003] When using the model to recognize facial attributes, the model is directly used to recognize facial images, that is, feature extraction and attribute classification are directly performed. This will introduce erroneous information, resulting in relatively low accuracy of model recognition. Summary of the invention
[0004] In view of this, the purpose of the embodiments of the present application is to provide a facial attribute recognition method, device and electronic device that are compatible with occlusion, which reduces the impact of occlusion on facial attribute recognition when recognizing occluded facial images and unobstructed facial images through one model, thereby improving the accuracy and robustness of facial attribute recognition.
[0005] In a first aspect, an embodiment of the present application provides a facial attribute recognition method compatible with occlusion, the method comprising:
[0006] Acquire a facial image captured for a target user;
[0007] According to the occlusion of the facial image, a global target image or a partial target image is determined from the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded;
[0008] The global target image or a part of the target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user.
[0009] In one embodiment, inputting the global target image or the partial target image into a pre-trained facial attribute recognition model includes:
[0010] Determine whether there is occlusion on the face in the facial image;
[0011] If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model;
[0012] If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
[0013] In one implementation, determining a partial target image from the facial image includes:
[0014] The pixel values of the pixels in the occluded area of the facial image are set to a preset value.
[0015] In one embodiment, after acquiring the facial image captured for the target user, the method further includes:
[0016] Detecting facial key point information from the facial image;
[0017] Based on the key point information, the facial image is calibrated.
[0018] In one embodiment, the facial attribute recognition model is pre-trained by the following steps:
[0019] The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes:
[0020] The facial attribute recognition training model obtains multiple image pairs in the training sample library, where one image pair includes a global target image and a partial target image, the global target image includes a complete facial image of a reference object, and the partial target image includes a partial facial image of the reference object;
[0021] Extracting features from the global target image to obtain a first feature map, and extracting features from a portion of the target image to obtain a second feature map;
[0022] A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map;
[0023] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition;
[0024] When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training;
[0025] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0026] In one embodiment, adjusting the model parameters includes:
[0027] According to the principle that a first loss value between the first predicted attribute and the actual attribute of the first sample image and a second loss value between the second predicted attribute and the actual attribute of the second sample image are simultaneously minimized, the model parameters of the incomplete facial attribute recognition training model are adjusted.
[0028] In a second aspect, an embodiment of the present application provides a facial attribute recognition device compatible with occlusion, the device comprising:
[0029] An acquisition module is used to acquire a facial image taken of a target user;
[0030] A processing module is used to determine a global target image or a partial target image from a facial image according to the occlusion condition of the facial image; the global target image includes a complete facial image of a target user, and the partial target image includes a partial facial image of the target user that is not occluded; the global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user.
[0031] In one embodiment, the processing module is used to input the global target image or the partial target image into a pre-trained facial attribute recognition model according to the following steps:
[0032] Determine whether there is occlusion on the face in the facial image;
[0033] If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model;
[0034] If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
[0035] In one implementation, the processing module is further configured to set pixel values of pixels in the occluded area of the facial image to preset values.
[0036] In one embodiment, the processing module is further used to detect facial key point information from the facial image after acquiring the facial image taken for the target user, and calibrate the facial image based on the key point information.
[0037] In one embodiment, the device further comprises a facial attribute recognition model pre-training module, which is used to pre-train the facial attribute recognition model by the following steps:
[0038] The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes:
[0039] The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object;
[0040] Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map;
[0041] A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map;
[0042] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition;
[0043] When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training;
[0044] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0045] In one embodiment, the facial attribute recognition model pre-training module is also used to adjust the model parameters of the incomplete facial attribute recognition training model according to the principle of minimizing the first loss value and the second loss value at the same time when adjusting the model parameters.
[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a storage medium, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the method of the first aspect described above.
[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in the first aspect are executed.
[0048] The occlusion-compatible facial attribute recognition method provided in the embodiment of the present application obtains a facial image taken for a target user, and determines a global target image or a partial target image from the facial image according to the occlusion condition of the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded; the global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user, so that by associating the global target image or the partial target image of the same target user and inputting it into the facial attribute recognition training model to be trained, the final model for recognizing facial attributes is trained, thereby avoiding the influence of the occlusion area on the facial attribute recognition, and realizing one model for recognizing occluded facial images and unoccluded facial images without reducing the recognition accuracy of the model.
[0049] Furthermore, according to one embodiment, by determining whether the face is obstructed, the unobstructed facial area is retained for attribute recognition, ensuring that targeted processing can be performed according to the facial occlusion situation. For example, for a face with partial occlusion, such as lower occlusion, a more representative upper unobstructed area of the face is selected to fully utilize the feature information of the unobstructed part of the face, reduce the impact of obstructions on the overall facial attribute recognition, and improve the accuracy and robustness of facial attribute recognition. The facial attribute recognition method of the present application is compatible with occluded / unobstructed face scenes. While ensuring the recognition accuracy of non-occluded face scenes, it can also adapt well to scenes with occluded faces, expanding the application scope and scenarios of facial attribute recognition algorithms.
[0050] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0052] Figure 1 A first flow chart of a facial attribute recognition method compatible with occlusion provided in an embodiment of the present application is shown;
[0053] Figure 2 A second flow chart of a facial attribute recognition method compatible with occlusion provided in an embodiment of the present application is shown;
[0054] Figure 3A schematic diagram of the structure of a facial attribute recognition device compatible with occlusion provided in an embodiment of the present application is shown;
[0055] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0057] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0058] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0059] When recognizing facial attributes such as human face attributes (such as gender, age, expression, etc.), different models are generally used to recognize images with occlusion and images without occlusion. For occluded images, when training a model to recognize occluded images, the model is generally trained using local non-occluded faces as samples. The model trained in this way has poor robustness and low recognition accuracy.
[0060] Based on this, the present application proposes a facial attribute recognition method compatible with occlusion, the method comprising: obtaining a facial image taken for a target user; determining a global target image or a partial target image from the facial image according to the occlusion of the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded; the global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user. A complete facial image or a facial image with occlusion is recognized by a model. The facial attribute recognition model is obtained by training using at least one training sample, each training sample is composed of a global target image and a partial target image of the same reference object, the global target image is an image of the reference object that is not occluded, the partial target image is the same as the global target image, and the different parts are in an occluded state, that is, the partial target image is a partially occluded image of the global target image. In this way, the facial attribute recognition model to be trained is input into the global target image and the partial target image of the same reference object, and the model of the final recognition attribute is trained, thereby avoiding the influence of the occluded area on the facial attribute recognition, and realizing a model to recognize the facial image with occlusion and the facial image without occlusion without reducing the recognition accuracy of the model. The embodiments of the present application will be described in detail based on this idea.
[0061] The training of the facial attribute recognition model is first described below. In the embodiment of the present application, the facial attribute recognition model is pre-trained by the following steps:
[0062] The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes:
[0063] The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object;
[0064] Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map;
[0065] A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map;
[0066] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition;
[0067] When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training;
[0068] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0069] Here, both the predicted attributes and the actual attributes are facial attributes, and the facial attributes include at least one of the attributes under the gender category, the attributes under the age category, or the attributes under the expression category; the first feature map can be an eigenvalue matrix, and the eigenvalues in the matrix represent the facial attribute feature information included in the corresponding area in the first sample image; the second feature map can also be a feature matrix, and the eigenvalues in the matrix represent the facial attribute feature information included in the corresponding area in the second sample image, wherein the size of the area in the first sample image and the size of the area in the second sample image can be determined according to actual conditions; the feature extraction model can be a combination of convolutional neural network layers, activation functions (for example, ReLU, Tanh, Sigmoid, etc.), batch normalization operations (Batch Normalization), and other layers; the first predicted attribute and the second predicted attribute are both predicted attributes, and the attribute category of the predicted attribute is the same as the attribute category of the actual attribute.
[0070] In a specific implementation process, when acquiring a training sample library, a large number of sample images may be acquired, and actual attributes of reference objects included in the sample images may be manually annotated.
[0071] Taking into account that when shooting the sample image, the camera is not necessarily facing the face of the reference object, it is necessary to calibrate the reference object in the sample image so that the calibrated sample image is the standard face image of the reference object (the standard face image is obtained when the camera is facing the face of the reference object), and the calibrated sample image is used as the sample image in the training sample library.
[0072] When calibrating a sample image, facial key point information of the image of a reference object included in the sample image, i.e., facial key points (such as eyes, nose, mouth, etc.) can be detected, and position information of the facial key points in the sample image can be obtained. Using the above position information obtained, the reference object in the sample image can be aligned with its face, that is, the face of the reference object in the sample image can be calibrated to a standard face. In this way, it can be ensured that the position of the reference object in different sample images is the same.
[0073] After obtaining the first sample image, part of the area in the first sample image can be blocked. The area to be blocked is a fixed area or a non-fixed area. For the fixed area, the position of the fixed area in the image is fixed, and the size of the fixed area is preset. The size can be determined according to the proportion of the blocked part of the object in the facial image in the facial image. For example, the area that the facial attribute recognition model to be trained needs to recognize is the area excluding the mask area, and the mask area is located in the lower middle part of the facial image. In this case, the fixed area at a fixed position in the facial image needs to be blocked.
[0074] When a partial area in the first sample image is blocked, the pixel values of the pixel points in the partial area can be set to a preset value, or a blank image can be used to block the partial area and retain the remaining unblocked image, and the image after blocking the first sample image is used as the second sample image.
[0075] After obtaining the first sample image and the second sample image, the first sample image and the second sample image of the same reference object can be associated and input into the feature extraction model of the incompletely trained facial attribute recognition model, that is, the first sample image and the second sample image are input into the feature extraction model of the incompletely trained facial attribute recognition model as an image pair, and the feature extraction model extracts feature information from the first sample image and from the second sample image, respectively, to obtain a first feature map corresponding to the first sample image and a second feature map corresponding to the second sample image. The first feature map includes more facial attribute information than the second feature map.
[0076] The first feature map and the second feature map are respectively associated and input into an attribute classification model in an incompletely trained facial attribute recognition model, and the attribute classification model outputs a first predicted attribute corresponding to the first sample image and a second predicted attribute corresponding to the second sample image.
[0077] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition.
[0078] When it is determined that the training is not completed, the model parameters are adjusted and the next iterative training is performed. When adjusting the model parameters of the facial attribute recognition model that has not completed the training, the first loss value between the first predicted attribute and the actual attribute of the first sample image, and the second loss value between the second predicted attribute and the actual attribute of the second sample image can be calculated, and the model parameters of the facial attribute recognition model that has not completed the training are adjusted according to the principle that the first loss value and the second loss value are simultaneously minimized. The first loss value can be the distance or similarity between the first predicted attribute and the actual attribute of the first sample image, and the second loss value can be the distance or similarity between the second predicted attribute and the actual attribute of the second sample image. The algorithm for calculating the distance and similarity is a conventional algorithm in the art, wherein the conventional gradient descent algorithm can be used to complete the update of the parameters in the model, and the L1 loss function is not described in detail here.
[0079] In order to improve the recognition accuracy of the facial attribute recognition model, when adjusting the model parameters, the similarity between the first feature map and the second feature map can also be considered, so that the second feature map is infinitely close to the first feature map, that is, the feature information extracted from the partial facial image is close to the feature information extracted from the global facial image, that is, the third loss value between the first feature map and the second feature map is calculated, and the model parameters of the facial attribute recognition model are adjusted according to the principle that the first loss value, the second loss value and the third loss value are simultaneously minimized to obtain a facial attribute recognition model that has completed training. The third loss value can be the distance or similarity between the first feature map and the second feature map.
[0080] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0081] After the facial attribute recognition model is obtained, the facial attribute recognition model can be used to perform facial attribute recognition. In view of this situation, the embodiment of the present application provides a facial attribute recognition method compatible with occlusion, such as Figure 1 As shown, the method comprises the following steps:
[0082] S101, obtaining a facial image of a target user.
[0083] Here, the facial image is collected by an image acquisition device, which may be a camera, a photo taking device, etc. The image acquisition device may be a device in an access control system, such as an access control system set up by an enterprise or an access control system set up by a residential community. It may also be an image acquisition device set up indoors or outdoors, such as a shopping mall entrance, an office building entrance, a public transportation security checkpoint, etc. It can be determined based on actual conditions, but it should be understood that the present application is not limited to this.
[0084] S102, determining a global target image or a partial target image from the facial image according to the occlusion condition of the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded;
[0085] Here, the collected facial image may be occluded. In the case of occlusion, for example, when the target user wears a mask, the masked area is the occluded area in the facial image. The facial image is used as a partial target image. In the case of no occlusion, the facial image without occlusion is used as the global target image.
[0086] Here, if it is determined that the target user in the facial image is blocked, a partial target image is determined from the facial image, and the partial target image is input into the facial attribute recognition model; the partial target image is identical to a portion of the facial image, and a different portion is in a blocked state;
[0087] If it is determined that the target user in the facial image is not blocked, the facial image is input into a pre-trained facial attribute recognition model.
[0088] Here, the algorithm for detecting whether there is occlusion in the facial image includes an image classification method based on deep learning, etc. The above method can be used to detect mask occlusion, sunglasses occlusion, etc. in the facial image. The aforementioned method may be a conventional method in the field, and this application is not limited to this.
[0089] The size of the partial target image may be the same as that of the facial image. The partial target image is obtained by performing an occlusion process on the facial image, that is, occluding a fixed area in the facial image, which is described in detail below.
[0090] When it is determined that the target user in the facial image is blocked, a portion of the target image is determined from the facial image according to the following steps:
[0091] Before determining a portion of the target image from a facial image, facial key point information may be detected from the facial image, wherein the key point information is facial key point information, and the facial key points include eyes, nose, mouth, etc. The facial key point information is position information of the facial key points in the facial image.
[0092] The target user in the facial image is calibrated using the facial key point information. That is, when the face of the target user in the facial image is not a standard face, the face of the target user in the facial image needs to be calibrated to a standard face to obtain a calibrated facial image. The calibrated facial image meets the input requirements of the model for the image size. This makes it easier for the model to recognize.
[0093] Since the occluded area in the occluded image that can be recognized by the trained facial attribute recognition model is fixed, after obtaining the calibrated facial image, the pixel values of the pixels in the image of the designated area are set to the preset values, thereby obtaining a partial target image. The position of the designated area in the facial image is preset, for example, it can be the area under the nose, or the area where the eyes are located, and the shape of the area can be a quadrilateral (such as a rectangle), which can be determined according to actual conditions.
[0094] S103, inputting the global target image or the partial target image into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user.
[0095] Here, the attribute recognition result includes at least one of the attribute recognition result under the gender category, the attribute recognition result under the expression category, and the attribute recognition result under the age category. For example, when the attribute recognition result is the attribute recognition result under the gender category, the result can be the gender of male or female. When the attribute recognition result is the attribute recognition result under the age category, the result can be the age of 15, the age of 27, etc. When the attribute recognition result is the attribute recognition result under the expression category, the result can be a happy expression, a sad expression, etc.
[0096] Both the global target image and the partial target image are face images, and the sizes of the global target image and the partial target image are the same. The difference between the global target image and the partial target image is that the pixel values of the pixels of the partial image in the partial target image that is in an occluded state are preset values, and the preset values can be set according to actual conditions. For example, the preset values are 0, 1, 20, etc. When the preset value is 0, it indicates that the occluded part of the global target image is black.
[0097] As an optional embodiment, the area corresponding to the partial target image in the occluded state is a fixed area. For example, when the target user wears a mask, the partial target image in the occluded state is the area occluded by the mask. For another example, when the target user wears sunglasses, the partial target image in the occluded state is the area occluded by the sunglasses. The area where the partial target image in the occluded state in the facial image is located determines the occluded area in the facial image that can be recognized by the facial attribute recognition model. That is, the occluded area in the facial image of the target user is the same as the occluded area in the sample image. For such images, the recognition accuracy of the facial attribute recognition model is relatively high.
[0098] Here, after obtaining a partial target image, the partial target image is input into a facial attribute recognition model for feature extraction to obtain a feature map corresponding to the partial target image, and the obtained feature map is input into an attribute classification model of the facial attribute recognition model to obtain an attribute recognition result of the target user.
[0099] For example, the target user is a female, and the facial image is an image of a person wearing a mask. After obtaining the facial image, the pixel value of the mask-covered area in the facial image can be set to 0 to obtain a partial target image. The partial target image is input into the facial attribute recognition model to obtain the attribute recognition result of the target user as a female wearing a mask.
[0100] When the target user in the facial image is not obscured, the facial image can be input into the feature extraction model in the facial attribute recognition model to extract global feature information, and the feature extraction result is input into the attribute classification model of the facial attribute classification model to obtain the attribute recognition result of the target user, for example, the attribute recognition result is female, not wearing a mask, etc.
[0101] In one embodiment, reference Figure 2 After obtaining the facial image to be recognized, the facial key point information is detected, and the facial image is calibrated / aligned based on the facial key point information, and it is determined whether there is occlusion in the calibrated / aligned facial image. If so, the non-occluded area of the face is retained in the facial image to obtain a partial target image, and the partial target image is input into the facial attribute recognition model to obtain the attribute recognition result. If not, the entire facial image is input into the facial attribute recognition model as the global target image to obtain the attribute recognition result.
[0102] Through the above-mentioned facial attribute recognition model, it is possible to recognize facial images in which the face is occluded, and it is also possible to recognize facial images in which the face is not occluded.
[0103] Reference Figure 3 FIG. 1 is a schematic diagram of a facial attribute recognition device compatible with occlusion provided in an embodiment of the present application, the device comprising:
[0104] An acquisition module 31 is used to acquire a facial image captured of a target user;
[0105] The processing module 32 is used to determine a global target image or a partial target image from the facial image according to the occlusion condition of the facial image; the global target image includes the complete facial image of the target user, and the partial target image includes the partial facial image of the target user that is not occluded; the global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain the attribute recognition result of the target user.
[0106] In one embodiment, the processing module 32 is used to input the global target image or the partial target image into a pre-trained facial attribute recognition model according to the following steps:
[0107] Determine whether there is occlusion on the face in the facial image;
[0108] If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model;
[0109] If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
[0110] In one embodiment, the processing module 32 is further configured to:
[0111] Detecting facial key point information from the facial image;
[0112] Based on the key point information, the facial image is calibrated.
[0113] The processing module 32 is used to determine the target image from the facial image according to the following steps:
[0114] In the calibrated facial image, the pixel values of the pixels in the image of the designated area are set to the preset values.
[0115] In one embodiment, the apparatus further includes a training module 33 (i.e., a facial attribute recognition model pre-training module) for pre-training a facial attribute recognition model by the following steps:
[0116] The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes:
[0117] The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object;
[0118] Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map;
[0119] A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map;
[0120] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition;
[0121] When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training;
[0122] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0123] In one implementation, when adjusting the model parameters, the training module 33 adjusts the model parameters of the untrained facial attribute recognition training model according to the principle that the first loss value and the second loss value are simultaneously minimized.
[0124] In one implementation, in the second sample image, pixel values of pixel points of the partial image in an occluded state are preset values.
[0125] The present application embodiment also provides an electronic device 40, such as Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device 40 provided in an embodiment of the present application, including:
[0126] Processor 41, memory 42, and bus 43. The memory 42 stores machine-readable instructions executable by the processor 41 (e.g., Figure 3 The execution instructions corresponding to the acquisition module 31 and the processing module 32 are obtained in the device, etc. When the electronic device 40 is running, the processor 41 communicates with the memory 42 through the bus 43, and the machine-readable instructions are executed by the processor 41 to perform the following processing:
[0127] Acquire a facial image captured for a target user;
[0128] According to the occlusion of the facial image, a global target image or a partial target image is determined from the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded;
[0129] The global target image or a part of the target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user.
[0130] In a possible implementation manner, the instructions executed by the processor 41, inputting the global target image or the partial target image into a pre-trained facial attribute recognition model, include:
[0131] Determine whether there is occlusion on the face in the facial image;
[0132] If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model;
[0133] If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
[0134] In a possible implementation manner, in the instruction executed by the processor 41,
[0135] Determining a partial target image from the facial image includes:
[0136] The pixel values of the pixels in the occluded area of the facial image are set to a preset value.
[0137] In a possible implementation manner, the instructions executed by the processor 41, after acquiring the facial image captured for the target user, further include:
[0138] Detecting facial key point information from the facial image;
[0139] Based on the key point information, the facial image is calibrated.
[0140] In a possible implementation manner, in the instructions executed by the processor 41, the facial attribute recognition model is trained according to the following steps:
[0141] The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes:
[0142] The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object;
[0143] Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map;
[0144] A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map;
[0145] Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition;
[0146] When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training;
[0147] When it is determined that the training is completed, the facial attribute recognition model is obtained.
[0148] In a possible implementation manner, in the instruction executed by the processor 41, the adjustable parameter includes:
[0149] According to the principle that the first loss value and the second loss value are simultaneously minimized, the model parameters of the facial attribute recognition training model whose training has not been completed are adjusted.
[0150] As those skilled in the art know, with the development of computer hardware, the specific implementation method and name of the bus may undergo various changes. The bus referred to here conceptually covers any information transmission line that can provide services to various components in an electronic device, including but not limited to FSB, HT, QPI, Infinity Fabric, etc.
[0151] In the embodiments of the present application, the processor may be a general-purpose processor, including a central processing unit (CPU), or a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a tensor processing unit (TPU), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0152] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned facial attribute recognition method compatible with occlusion are executed.
[0153] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned facial attribute recognition method compatible with occlusion can be executed. The present application obtains a facial image taken for a target user, and determines a global target image or a partial target image from the facial image according to the occlusion of the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded. The global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain the attribute recognition result of the target user. In this way, the facial attribute recognition model to be trained is input by associating the global target image and the partial target image of the same reference object, and the model for finally recognizing the attribute is trained, thereby avoiding the influence of the occlusion area on the attribute recognition, and realizing a model for recognizing occluded facial images and unoccluded facial images without reducing the recognition accuracy of the model. The present application method can improve the accuracy and robustness of facial attribute recognition.
[0154] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0155] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0157] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0158] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A facial attribute recognition method compatible with occlusion, characterized in that: The method includes: Acquire a facial image captured for a target user; According to the occlusion of the facial image, a global target image or a partial target image is determined from the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded; Inputting the global target image or the partial target image into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user; Inputting the partial target image into a pre-trained facial attribute recognition model, including: Determine whether there is occlusion on the face in the facial image; If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model; Determining a partial target image from the facial image includes: Setting the pixel values of the pixels in the occluded area of the facial image to a preset value; The facial attribute recognition model is pre-trained by the following steps: The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes: The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object; Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map; A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map; Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition; When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training; When it is determined that the training is completed, obtaining the facial attribute recognition model; According to the principle that the first loss value and the second loss value are simultaneously minimized, adjusting the model parameters of the facial attribute recognition training model whose training has not been completed; In order to improve the recognition accuracy of the facial attribute recognition model, when adjusting the model parameters, according to the similarity between the first feature map and the second feature map, the second feature map is made infinitely close to the first feature map, that is, the feature information extracted from the partial facial image is made close to the feature information extracted from the global facial image, that is, the third loss value between the first feature map and the second feature map is calculated, and according to the principle that the first loss value, the second loss value and the third loss value are simultaneously minimized, the model parameters of the facial attribute recognition model are adjusted to obtain a facial attribute recognition model that has completed training, wherein the third loss value can be the distance or similarity between the first feature map and the second feature map.
2. The method according to claim 1, characterized in that Inputting the global target image into a pre-trained facial attribute recognition model, comprising: Determine whether there is occlusion on the face in the facial image; If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
3. The method according to claim 1, characterized in that After obtaining the facial image shot for the target user, the following steps are also included: Detecting facial key point information from the facial image; Based on the key point information, the facial image is calibrated.
4. A facial attribute recognition device compatible with occlusion, characterized in that: The device includes: An acquisition module is used to acquire a facial image taken of a target user; A processing module is used to determine a global target image or a partial target image from the facial image according to the occlusion of the facial image; the global target image includes a complete facial image of the target user, and the partial target image includes a partial facial image of the target user that is not occluded; the global target image or the partial target image is input into a pre-trained facial attribute recognition model to obtain an attribute recognition result of the target user; The processing module is used to input the partial target image into a pre-trained facial attribute recognition model according to the following steps: Determine whether there is occlusion on the face in the facial image; If it is determined that there is occlusion, a partial target image is determined from the facial image, and the partial target image is input into a pre-trained facial attribute recognition model; The processing module is used to determine a partial target image from the facial image according to the following steps: Setting the pixel values of the pixels in the occluded area of the facial image to a preset value; The facial attribute recognition model is pre-trained by the following steps: The facial attribute recognition training model is trained multiple times based on the training sample library. One iteration of training includes: The facial attribute recognition training model obtains a plurality of image pairs in a training sample library, wherein one image pair includes a first sample image and a second sample image, wherein the first sample image includes a complete facial image of a reference object, and the second sample image includes a partial facial image of the reference object; Extracting features from the first sample image to obtain a first feature map, and extracting features from the second sample image to obtain a second feature map; A first predicted attribute is obtained according to the prediction of the first feature map, and a second predicted attribute is obtained according to the prediction of the second feature map; Determine a first loss value between a first predicted attribute and an actual attribute of a first sample image, determine a second loss value between a second predicted attribute and an actual attribute of a second sample image, and determine whether the facial attribute recognition training model is trained based on the first loss value and the second loss value and a predetermined convergence condition; When it is determined that the training is not completed, adjust the model parameters and conduct the next iterative training; When it is determined that the training is completed, obtaining the facial attribute recognition model; According to the principle that the first loss value and the second loss value are simultaneously minimized, adjusting the model parameters of the facial attribute recognition training model whose training has not been completed; In order to improve the recognition accuracy of the facial attribute recognition model, when adjusting the model parameters, according to the similarity between the first feature map and the second feature map, the second feature map is made infinitely close to the first feature map, that is, the feature information extracted from the partial facial image is made close to the feature information extracted from the global facial image, that is, the third loss value between the first feature map and the second feature map is calculated, and according to the principle that the first loss value, the second loss value and the third loss value are simultaneously minimized, the model parameters of the facial attribute recognition model are adjusted to obtain a facial attribute recognition model that has completed training, wherein the third loss value can be the distance or similarity between the first feature map and the second feature map.
5. The device according to claim 4, characterized in that The processing module is used to input the global target image or the partial target image into a pre-trained facial attribute recognition model according to the following steps: Determine whether there is occlusion on the face in the facial image; If it is determined that there is no occlusion, the facial image is determined as a global target image, and the global target image is input into a pre-trained facial attribute recognition model.
6. An electronic device, characterized in that: include: A processor and a storage medium, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of any method described in claims 1 to 3.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are executed.