Face attribute recognition method and device, electronic equipment and computer storage medium
By extracting and matching facial recognition technology in the open world, the efficiency and accuracy of the existing technology in the recognition of the elderly or children in public places is solved, and accurate recognition and efficient matching of any face attributes are achieved.
Patent Information
- Application Number
- CN202411442035.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-03-18
AI Technical Summary
In the open world, existing facial recognition technology encounters the problem of single model performance and single input, and it is difficult to identify the loss of elderly people or children in public places such as airports, railway stations, shopping malls, etc., resulting in waste of manpower and material resources and poor results.
By acquiring the search data, including sub-data of at least one modality, feature extraction is performed, a first feature vector is generated, and matched with the updated face image database. If there is a matching target feature vector, the target face image is returned as the search result.
It realizes accurate recognition of any face attributes in the open world, and is not limited by a single image modality, face attribute types and complexity, improves recognition efficiency and accuracy, and is suitable for public safety scenarios and personalized services.
Smart Images
Figure CN119646247A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a face attribute recognition method, a face attribute recognition device, an electronic device and a computer-readable storage medium. Background Art
[0002] With the development of society, the demand for public security monitoring is growing, and the application scope of face recognition has been seen everywhere. However, face recognition technology can only distinguish whether it is a whitelist or a blacklist. For public places such as airports, railway stations, and shopping malls, if an elderly person or a child is missing, the face recognition algorithm fails due to the lack of data on the missing person, and security can only use cameras to find the missing person, which not only wastes manpower and material resources, but also has poor results. If the search range can be narrowed, it will save a lot of time and the target's position can be quickly located. The use of face attribute recognition methods can quickly locate the person's position and narrow the search range.
[0003] Traditional face attribute recognition methods require providing pre-defined attribute features or using trained models. They are generally only applicable to specific face attributes, and their input is often a face image. However, these methods have encountered challenges in the open world, namely, the single model performance and single model input. Since there is no face image in actual applications, there is a need for arbitrary face attribute recognition when facing open world applications, and it is not restricted by model input. Therefore, a method that can accurately and efficiently recognize the attributes of arbitrary faces in the open world is needed. Summary of the invention
[0004] In view of the above problems, embodiments of the present invention are proposed to provide a face attribute recognition method, a face attribute recognition device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.
[0005] The embodiment of the present invention discloses a method for recognizing facial attributes, the method comprising:
[0006] Acquire search data; the search data includes sub-data of at least one modality;
[0007] Perform feature extraction on the sub-data of each modality to obtain a sub-feature vector corresponding to the sub-data of each modality, and generate a first feature vector corresponding to the search data based on each sub-feature vector;
[0008] Matching the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image;
[0009] If there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as the retrieval result.
[0010] In one or more embodiments, before obtaining the retrieved data, the method further includes:
[0011] The preset face image database is updated to obtain an updated face image database.
[0012] In one or more embodiments, updating the preset face image database to obtain an updated face image database includes:
[0013] Get the face image to be stored;
[0014] Generate a first candidate face image of at least one scale based on the face image to be stored; the face image to be stored has corresponding face attribute description information;
[0015] Based on each of the first candidate facial images and the facial attribute description information, determining at least one second candidate facial image from each of the first candidate facial images;
[0016] Generate a final second feature vector based on the face image to be stored and each of the second candidate face images;
[0017] The face image to be stored and the second feature vector are stored in a face image database.
[0018] In one or more embodiments, generating a first candidate face image of at least one scale based on the face image to be stored includes:
[0019] Image extraction is performed on the face images to be stored using at least one preset scale to obtain at least one first candidate face image corresponding to each scale; the scale is used to characterize the proportion of the first candidate face image in the face images to be stored.
[0020] In one or more embodiments, the determining at least one second candidate facial image from each of the first candidate facial images based on each of the first candidate facial images and the facial attribute description information includes:
[0021] Performing feature extraction on each of the first candidate facial images to obtain a first image feature vector corresponding to each of the first candidate facial images;
[0022] Performing feature extraction on the facial attribute description information to obtain a text feature vector;
[0023] Perform feature similarity matching on each of the first image feature vectors and the text feature vector to obtain a feature similarity distribution map corresponding to each of the first candidate face images;
[0024] Determine the top N target feature similarity distribution maps from each of the feature similarity distribution maps based on the order of variance from large to small;
[0025] The first candidate face images corresponding to the target feature similarity distribution graphs are used as second candidate face images.
[0026] In one or more embodiments, generating a final second feature vector based on the face image to be stored and each of the second candidate face images includes:
[0027] Extracting features of the face image to be stored in the database to obtain a second image feature vector;
[0028] Performing feature extraction on each of the second candidate facial images to obtain a third image feature vector corresponding to each of the second candidate facial images;
[0029] Calculating each of the third image feature vectors and the scale corresponding to each of the second candidate face images to obtain each fourth image feature vector;
[0030] Each of the fourth image feature vectors is operated on the second image feature vector to obtain a final second feature vector.
[0031] In one or more embodiments, the sub-data of the at least one modality includes at least one of text and images.
[0032] Accordingly, an embodiment of the present invention discloses a face attribute recognition device, the device comprising:
[0033] An acquisition module, used to acquire search data; the search data includes sub-data of at least one modality;
[0034] A feature extraction module, used to extract features from the sub-data of each modality, obtain sub-feature vectors corresponding to the sub-data of each modality, and generate a first feature vector corresponding to the search data based on each sub-feature vector;
[0035] A matching module, used for matching the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image;
[0036] The determination module is used to take the target face image corresponding to the target second feature vector as the retrieval result if there is a matching target second feature vector.
[0037] In one or more embodiments, further comprising:
[0038] The updating module is used to update the preset face image database before obtaining the search data to obtain the updated face image database.
[0039] In one or more embodiments, the update module includes:
[0040] The acquisition submodule is used to acquire the face images to be stored;
[0041] A first generation submodule is used to generate a first candidate face image of at least one scale based on the face image to be stored; the face image to be stored has corresponding face attribute description information;
[0042] a screening submodule, configured to determine at least one second candidate facial image from each of the first candidate facial images based on each of the first candidate facial images and the facial attribute description information;
[0043] A second generating submodule, configured to generate a final second feature vector based on the face image to be stored and each of the second candidate face images;
[0044] The storage submodule is used to store the face image to be stored and the second feature vector in a face image database.
[0045] In one or more embodiments, the first generating submodule is specifically used to:
[0046] Image extraction is performed on the face images to be stored using at least one preset scale to obtain at least one first candidate face image corresponding to each scale; the scale is used to characterize the proportion of the first candidate face image in the face images to be stored.
[0047] In one or more embodiments, the screening submodule is specifically used to:
[0048] Performing feature extraction on each of the first candidate facial images to obtain a first image feature vector corresponding to each of the first candidate facial images;
[0049] Performing feature extraction on the facial attribute description information to obtain a text feature vector;
[0050] Perform feature similarity matching on each of the first image feature vectors and the text feature vector to obtain a feature similarity distribution map corresponding to each of the first candidate face images;
[0051] Determine the top N target feature similarity distribution maps from each of the feature similarity distribution maps based on the order of variance from large to small;
[0052] The first candidate face images corresponding to the target feature similarity distribution graphs are used as second candidate face images.
[0053] In one or more embodiments, the second generating submodule is specifically used to:
[0054] Extracting features of the face image to be stored in the database to obtain a second image feature vector;
[0055] Performing feature extraction on each of the second candidate facial images to obtain a third image feature vector corresponding to each of the second candidate facial images;
[0056] Calculating each of the third image feature vectors and the scale corresponding to each of the second candidate face images to obtain each fourth image feature vector;
[0057] Each of the fourth image feature vectors is operated on the second image feature vector to obtain a final second feature vector.
[0058] In one or more embodiments, the sub-data of the at least one modality includes at least one of text and images.
[0059] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned facial attribute recognition method embodiment when executed by the processor.
[0060] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned facial attribute recognition method embodiment is implemented.
[0061] The embodiments of the present invention include the following advantages:
[0062] After obtaining the search data, feature extraction is performed on the sub-data of each modality to obtain the sub-feature vectors corresponding to the sub-data of each modality, and the first feature vector corresponding to the search data is generated based on each sub-feature vector; the first feature vector is matched with the updated face image database, and if there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as the search result. Through the above method, when performing face attribute recognition, without being restricted by a single image modality, face attribute type and complexity, any face attribute in an open world scene can be accurately recognized through a text description or a face image data. Compared with traditional face attribute recognition, the embodiments of the present invention eliminate the tedious process of manually labeling various attribute categories and training multiple face attribute classifiers, greatly improving the efficiency and accuracy of face attribute recognition, and avoiding the problem of difficulty in collecting special attribute data. It also has better robustness and adaptability, so it can cope with the needs of missing person inquiries in public safety scenarios, the advancement of human-computer interaction, personalized services, etc. At the same time, it uses the progress of computer vision and deep learning technology to solve the challenges of traditional recognition technology in accuracy, privacy protection and multimodal data fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 is a flowchart of the steps of a first embodiment of a method for face attribute recognition of the present invention;
[0064] Figure 2 is a flowchart of the steps of Embodiment 2 of a method for recognizing facial attributes of the present invention;
[0065] Figure 3 It is a schematic diagram of the process of updating the face image database of the present invention;
[0066] Figure 4 It is a structural block diagram of an embodiment of a face attribute recognition device of the present invention. DETAILED DESCRIPTION
[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0068] One of the core concepts of the embodiments of the present invention is that, when performing facial attribute recognition, without being restricted by a single image modality, facial attribute type, and complexity, any facial attribute in an open world scene can be accurately recognized through a text description or a facial image data. Compared with traditional facial attribute recognition, the embodiments of the present invention eliminate the tedious process of manually labeling various attribute categories and training multiple facial attribute classifiers, greatly improving the efficiency and accuracy of facial attribute recognition, and avoiding the problem of difficulty in collecting special attribute data. It also has better robustness and adaptability, so that it can cope with the needs of missing person inquiries in public safety scenes, the advancement of human-computer interaction, personalized services, etc., and at the same time, it uses the advancement of computer vision and deep learning technology to solve the challenges of traditional recognition technology in accuracy, privacy protection, and multimodal data fusion.
[0069] Reference Figure 1 , shows a step flow chart of a first embodiment of a facial attribute recognition method of the present invention, which can be applied to a system of multimodal facial attribute recognition based on visual content.
[0070] Among them, multimodal refers to information processing methods that involve or combine multiple different modes, media or types. In different contexts, multimodality can have different meanings and applications, such as multimodal interaction: refers to the ability of a computer system to understand and process information from multiple sensory inputs (such as vision, hearing, and touch); multimodal learning: in machine learning, multimodal learning involves combining information from different types of data sources (such as text, images, and audio) to improve the performance and generalization ability of the model.
[0071] Facial attributes refer to various features and attributes related to the human face, including but not limited to age, gender, race, expression, glasses, hats, masks, hairstyle, beard, makeup, and hundreds of other attributes.
[0072] Furthermore, open-set face attribute recognition means that during the model training and testing process, the system needs to identify unknown categories of face attributes and face data. Unknown categories of face attributes can be described in text form, so face attributes can be aligned based on text input to identify and retrieve the corresponding face attributes. Traditional face attribute recognition only needs to identify categories that have been seen in the training set and attributes that have been declared in advance. This situation is called closed-set detection. However, in the real world, we often encounter unknown new categories of data and samples that are different from the training data set. The goal of open-world face attribute recognition is to enable the model to distinguish between known attributes and unknown attributes, rather than just identifying between known attributes.
[0073] Based on this, the embodiment of the present invention may specifically include the following steps:
[0074] Step 101, obtaining retrieval data; the retrieval data includes sub-data of at least one modality.
[0075] Specifically, users can input data to be retrieved (recorded as "retrieval data"). For example, the retrieval data can be the text of "a middle-aged man, wearing a hat, glasses, looking forward", or the image of "a middle-aged man, wearing a hat, glasses, looking forward", or the text and image of "a middle-aged man, wearing a hat, glasses, looking forward".
[0076] That is to say, at least one modality of sub-data includes at least one of text and image. If the search data has only one modality of sub-data, then the sub-data is the search data. For example, continuing with the above example, if the search data is text or image, then the modality of the sub-data is one (text or image); if the search data is text and image, then the modality of the sub-data is two (text and image).
[0077] It should be noted that, in actual applications, the types of modalities that may be included in the search data may be set according to actual needs. For example, the search data may also include sub-data of audio, video and other modalities, which is not limited in the embodiments of the present invention.
[0078] Step 102 , extracting features from the sub-data of each modality to obtain sub-feature vectors corresponding to the sub-data of each modality, and generating a first feature vector corresponding to the search data based on each sub-feature vector.
[0079] After obtaining the search data input by the user, feature extraction can be performed on the sub-data of each modality in the search data to obtain the feature vector corresponding to the sub-data (denoted as "sub-feature vector"). For example, if the search data includes text and images, feature extraction is performed on the text to obtain the text feature vector. At the same time, feature extraction is performed on the image to obtain the image feature vector.
[0080] Then, each sub-feature vector is fused to obtain the feature vector corresponding to the search data (referred to as the "first feature vector"). For example, continuing with the above example, the text feature vector is fused with the image feature vector to obtain the final first feature vector. Among them, the method of fusing sub-feature vectors is applicable to the embodiment of the present invention, that is, the embodiment of the present invention does not limit the specific method of fusion.
[0081] Furthermore, when extracting features from text, BERT (Bidirectional Encoder Representations from Transformers) can be used, and when extracting features from images, ViT (Vision Transformer) can be used. Of course, other methods can also be used to extract features from text and images. In practical applications, the specific feature extraction method can be set according to actual needs, and the embodiments of the present invention do not limit this.
[0082] Step 103, matching the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image.
[0083] Specifically, an embodiment of the present invention can pre-deploy a face image database, which can store at least one face image and a feature vector (referred to as a "second feature vector") corresponding to each face image. In this way, after obtaining the first feature vector, the first feature vector can be matched with each second feature vector in the face image database.
[0084] It should be noted that before matching the first feature vector with the face image database, the face image database can be updated first, that is, the first feature vector can be matched with the updated face image database. In this way, it can be ensured that each match uses the latest face image database, thereby ensuring the accuracy of the retrieval.
[0085] Step 104: If there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as a retrieval result.
[0086] Specifically, if there is a second feature vector (referred to as the "target second feature vector") matching the first feature vector in the updated face image database, then the face image corresponding to the target second feature vector (referred to as the "target face image") can be used as the retrieval result.
[0087] In an embodiment of the present invention, after obtaining the search data, feature extraction is performed on the sub-data of each modality to obtain the sub-feature vector corresponding to the sub-data of each modality, and the first feature vector corresponding to the search data is generated based on each sub-feature vector; the first feature vector is matched with the updated face image database, and if there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as the search result. In the above manner, when performing face attribute recognition, without being restricted by a single image modality, face attribute type and complexity, any face attribute in an open world scene can be accurately recognized through a text description or a face image data. Compared with traditional face attribute recognition, the embodiments of the present invention eliminate the tedious process of manually labeling various attribute categories and training multiple face attribute classifiers, greatly improving the efficiency and accuracy of face attribute recognition, and avoiding the problem of difficulty in collecting special attribute data. It also has better robustness and adaptability, so it can cope with the needs of missing person inquiries in public safety scenarios, the advancement of human-computer interaction, personalized services, etc. At the same time, it uses the progress of computer vision and deep learning technology to solve the challenges of traditional recognition technology in accuracy, privacy protection and multimodal data fusion.
[0088] Reference Figure 2 , shows a flowchart of the steps of Embodiment 2 of a method for recognizing facial attributes of the present invention, which may specifically include the following steps:
[0089] Step 201, updating a preset face image database to obtain an updated face image database.
[0090] Before providing the search service, the preset face image database can be updated to obtain the updated face image database, and then the search service can be provided using the updated face image database. In this way, it can be ensured that each match uses the latest face image database, thereby ensuring the accuracy of the search. Figure 3 , shows a schematic diagram of the process of updating the face image database.
[0091] In the embodiment of the present invention, the updating of the preset face image database to obtain the updated face image database includes:
[0092] Get the face image to be stored;
[0093] Generate a first candidate face image of at least one scale based on the face image to be stored; the face image to be stored has corresponding face attribute description information;
[0094] Based on each of the first candidate facial images and the facial attribute description information, determining at least one second candidate facial image from each of the first candidate facial images;
[0095] generating a final second feature vector based on the face image to be stored and each of the second candidate face images;
[0096] The face image to be stored and the second feature vector are stored in a face image database.
[0097] Specifically, the user can input the facial image to be stored in the facial image database (referred to as "face image to be stored") and the facial attribute description information corresponding to the face image to be stored. After the system obtains the face image to be stored, it can decompose the face image to be stored and generate face images of multiple scales (referred to as "first candidate face image").
[0098] In the embodiment of the present invention, the step of generating a first candidate face image of at least one scale based on the face image to be stored includes:
[0099] Image extraction is performed on the face images to be stored using at least one preset scale to obtain at least one first candidate face image corresponding to each scale; the scale is used to characterize the proportion of the first candidate face image in the face images to be stored.
[0100] Specifically, when decomposing the face image to be stored, multiple preset scales can be used for image extraction to obtain face images of multiple scales, wherein each scale may correspond to multiple face images. In this way, different degrees of local information in the face image to be stored can be obtained through decomposition, so that more reliable data can be provided in the subsequent feature similarity calculation, thereby improving the robustness of the feature.
[0101] For example, three scales are preset, and each scale corresponds to four face images. Then, for the images to be stored, 12 first candidate face images of three scales can be generated.
[0102] The scale is used to represent the proportion of the first candidate face image in the face images to be stored. For example, if the scale is 0.9, it means that the first candidate face image accounts for 90% of the face images to be stored; if the scale is 0.2, it means that the first candidate face image accounts for 20% of the face images to be stored, and so on.
[0103] It should be noted that, in practical applications, the specific number of scales and the number of face images corresponding to each scale can be set according to actual needs, and the embodiments of the present invention do not limit this.
[0104] After obtaining each first candidate facial image, the facial attribute description information can be used to screen each first candidate facial image, discarding first candidate facial images with less relevance, thereby obtaining the remaining first candidate facial images (referred to as "second candidate facial images").
[0105] In the embodiment of the present invention, the step of determining at least one second candidate facial image from each of the first candidate facial images based on each of the first candidate facial images and the facial attribute description information includes:
[0106] Performing feature extraction on each of the first candidate facial images to obtain a first image feature vector corresponding to each of the first candidate facial images;
[0107] Performing feature extraction on the facial attribute description information to obtain a text feature vector;
[0108] Perform feature similarity matching on each of the first image feature vectors and the text feature vector to obtain a feature similarity distribution map corresponding to each of the first candidate face images;
[0109] Determine the top N target feature similarity distribution maps from each of the feature similarity distribution maps based on the order of variance from large to small;
[0110] The first candidate face images corresponding to the target feature similarity distribution graphs are used as second candidate face images.
[0111] Specifically, feature extraction can be performed on each first candidate facial image to obtain an image feature vector corresponding to each first candidate facial image (referred to as a "first image feature vector"). At the same time, feature extraction can be performed on the facial attribute description information to obtain a text feature vector.
[0112] Then, feature similarity matching is performed on each first image feature vector and the text feature vector, thereby obtaining a feature similarity distribution map corresponding to each first candidate facial image, that is, the number of feature similarity distribution maps is the same as the number of first candidate facial images, and they correspond one to one.
[0113] Since the larger the variance, the better it can distinguish the relationship between two feature vectors, the feature similarity distribution maps can be sorted in descending order according to the variance, and then the top N feature similarity distribution maps (referred to as "target feature similarity distribution maps") are selected, and the first candidate face image corresponding to each target feature similarity distribution map is used as the second candidate face image.
[0114] After determining each second candidate facial image, the final feature vector (referred to as “second feature vector”) corresponding to the facial image to be stored can be determined based on the feature vectors of each second candidate facial image and the feature vectors of the facial image to be stored.
[0115] In the embodiment of the present invention, the generating of the final second feature vector based on the face image to be stored and each of the second candidate face images includes:
[0116] Extracting features of the face image to be stored in the database to obtain a second image feature vector;
[0117] Performing feature extraction on each of the second candidate facial images to obtain a third image feature vector corresponding to each of the second candidate facial images;
[0118] Calculating each of the third image feature vectors and the scale corresponding to each of the second candidate face images to obtain each fourth image feature vector;
[0119] Each of the fourth image feature vectors is operated on the second image feature vector to obtain a final second feature vector.
[0120] Specifically, feature extraction can be performed on the facial images to be stored to obtain the image feature vector of the facial images to be stored (recorded as the "second image feature vector"). Feature extraction can also be performed on each second candidate facial image to obtain the image feature vector corresponding to each second candidate facial image (recorded as the "third image feature vector").
[0121] Since each second candidate human face image has a corresponding scale, each third image feature vector can be operated with the corresponding scale to obtain each image feature vector after operation (recorded as "fourth image feature vector").
[0122] Each fourth image feature vector is operated on the second image feature vector to obtain a final second feature vector corresponding to the face image to be stored.
[0123] Step 202, obtaining retrieval data; the retrieval data includes sub-data of at least one modality.
[0124] Specifically, the user can input search data. For example, the search data can be the text of "a middle-aged man, wearing a hat, glasses, looking forward", or the image of "a middle-aged man, wearing a hat, glasses, looking forward", or the text and image of "a middle-aged man, wearing a hat, glasses, looking forward".
[0125] That is to say, at least one modality of sub-data includes at least one of text and image. If the search data has only one modality of sub-data, then the sub-data is the search data. For example, continuing with the above example, if the search data is text or image, then the modality of the sub-data is one (text or image); if the search data is text and image, then the modality of the sub-data is two (text and image).
[0126] It should be noted that, in actual applications, the types of modalities that may be included in the search data may be set according to actual needs. For example, the search data may also include sub-data of audio, video and other modalities, which is not limited in the embodiments of the present invention.
[0127] Step 203 , extracting features from the sub-data of each modality to obtain sub-feature vectors corresponding to the sub-data of each modality, and generating a first feature vector corresponding to the search data based on each sub-feature vector.
[0128] After obtaining the search data input by the user, feature extraction can be performed on the sub-data of each modality in the search data to obtain the sub-feature vector corresponding to the sub-data. For example, if the search data includes text and images, feature extraction is performed on the text to obtain the text feature vector. At the same time, feature extraction is performed on the image to obtain the image feature vector.
[0129] Then, each sub-feature vector is fused to obtain the first feature vector corresponding to the search data. For example, continuing with the above example, the text feature vector is fused with the image feature vector to obtain the final first feature vector. Among them, the method of fusing sub-feature vectors is applicable to the embodiment of the present invention, that is, the embodiment of the present invention does not limit the specific method of fusion.
[0130] Furthermore, BERT can be used when extracting features from text, and ViT can be used when extracting features from images. Of course, other methods can also be used to extract features from text and images. In practical applications, the specific feature extraction method can be set according to actual needs, and the embodiments of the present invention do not limit this.
[0131] Step 204: Match the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image.
[0132] Specifically, an embodiment of the present invention can pre-deploy a face image database, which can store at least one face image and a second feature vector corresponding to each face image. In this way, after obtaining the first feature vector, the first feature vector can be matched with each second feature vector in the face image database.
[0133] It should be noted that before matching the first feature vector with the face image database, the face image database can be updated first, that is, the first feature vector can be matched with the updated face image database. In this way, it can be ensured that each match uses the latest face image database, thereby ensuring the accuracy of the retrieval.
[0134] Step 205: If there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as a retrieval result.
[0135] Specifically, if there is a target second feature vector matching the first feature vector in the updated face image database, then the target face image corresponding to the target second feature vector can be used as the retrieval result.
[0136] In the embodiment of the present invention, the face image can be decomposed into multiple scales, so as to complete the analysis of the face image at different scales, so that the small scale may pay more attention to details (such as facial expressions), while the large scale may capture more overall structural information (such as the outline of a hat).
[0137] Furthermore, image feature extraction and text feature extraction are used to extract facial image attribute features and semantic features respectively, and the image features are paired with the semantic features to establish the consistency of the image and semantic information in the implicit space.
[0138] Furthermore, by building a facial image database, real-time retrieval can be achieved at the level of retrieval of facial attributes of tens of millions of people, and multiple modal inputs can be supported at the same time, thus achieving more accurate retrieval.
[0139] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0140] Reference Figure 3, shows a structural block diagram of an embodiment of a face attribute recognition device of the present invention, which may specifically include the following modules:
[0141] The acquisition module 301 is used to acquire search data; the search data includes sub-data of at least one modality;
[0142] A feature extraction module 302 is used to extract features from the sub-data of each modality, obtain a sub-feature vector corresponding to the sub-data of each modality, and generate a first feature vector corresponding to the search data based on each sub-feature vector;
[0143] A matching module 303, configured to match the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image;
[0144] The determination module 304 is configured to use the target face image corresponding to the target second feature vector as a search result if there is a matching target second feature vector.
[0145] In an embodiment of the present invention, it also includes:
[0146] The updating module is used to update the preset face image database before obtaining the search data to obtain the updated face image database.
[0147] In an embodiment of the present invention, the update module includes:
[0148] The acquisition submodule is used to acquire the face images to be stored;
[0149] A first generating submodule is used to generate a first candidate face image of at least one scale based on the face image to be stored; the face image to be stored has corresponding face attribute description information;
[0150] a screening submodule, configured to determine at least one second candidate facial image from each of the first candidate facial images based on each of the first candidate facial images and the facial attribute description information;
[0151] A second generating submodule, configured to generate a final second feature vector based on the face image to be stored and each of the second candidate face images;
[0152] The storage submodule is used to store the face image to be stored and the second feature vector in a face image database.
[0153] In the embodiment of the present invention, the first generating submodule is specifically used for:
[0154] Image extraction is performed on the face images to be stored using at least one preset scale to obtain at least one first candidate face image corresponding to each scale; the scale is used to characterize the proportion of the first candidate face image in the face images to be stored.
[0155] In the embodiment of the present invention, the screening submodule is specifically used for:
[0156] Performing feature extraction on each of the first candidate facial images to obtain a first image feature vector corresponding to each of the first candidate facial images;
[0157] Performing feature extraction on the facial attribute description information to obtain a text feature vector;
[0158] Perform feature similarity matching on each of the first image feature vectors and the text feature vector to obtain a feature similarity distribution map corresponding to each of the first candidate face images;
[0159] Determine the top N target feature similarity distribution maps from each of the feature similarity distribution maps based on the order of variance from large to small;
[0160] The first candidate face images corresponding to the target feature similarity distribution graphs are used as second candidate face images.
[0161] In the embodiment of the present invention, the second generating submodule is specifically used for:
[0162] Extracting features of the face image to be stored in the database to obtain a second image feature vector;
[0163] Performing feature extraction on each of the second candidate facial images to obtain a third image feature vector corresponding to each of the second candidate facial images;
[0164] Calculating each of the third image feature vectors and the scale corresponding to each of the second candidate face images to obtain each fourth image feature vector;
[0165] Each of the fourth image feature vectors is operated on the second image feature vector to obtain a final second feature vector.
[0166] In an embodiment of the present invention, the sub-data of the at least one modality includes at least one of text and image.
[0167] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0168] An embodiment of the present invention further provides an electronic device, including:
[0169] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned face attribute recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0170] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned facial attribute recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0172] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0173] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0178] The above is a detailed introduction to a facial attribute recognition method and a facial attribute recognition device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for recognizing facial attributes, characterized in that: The method comprises: Acquire search data; the search data includes sub-data of at least one modality; Perform feature extraction on the sub-data of each modality to obtain a sub-feature vector corresponding to the sub-data of each modality, and generate a first feature vector corresponding to the search data based on each sub-feature vector; Matching the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image; If there is a matching target second feature vector, the target face image corresponding to the target second feature vector is used as the retrieval result.
2. The method for face attribute recognition according to claim 1, characterized in that: Before obtaining the retrieved data, the method further includes: The preset face image database is updated to obtain an updated face image database.
3. The method for face attribute recognition according to claim 2, characterized in that: The updating of the preset face image database to obtain an updated face image database includes: Get the face image to be stored; Generate a first candidate face image of at least one scale based on the face image to be stored; the face image to be stored has corresponding face attribute description information; Based on each of the first candidate facial images and the facial attribute description information, determining at least one second candidate facial image from each of the first candidate facial images; Generate a final second feature vector based on the face image to be stored and each of the second candidate face images; The face image to be stored and the second feature vector are stored in a face image database.
4. The method for face attribute recognition according to claim 3, characterized in that: The step of generating a first candidate face image of at least one scale based on the face image to be stored in the database includes: Image extraction is performed on the face images to be stored using at least one preset scale to obtain at least one first candidate face image corresponding to each scale; the scale is used to characterize the proportion of the first candidate face image in the face images to be stored.
5. The method for face attribute recognition according to claim 3, characterized in that: The step of determining at least one second candidate facial image from each of the first candidate facial images based on each of the first candidate facial images and the facial attribute description information comprises: Performing feature extraction on each of the first candidate facial images to obtain a first image feature vector corresponding to each of the first candidate facial images; Performing feature extraction on the facial attribute description information to obtain a text feature vector; Perform feature similarity matching on each of the first image feature vectors and the text feature vector to obtain a feature similarity distribution map corresponding to each of the first candidate face images; Determine the top N target feature similarity distribution maps from each of the feature similarity distribution maps based on the order of variance from large to small; The first candidate face images corresponding to the target feature similarity distribution graphs are used as second candidate face images.
6. The method for face attribute recognition according to claim 3, characterized in that: The generating a final second feature vector based on the face image to be stored and each of the second candidate face images includes: Extracting features of the face image to be stored in the database to obtain a second image feature vector; Performing feature extraction on each of the second candidate facial images to obtain a third image feature vector corresponding to each of the second candidate facial images; Calculating each of the third image feature vectors and the scale corresponding to each of the second candidate face images to obtain each fourth image feature vector; Each of the fourth image feature vectors is operated on the second image feature vector to obtain a final second feature vector.
7. The method for facial attribute recognition according to claim 1, characterized in that: The sub-data of the at least one modality includes at least one of text and image.
8. A facial attribute recognition device, characterized in that: The device comprises: An acquisition module, used to acquire search data; the search data includes sub-data of at least one modality; A feature extraction module, used to extract features from the sub-data of each modality, obtain sub-feature vectors corresponding to the sub-data of each modality, and generate a first feature vector corresponding to the search data based on each sub-feature vector; A matching module, used for matching the first feature vector with an updated face image database; the face image database includes at least one face image and a second feature vector corresponding to each face image; The determination module is used to take the target face image corresponding to the target second feature vector as the retrieval result if there is a matching target second feature vector.
9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the face attribute recognition method as claimed in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the facial attribute recognition method according to any one of claims 1 to 7 are implemented.