A face-based group recognition method and a training method for the group recognition model.

Abstract

This invention provides a training method for a face-based group recognition model. The group recognition model includes an encoder and a group prediction layer. The training method includes: A1. Obtaining a first training set and a second training set, which include multiple face image samples, wherein the face image samples in the second training set are labeled with labels indicating their respective groups; A2. Using the face image samples in the first training set, performing multiple rounds of self-supervised iterative training on the encoder to train the encoder to extract face features, wherein during self-supervised training, the encoder parameters are updated using the loss value determined by the decoder based on the face image reconstructed from the face features and the corresponding face image sample; A3. Using the second training set, performing multiple rounds of iterative training on the group recognition model to identify the group corresponding to the input face image sample, wherein the initial parameters of the encoder of the group recognition model are the parameters of the encoder trained in step A2.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to the technical field of group attribute recognition using facial images, and more specifically, to a face-based group recognition method and a training method for a group recognition model. Background Technology

[0002] Ethnic group inference has potential research value in multiple fields such as archaeology, anthropology, and sociology. Therefore, how to accurately infer an individual's ethnic group has become a technical problem of concern and research. Since an individual's ethnic information is highly correlated with their genetic makeup, facial features, and other biological characteristics, researchers in various fields have proposed various solutions from different perspectives. Among them, the most common are ethnic group inference methods based on DNA genetic information and ethnic attribute recognition methods based on facial features, which will be introduced below:

[0003] 1. The main technical approach of population inference methods based on DNA genetic information is as follows: extract DNA from individuals of unknown origin, obtain population-specific SNP genotypes based on publicly available or self-developed SNP locus systems, and import these genotypes into the population structure analysis software Structure for analysis to obtain population genetic principal component analysis. However, this technical approach has the following problems and shortcomings: DNA data is difficult to obtain in practice, related research equipment is expensive, prediction samples are time-consuming, and it is difficult to use on a large scale in the short term.

[0004] 2. The main technical means of face-based ethnic attribute recognition methods is to acquire the facial image of the corresponding individual, input it into a pre-trained model, and output the ethnic group corresponding to the individual. Because contemporary anthropological research suggests that facial features are influenced by a combination of external factors such as genes and geographical environment, ethnic groups living in the same environment for a long time tend to have similar facial features. In this case, researchers can infer the ethnic information contained in facial images and then regard the ethnic group as a facial attribute, i.e., an ethnic attribute. Therefore, various face ethnic attribute recognition methods based on traditional machine learning (such as patent publications: CN108319935A, CN109948519A, CN104143079A) or deep learning (such as patent publications: CN112069898A, CN103824054B, CN108596011A) have emerged. Some of these methods only perform single-attribute classification and recognition of ethnic groups, while others simultaneously identify attributes such as ethnicity, gender, expression, and identity. These will be introduced separately below:

[0005] 2.1 The main technical means of attribute recognition methods based on traditional machine learning include: acquiring face images and labels; extracting discriminative features from face images using traditional hand-crafted feature descriptors, such as Gabor filtering, Local Binary Pattern (LBP) features, Histogram of Oriented Gradients (HOG) features, Scale Invariant Feature Transform (SIFT), and dictionary learning; and performing single-attribute or multi-attribute recognition or classification of groups according to actual needs using discriminative methods such as k-nearest neighbors, Support Vector Machines (SVM), and Decision Trees. These techniques have achieved excellent performance in coarse-grained group recognition, but they struggle to distinguish faces between multiple subgroups that are closely related and whose facial features lack significant differentiation.

[0006] 2.2 The main technical means of deep learning-based attribute recognition methods include: These methods update the model parameters in the deep neural network based on the prediction loss between attribute prediction results on the training set data and the true attribute annotations to obtain a pre-trained classification model, which is then tested on a test set. Currently, existing deep learning-based attribute recognition methods are mainly strongly supervised, relying on massive amounts of face image data and explicit annotation information. Furthermore, since manually annotating different attributes is time-consuming and labor-intensive, it is difficult to obtain the large-scale labeled data required for these methods in practice. Once the problem of insufficient labeled data is encountered, the performance of these methods may be significantly reduced.

[0007] In summary, DNA-based population inference methods are difficult to widely adopt due to high costs and implementation difficulties. Among face-based population attribute recognition methods, traditional machine learning-based methods struggle to distinguish between closely related populations with indistinct facial features, resulting in low classification accuracy. Deep learning-based methods rely on massive amounts of face image data and explicit annotations; however, they fail to achieve good recognition accuracy when limited annotation data is available. Considering the scarcity of labeled data in existing publicly available face population attribute datasets and the lack of sufficient labeled data in most application scenarios, it is necessary to improve existing technologies to enhance the accuracy of fine-grained population classification using deep neural network models. Summary of the Invention

[0008] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a face-based group recognition method and a training method for a group recognition model.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] According to a first aspect of the present invention, a training method for a face-based group recognition model is provided. The group recognition model includes an encoder and a group prediction layer. The training method includes: A1, acquiring a first training set and a second training set comprising multiple face image samples, wherein the face image samples in the second training set are labeled with tags indicating their respective groups; A2, performing multiple rounds of self-supervised iterative training on the encoder using the face image samples in the first training set to train the encoder to extract face features, wherein during self-supervised training, the encoder parameters are updated using a loss value determined between a face image reconstructed by the decoder based on the face features and the corresponding face image sample; A3, performing multiple rounds of iterative training on the group recognition model using the second training set to identify the group corresponding to the input face image sample, wherein the initial parameters of the encoder of the group recognition model are the parameters of the encoder trained in step A2.

[0011] In some embodiments of the present invention, the ethnic group indicated in the label is at least one of the ethnic group attribute, sub-ethnic group attribute, and regional attribute to which the face image sample belongs.

[0012] In some embodiments of the present invention, in step A2, during training, an L1 loss function or an L2 loss function is used to determine the loss value based on the reconstructed face image and the corresponding face image sample.

[0013] In some embodiments of the present invention, in step A3, during training, the cross-entropy loss function is used to update the parameters of the encoder and the population prediction layer based on the loss value determined by the output population and the corresponding label.

[0014] In some embodiments of the present invention, the encoder is provided with an attention module based on an attention mechanism to extract attention-containing feature maps, and the encoder extracts facial features based on the attention-containing feature maps.

[0015] In some embodiments of the present invention, the encoder includes: an initial processing layer for performing convolution and pooling processing on the image input to the encoder to obtain an initial feature map; multiple stacked attention modules, each attention module for extracting an attention-containing feature map based on its input, wherein the size of the attention-containing feature maps output by each attention module is different at different stages, wherein the input of the first attention module is the initial feature map; an intermediate processing layer disposed between every two attention modules, each intermediate processing layer for downsampling the output of its preceding attention module and using the downsampled feature map as the input of its subsequent attention module; and a multi-scale fusion layer for performing size alignment and fusion processing on the attention-containing feature maps output by multiple attention modules to finally output facial features.

[0016] In some embodiments of the present invention, any one of the plurality of attention modules is a spatial attention module, a channel attention module, and a residual attention module.

[0017] In some embodiments of the present invention, the residual attention module includes: a backbone module, which performs convolution processing on the input of its attention module to obtain a backbone feature map; one or more attention mask modules, each attention mask module performing downsampling, upsampling, and residual processing on the input of its attention module to simultaneously extract global and local information to generate a mask, wherein the mask indicates the attention weight of each pixel in the feature map output by the backbone module; an attention intermediate processing module, which multiplies the mask obtained by each attention mask module with the backbone feature map pixel by pixel and then sums the result with the backbone feature map pixel by pixel to obtain an intermediate feature map containing attention; and an attention fusion processing module, which performs convolution processing on the intermediate feature map containing attention to obtain an attention-containing feature map.

[0018] In some embodiments of the present invention, the residual attention module includes a variety of attention mask modules. Different types of attention mask modules contain different numbers of one or more residual components. Each residual component performs downsampling, upsampling and residual processing on its input to simultaneously extract global and local information to generate intermediate or final masks. In this case, the last residual component in an attention mask module generates the final mask, and the other residual components generate intermediate masks.

[0019] According to a second aspect of the present invention, a face-based group recognition method is provided, comprising: inputting a face image to be recognized into a group recognition model trained according to the training method described in the first aspect, so as to identify the corresponding group.

[0020] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the methods of the first aspect and / or the second aspect by executing the executable instructions.

[0021] Compared with the prior art, the advantages of the present invention are as follows:

[0022] (1) The present invention constructs a self-supervised task, which allows the encoder to learn the ability to extract facial features from a large number of unlabeled face images through the training process. Then, the parameters of this encoder are transferred to a limited group attribute data (a small amount of labeled data, corresponding to the face image samples with labels in the group training set) for retraining (fine-tuning), thereby improving the accuracy of the model and making it more accurate than the existing technology (the scheme of directly using the labels in the labeled dataset for strongly supervised training).

[0023] (2) In order to further expand the application scenarios of the present invention and improve its performance in fine-grained group tasks, the present invention introduces an encoder based on the attention mechanism into the model to enhance facial features, thereby covering group attribute classification tasks of different granularities, including groups, subgroups and regions. This improvement of the present invention has achieved very advanced performance on face group datasets of multiple subgroups, thereby expanding the application scenarios in the current group classification tasks and efficiently completing the group attribute identification when the existing labeled data is extremely scarce, so as to further improve the accuracy of the model.

[0024] (3) This invention utilizes multiple stacked attention modules to collect features on feature maps of different dimensions and sizes, and then adjusts them to the same dimension for feature fusion. This can better collect global features and local detailed features, which helps to improve the accuracy of the model in fine-grained group recognition tasks. Attached Figure Description

[0025] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0026] Figure 1 This is a schematic diagram of the modules of a first illustrative group identification model according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the modules of a second illustrative group identification model according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a third illustrative group identification model according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of the fourth illustrative group identification model according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of the residual component according to an embodiment of the present invention;

[0031] Figure 6 This is a flowchart illustrating the training method of a face-based group recognition model according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0033] As mentioned in the background section, deep learning-based attribute recognition methods rely on massive amounts of face image data and explicit annotation information. When faced with limited labeled data, achieving good recognition accuracy becomes difficult. To address the challenge of limited labeled data and difficulty in achieving fine-grained group classification in existing face group attribute classification tasks, this invention designs a self-supervised learning-based face group attribute recognition scheme. For the very limited labeled data for group attributes, this invention constructs a self-supervised task. Through training, the encoder learns the ability to extract facial features from a large number of unlabeled face images. Then, the encoder's parameters are transferred to limited group attribute data (a small amount of labeled data, corresponding to labeled face image samples in the group training set) for retraining (fine-tuning), thereby improving the model's accuracy and achieving better prediction accuracy compared to existing techniques (schemes that directly utilize labels in labeled datasets for supervised training).

[0034] Furthermore, most existing methods primarily focus on coarse-grained group classification (e.g., distinguishing different groups), while neglecting finer-grained group attribute classification tasks (e.g., classifying faces among multiple subgroups). Therefore, they lack specific performance in fine-grained group classification tasks. To further expand the application scenarios of this invention and improve its performance in fine-grained group classification tasks, this invention introduces an attention-based encoder to enhance facial features, thereby covering group attribute classification tasks of different granularities, including groups, subgroups, and regions. This improvement has already achieved state-of-the-art performance on face group datasets of multiple subgroups, thus expanding the application scenarios in current group classification tasks and efficiently completing group attribute identification when existing labeled data is extremely scarce, further improving the model's accuracy.

[0035] To better illustrate the technical solution of this invention, the following is a detailed description of four aspects: model structure, training samples, training process, and application scenarios.

[0036] I. Model Structure

[0037] According to one embodiment of the present invention, the group identification model employs an attention-free structure. See also... Figure 1The population identification model includes an encoder 100 and a population prediction layer 200, wherein the encoder 100 can employ an existing deep neural network model. According to one embodiment of the present invention, the encoder 100 and the population prediction layer 200 of the population identification model can, for example, employ a VGG model (VGG16 model or VGG19 model) or a ResNet model. When those skilled in the art use existing models as the population identification model of the present invention, they can select the appropriate structure as the encoder 100 or the population prediction layer 200 according to the needs of the implementer. For example, taking the VGG16 model as an example, the structure before the third max pooling layer can be used as the encoder 100, and the structure after the third max pooling layer can be used as the population prediction layer 200. Alternatively, the structure of an existing deep neural network model can be adjusted to serve as the population identification model of the present invention, for example, by adding or removing some convolutional layers from the original VGG19 model or ResNet model.

[0038] Fine-grained group attribute classification tasks require models to pay more attention to details (such as local facial features, texture, shape, etc.), which conventional deep neural network models do not adequately address. Therefore, further, according to an embodiment of the present invention, the group recognition model employs an attention-based structure, see [link to relevant documentation]. Figure 2 An attention mechanism is incorporated into the encoder of the ethnic group recognition model. According to one embodiment of the invention, the encoder of the ethnic group recognition model includes one or more attention modules based on the attention mechanism to extract attention-containing feature maps, and extracts facial features based on the attention-containing feature maps. Any one of these attention modules can be a spatial attention module, a channel attention module, or a residual attention module. These three attention modules can adopt existing structures in the art, which will not be elaborated upon in this invention. The ethnic group prediction layer of the ethnic group recognition model includes a fully connected layer and a Softmax layer. The facial features output by the encoder are sequentially processed by the fully connected layer and the Softmax layer to output the corresponding ethnic group. The following beneficial technical effects can be achieved by adopting this approach in the present invention: Classical group attribute recognition methods mostly take the predicted group of a face as the research objective, without fully considering the feature discrimination and mining between subgroups. Therefore, the coarse-grained group classification model learned by these methods may not be suitable for distinguishing facial differences between multiple subgroups, resulting in a decrease in the performance of such fine-grained group attribute recognition. Therefore, the present invention leverages the advantage of attention mechanism in mining detailed features by adding an attention module to the encoder, which allows the group recognition model to pay more attention to some detailed features that are beneficial to fine-grained group classification, thereby further improving the prediction accuracy of the model.

[0039] To further improve model performance, one could consider incorporating an attention mechanism into the model while focusing on both global and local features. See also Figure 3 According to an embodiment of the present invention, a population recognition model includes an encoder 100 and a population prediction layer 200. The encoder 100 includes: an initial processing layer 110, used to perform convolution and pooling processing on the image input to the encoder to obtain an initial feature map; multiple stacked attention modules 120, each attention module 120 used to extract attention-containing feature maps from the input based on an attention mechanism, the sizes of the attention-containing feature maps output by each attention module 120 being different, wherein the input of the first attention module is the initial feature map; an intermediate processing layer 130 disposed between every two attention modules 120, each intermediate processing layer 130 used to downsample the output of its preceding attention module and use the downsampled feature map as the input of its subsequent attention module; and a multi-scale fusion layer 140, used to perform size alignment and fusion processing on the attention-containing feature maps output by the multiple attention modules 120 to output facial features. For ease of understanding, in Figure 3 The diagram illustrates the size changes of the attention module and the multi-scale fusion layer during feature processing. Let A be the size (length and width directions, hereinafter the same) of the output of the first attention module. Then, the size of the output of the second attention module is... The size of the output of the third attention module is In the multi-scale fusion layer 140, the output of the third attention module is upsampled, and its size becomes... The output of the second and third attention modules (also in size) Element-wise addition (adding feature values ​​at corresponding (pixel) positions), output size is The feature map is then upsampled once more, its size changed to A, and then element-wise added to the output of the first attention module (also of size A). The output is a face feature of size A, which is then input into the population prediction layer 200 to output the predicted population. It should be understood that this is only illustrative, and other implementations are possible, such as: the output of the second attention module having a size of... The size of the output of the third attention module is For example, in the multi-scale fusion layer 140, the output of the third attention module can be upsampled twice to obtain a feature map of size A (let's call it feature map A). Figure 1 The output of the second attention module is then upsampled once to obtain a feature map of size A (let's call it feature map A). Figure 2 ); then features Figure 1 ,feature Figure 2The feature map output by the first attention module (of size A) is element-wise summed to output a face feature of size A. This invention achieves at least the following beneficial technical effects using this method: By utilizing multiple stacked attention modules to collect features from feature maps of different dimensions, and then adjusting them to the same dimension for superposition, it can better collect global features and local detailed features, which helps improve the model's accuracy in fine-grained group recognition tasks.

[0040] In addition to utilizing existing attention modules as described above, improved attention modules can be used to further enhance model accuracy. For example, while existing spatial attention and channel attention modules have observed subtle differences in local features, it is still difficult to distinguish between faces from multiple subgroups based solely on local features. Therefore, in such scenarios, improved attention modules can be used to combine local and global features simultaneously to help differentiate these fine-grained face needs. According to an embodiment of the present invention, see... Figure 4 An improved residual attention module is employed, comprising: a backbone module 121, which performs convolution processing on the input of its respective attention module to obtain a backbone feature map; one or more attention mask modules 122, each attention mask module 122 performing downsampling, upsampling, and residual processing on the input of its respective attention module to simultaneously extract global and local features to generate a mask, wherein the mask indicates the attention weight of each pixel in the feature map output by the backbone module; an attention intermediate processing module 123, which performs element-wise multiplication (element-wise production) and element-wise addition (element-wise addition) on the mask obtained by each attention mask module 122 with the backbone feature map to obtain an intermediate feature map containing attention; and an attention fusion processing module 124, which performs convolution processing on the intermediate feature map containing attention to obtain an attention-containing feature map. According to an embodiment of the present invention, the residual attention module includes multiple attention mask modules 122, each attention mask module including one or more residual components. See also Figure 5According to one embodiment of the present invention, each residual component includes, in sequence: a max pooling layer 122a, a residual unit 122b, a max pooling layer 122c, a residual unit 122d, a residual unit 122e, an upsampling layer 122f, a residual unit 122g, an upsampling layer 122h, a 1x1 convolutional layer 122i, a 1x1 convolutional layer 122j, and a sigmoid function layer 122k (used to convert the input into feature values ​​between 0 and 1 using the sigmoid function). The output of residual unit 122b is further processed by a residual unit 122l. The result of the processing by residual unit 122l is used as the input of residual unit 122g after residual calculation with the output of upsampling layer 122f. According to one embodiment of the present invention, different types of attention mask modules contain different numbers of one or more residual components. If the required classification granularity of the population is finer, the more types of attention mask modules are in the residual attention module. For example, if the task is to distinguish populations, only an attention mask module including one residual component is used; if the task is to distinguish sub-populations, an attention mask module including one residual component and an attention mask module including two stacked residual components are used; if the task is to distinguish regions, an attention mask module including one residual component, an attention mask including two stacked residual components, and an attention mask module including three stacked residual components are used. Each residual component downsamples, upsamples, and performs residual processing on its input to simultaneously extract global and local information to generate an intermediate or final mask. The last residual component in an attention mask module generates the final mask, and the other residual components generate intermediate masks. This invention achieves at least the following beneficial technical effects by using this method: The invention collects global features by downsampling the feature map in the residual attention module, and then upsamples the global features and adds them to the local features transmitted from the residual connection (residual unit 122l). This allows for consideration of both global and local features beneficial for group classification at the mask location, thereby further improving the performance of the group recognition model; combined with… Figure 3 The structure shown in this invention considers the full fusion of global and local features at two levels: one is the fusion of global and local features between modules, and the other is the fusion of global and local features extracted at the mask. This enables the expansion of group attribute recognition tasks from coarse-grained to fine-grained, promoting a comprehensive improvement in the model's group prediction performance. It should be understood that... Figure 4The residual attention module shown is merely illustrative. Its attention mask modules and residual components are modular and plug-in-like. A residual attention module can contain only one type of attention mask module or multiple attention mask modules, as specified by the user. For example, an interface for building a group recognition model can be provided, allowing users to customize the number of attention mask modules and the number of residual components within each module before training the model. This enables users to build group recognition models adapted to their specific group recognition scenarios, meeting the needs of group recognition scenarios at different granularities.

[0041] According to an example of the invention, see again Figure 4 Here is an illustrative structure of a group recognition model with a residual attention mechanism (the length × width of the input face image is assumed to be 224 × 224):

[0042] First, let me explain the meaning of the following data formats as a whole:

[0043] Convolutional kernel configuration: length × width, number of convolutional kernels, stride; the number of convolutional kernels corresponds to the number of output channels;

[0044] Output dimensions: length × width;

[0045] Residual Units: The structure is the same as that of residual units in existing ResNet models. All residual units appearing in this example adopt this structure.

[0046] Encoder 100:

[0047] The initial processing layer 110 includes a convolutional layer (convolutional kernel configuration: 7×7, 64, 2; output size: 112×112) and a pooling layer (max pooling, output size: 56×56).

[0048] First attention module 120 (output size: 56×56):

[0049] Backbone module 121 (output size: 56×56) includes one or more residual units.

[0050] The first attention mask module 122 (output size: 56×56) includes: 1 Figure 5 The residual components shown;

[0051] The second attention mask module 122 (output size: 56×56) includes: 2 Figure 5 The residual components shown;

[0052] The attention intermediate processing module 123 is used to multiply the mask obtained by the first attention mask module 122 with the backbone feature map output by the backbone module 121 in an element-wise manner and then sum the results, outputting the summed result (output size: 56×56, denoted as feature map x). It is also used to multiply the mask obtained by the second attention mask module 122 with the backbone feature map output by the backbone module 121 in an element-wise manner and then sum the results, outputting the summed result (output size: 56×56, denoted as feature map y). Finally, feature map x, feature map y, and the backbone feature map are summed to obtain an intermediate feature map containing attention (output size: 56×56).

[0053] The first intermediate processing layer 130 (output size: 28×28) includes a residual unit, in which the stride of a certain convolution is set to 2 so that the output size is halved;

[0054] Second attention module 120 (output size: 28×28):

[0055] The structure is the same as the first attention module, except that the size of the output is changed to 28×28;

[0056] The second intermediate processing layer 130 (output size: 14×14) includes a residual unit, in which the stride of a certain convolution is set to 2 so that the output size is halved;

[0057] The third attention module 120 (output size: 14×14):

[0058] The structure is the same as the first attention module, except that the size of the output is changed to 14×14.

[0059] The multi-scale fusion layer 140 (output size: 56×56, number of channels: 256) first aligns the output of the third attention module with the output of the second attention module through deconvolution (becoming 28×28; if the number of channels is inconsistent, the number of channels can also be aligned) and fuses them (output size: 28×28) to obtain an intermediate fusion result; then, it aligns the intermediate fusion result with the output of the first attention module through deconvolution (becoming 56×56; if the number of channels is inconsistent, the number of channels can also be aligned) and fuses it to output facial features (output size: 56×56).

[0060] Population prediction layer 200 includes a fully connected network (such as a two-layer fully connected network) and a Softmax layer.

[0061] II. Training Samples

[0062] In order for the ethnic identification model to perform ethnic identification, a dataset needs to be prepared for training.

[0063] Since this invention requires two levels of training—one level is self-supervised training, and the other level is supervised training—two datasets need to be prepared.

[0064] The first dataset is a face dataset for self-supervised training, which includes multiple face image samples. Therefore, it is only necessary to collect frontal face images of different people, adjust each face image to a predetermined input size (e.g., 224×224) as a face image sample to form the face dataset. This face dataset can be collected independently or an existing face dataset (e.g., the VGGFace2 face dataset) can be directly used. The first training set is this face dataset or a subset of it. Extracting a subset from a dataset according to a predetermined ratio as a training set is a common practice among those skilled in the art, and will not be elaborated here.

[0065] The second dataset is a face group dataset for supervised training, which includes face image samples and a label indicating the group to which each face image sample belongs. The group indicated in the label is at least one of the subgroup attribute, group attribute, and region attribute to which the face image sample belongs. For example, according to one embodiment of the present invention, if the group indicated in the label is the subgroup attribute to which the face image sample belongs, the purpose of the corresponding group recognition model is to identify the subgroup to which a person belongs based on the face image. For example, the label space is: first subgroup, second subgroup, third subgroup, fourth subgroup, fifth subgroup, sixth subgroup, or a combination thereof, with the aim of distinguishing people from these subgroups. According to another embodiment of the present invention, if the group indicated in the label is the group attribute to which the face image sample belongs, the purpose of the corresponding group recognition model is to identify the group to which a person belongs based on the face image. For example, the label space is: first group, second group, third group, fourth group, or a combination thereof. According to another embodiment of the present invention, the ethnic group indicated in the label is the regional attribute to which the face image sample belongs. The purpose of the corresponding ethnic group recognition model is to identify the region to which a person belongs based on the face image. Here, the region can be based on a province, city, or district, or other regional classifications set by the implementer. The second training set is the face ethnic group dataset or a subset of the face ethnic group dataset.

[0066] III. Training Process

[0067] According to one embodiment of the present invention, see Figure 6A training method for a face-based group recognition model includes: A1, acquiring a first training set and a second training set comprising multiple face image samples, wherein the face image samples in the second training set are labeled with tags indicating their respective group; A2, performing multiple rounds of self-supervised iterative training on the encoder using the face image samples in the first training set to train the encoder to extract face features, wherein during self-supervised training, the encoder parameters are updated using the loss value determined by the decoder based on the face image reconstructed from the face features and the corresponding face image sample; A3, performing multiple rounds of iterative training on the group recognition model using the second training set to identify the group corresponding to the input face image sample, wherein the initial parameters of the encoder of the group recognition model are the parameters of the encoder trained in step A2. Preferably, during supervised training in step A2, the decoder used includes multiple deconvolution layers to reconstruct a face image of the same size as the face image sample based on the face features. In step A2, during training, an L1 loss function or an L2 loss function is used to determine the loss value based on the reconstructed face image and the corresponding face image sample. Therefore, without setting additional labels, the encoder can first perform unsupervised learning based on face image samples, thereby learning some knowledge useful for extracting face features in advance. Preferably, in step A3, during training, a cross-entropy loss function (e.g., binary cross-entropy or multi-class cross-entropy, depending on the number of classifications in the specific task) is used to update the parameters of the encoder and the group prediction layer based on the loss value determined by the output group and the corresponding label. Preferably, the ratio of the number of face image samples in the first training set to the number of face image samples in the second training set is between 100 and 50,000. For example, the first training set uses millions of face image samples, while the second training set can use only a few hundred labeled face image samples.The present invention achieves at least the following beneficial technical effects using this method: The training method of the present invention, based on a self-supervised auxiliary task for face reconstruction, utilizes the error loss between the reconstructed face and the input face as a supervision signal to first train the encoder in the group recognition model. This allows full utilization of information from large-scale unlabeled face data without adding any supervision signal, thereby efficiently training the encoder to extract face features. Then, the group recognition model is trained as a whole based on labeled data (face image samples and corresponding labels). Specifically, the prior knowledge of faces contained in the large-scale unlabeled face data obtained directly from the self-supervised auxiliary task of face reconstruction corresponds to the parameters of the encoder trained in step A2. By fine-tuning the model with only a small amount of attribute-labeled data (corresponding to step A3), the task of ethnic group attribute recognition can be effectively achieved (i.e., the training method of the present invention enables the ethnic group recognition model to achieve good accuracy even when the number of face image samples in the second training set is much smaller than the number of face image samples in the first training set). Compared with the prior art, the training method of the present invention can improve the accuracy of the ethnic group recognition model trained with a small amount of labeled data, achieving the goal of obtaining good ethnic group recognition performance with only a small amount of labeled data; moreover, since this training method only requires a small amount of labeled data, it can be quickly extended to different ethnic group recognition tasks, greatly reducing the workload of implementers.

[0068] IV. Application Scenarios

[0069] According to one embodiment of the present invention, a face-based ethnic group recognition method is provided, comprising: inputting a face image to be recognized into an ethnic group recognition model trained according to the training method of the face-based ethnic group recognition model of the foregoing embodiment, so as to identify the corresponding ethnic group. Of course, it can also be applied to the field of archaeology (such as determining the ethnic group to which a face image reconstructed based on a skull belongs) or other fields, and the present invention does not impose any limitations thereon.

[0070] To verify the effectiveness of the invention, the inventors conducted experiments, and the relevant experimental data are as follows:

[0071] (1) Experimental setup, as shown in Table 1:

[0072] Table 1

[0073]

[0074] (2) Experimental results

[0075] Following the experimental setup described above, and after training using the model structure of the last example in the aforementioned model structure embodiments and the training method of this invention, the results shown in Table 2 were obtained. Table 2 shows the prediction performance of the strongly supervised method proposed by Wang et al. and this invention on face datasets representing multiple subgroups, predicting faces of the first, second, and third subgroups, respectively. Compared to the baseline, this invention achieved better performance in all cases. Furthermore, when only 1% and 2% of the labeled data were trained, this invention achieved improvements of 4.02% and 6.38% respectively compared to the strongly supervised method represented by the baseline. This demonstrates that our proposed self-supervised method can effectively address the problem of insufficient labeled data for each subgroup and makes a significant contribution to performance improvement. Moreover, when 100% of the data was trained, the method of this invention even surpassed the baseline by 9.15%, because we introduced an improved attention mechanism in the self-supervised model to jointly enhance the mining of local and global facial features. Therefore, this invention can effectively utilize prior facial features in large-scale unlabeled data and improve the performance of fine-grained subgroup classification.

[0076] Table 2. Comparison of prediction accuracy between the present invention and methods that outperformed it on multiple subgroup datasets.

[0077]

[0078] For the method used by Wang Yu et al., please refer to the following literature:

[0079] Wang Y, Feng Y, Liao H, et al. Do they all look the same? decipheringchinese, japanese and koreans by fine-grained deep learning[C] / / 2018 IEEEConference on Multimedia Information Processing and Retrieval (MIPR). IEEE,2018: 39-44.

[0080] In addition, the inventors conducted experiments based on a self-made face dataset. Table 3 shows the prediction performance of the strongly supervised method proposed by Wang Yu et al. and the present invention on the self-made face dataset for predicting faces of seven subgroups. Since the size of face images varies for each class in this dataset, 9000 images were randomly selected from each class and recombined into a classification-balanced subset. Therefore, one part of Table 3 (left side) reports the model's performance on the entire dataset, and the other part (right side) reports the model's performance on the balanced subset. As shown in Table 3, when only 0.2% and 0.5% of the entire dataset were used for training, the present invention achieved higher accuracy than the baseline, improving by 1.46% and 2.98%, respectively. Under similar conditions, the improvement of the present invention on the balanced subset was even higher from the beginning and continued to expand to over 3.11%. Furthermore, the present invention comprehensively outperformed the baseline and achieved the best results with different proportions of training samples in the two datasets. This can be attributed to the fact that the training method of the present invention successfully alleviates the data hunger problem that is difficult for strongly supervised methods to handle, while also being adept at joint mining of local and global features. It's worth noting that strongly supervised methods like the baseline, which use ResNet50 as the backbone, suffer from overfitting when the training set is extremely sparse. Specifically, when trained with 321 samples, these methods perform poorly on balanced subsets, fluctuating between 16% and 17%, approximately 14.28%. This means that strongly supervised methods have very limited effectiveness on such fine-grained population datasets. Undeniably, the above results demonstrate that the method presented in this invention has certain advantages over current state-of-the-art methods, even though faces from seven different countries in the self-made face dataset are more difficult to distinguish.

[0081] Table 3. Comparison of prediction accuracy of the present invention and superior methods on a self-made dataset.

[0082]

[0083] In summary, experiments have shown that this invention enables the population identification model to achieve high population prediction accuracy under conditions of extremely limited labeled data. Furthermore, this invention does not require a large amount of labeled data, which can significantly reduce the labor and time costs of implementation. It also has strong scalability (due to the small amount of labeled data, it can be quickly applied to population attribute identification tasks of various granularities).

[0084] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0085] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0086] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0087] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a face-based group recognition model, characterized in that, The group identification model includes an encoder and a group prediction layer, and the training method includes: A1. Obtain a first training set and a second training set, which include multiple face image samples, wherein the face image samples in the second training set are labeled with labels indicating their respective groups; A2. The encoder is trained to extract facial features by performing multiple rounds of self-supervised iterative training using face image samples from the first training set. During self-supervised training, the encoder parameters are updated using the loss value determined by the decoder based on the face image reconstructed from the facial features and the corresponding face image samples. A3. Use the second training set to perform multiple rounds of iterative training on the group recognition model to identify the group corresponding to the input face image sample. The initial parameters of the encoder of the group recognition model are the parameters of the encoder trained in step A2. The encoder mentioned above includes: The initial processing layer is used to perform convolution and pooling on the image input from the encoder to obtain the initial feature map; Multiple attention modules are stacked, each of which is used to extract an attention-containing feature map based on its input. The size of the attention-containing feature maps output by each attention module is different from each other, and the input of the first attention module is the initial feature map. An intermediate processing layer is placed between every two attention modules. Each intermediate processing layer downsamples the output of the preceding attention module and uses the downsampled feature map as the input to the following attention module; and The multi-scale fusion layer is used to perform size alignment and fusion processing on the attention-containing feature maps output by multiple attention modules to output facial features.

2. The training method according to claim 1, characterized in that, The ethnic group indicated in the label is at least one of the ethnic group attribute, sub-ethnic group attribute, and regional attribute to which the face image sample belongs.

3. The training method according to claim 2, characterized in that, In step A2, during training, the L1 loss function or L2 loss function is used to determine the loss value based on the reconstructed face image and the corresponding face image sample.

4. The training method according to claim 3, characterized in that, In step A3, during training, the cross-entropy loss function is used to update the parameters of the encoder and the population prediction layer based on the loss value determined by the output population and the corresponding label.

5. The training method according to any one of claims 1-4, characterized in that, The encoder includes an attention module that extracts attention-containing feature maps based on an attention mechanism, and the encoder extracts facial features based on the attention-containing feature maps.

6. The training method according to claim 1, characterized in that, Any one of the multiple attention modules can be a spatial attention module, a channel attention module, or a residual attention module.

7. The training method according to claim 6, characterized in that, The residual attention module includes: The backbone module performs convolutional processing on the input of its respective attention module to obtain the backbone feature map. One or more attention mask modules, each attention mask module is used to downsample, upsample and residual process the input of its attention module to extract global and local information at the same time to generate a mask, the mask indicating the attention weight of each pixel of the feature map output by the backbone module; The attention intermediate processing module is used to multiply the mask obtained by each attention mask module with the backbone feature map by pixels, and then sum the result with the backbone feature map by pixels to obtain an intermediate feature map containing attention. The attention fusion processing module is used to perform convolution processing on the intermediate feature maps containing attention to obtain the feature maps containing attention.

8. The training method according to claim 7, characterized in that, The residual attention module includes various attention mask modules. Different types of attention mask modules contain different numbers of one or more residual components. Each residual component performs downsampling, upsampling and residual processing on its input to extract global and local information simultaneously to generate intermediate or final masks. In this case, the last residual component in an attention mask module generates the final mask, while the other residual components generate intermediate masks.

9. A face-based group recognition method, characterized in that, include: The face image to be identified is input into the group recognition model trained by the training method according to any one of claims 1-8 to identify the corresponding group.

10. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8 and 9.

11. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 8 and 9 by executing the executable instructions.

Images