Semi-supervised heterogeneous face recognition method based on modal adversarial disturbance

Abstract

The application discloses a semi-supervised heterogeneous face recognition method based on modal confrontation disturbance, and comprises the following steps: acquiring a heterogeneous face dataset comprising labeled data and unlabeled data; constructing a semi-supervised learning heterogeneous face recognition model comprising a multi-collaborative face representation module and a face modal confrontation disturbance module; wherein the multi-collaborative face representation module extracts stronger modal invariant discriminant features by utilizing the relevant interaction of diverse and modal invariant discriminant knowledge at different depths in the network; the face modal confrontation disturbance module is used for generating modal disturbance images, mapping data to a modal-agnostic subspace, and relieving data modal differences; and the semi-supervised learning heterogeneous face recognition model is trained based on the heterogeneous face dataset, so as to realize heterogeneous face recognition by using the trained model. The method can achieve good recognition accuracy by using only a small amount of labeled data, reduces the demand for labeled data and the algorithm complexity, and has better recognition performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of face recognition, and particularly relates to a semi-supervised heterogeneous face recognition method based on modal adversarial disturbance. BACKGROUND

[0002] With the development of computer technology, face recognition technology has been widely used in finance, justice, army, public security, border inspection, government, aerospace, power, factory, education, medical treatment and many enterprises and institutions.

[0003] Traditional face recognition technology is mainly based on visible light image face recognition. In recent years, multi-source face recognition technology based on active near-infrared image has developed rapidly. The huge domain gap between visible light and near-infrared images reduces the performance of face recognition models. At the same time, the problem of personal privacy protection and the expensive label consumption make it difficult to collect a sufficient number of heterogeneous faces from different source domains. At present, there are few related researches on semi-supervised heterogeneous face recognition (HFR) technology for less labeled data.

[0004] At present, the commonly used heterogeneous face recognition methods mainly include three types: face synthesis based method, public subspace mapping based method and feature descriptor based method.

[0005] The goal of the method based on face synthesis is to convert the face image into another domain through the method of synthesis, so as to directly apply the synthesized image in the face recognition algorithm. For example, the document “He R, Li Y, Wu X, et al. Coupled adversarial learning for semi-supervised heterogeneous face recognition [J]. Pattern Recognition, 2021, 110: 107618.” synthesizes new heterogeneous face images by adversarial learning on images and features. The document “Yang Z, Liang J, Fu C, et al. Heterogeneous Face Recognition via Face Synthesis With Identity-Attribute Disentanglement [J]. IEEE Transactions on Information Forensics and Security, 2022, 17: 1344-1358.” designs a face synthesis method with identity attribute disentanglement to generate face images with random combinations of identity and attribute. The document “Luo M, Wu H, Huang H, et al. Memory-modulated transformer network for heterogeneous face recognition [J]. IEEE Transactions on Information Forensics and Security, 2022, 17: 2095-2109.” designs a conversion network with memory capability to perform unsupervised example-guided image conversion.

[0006] The method based on common subspace mapping hopes to map the data of different modalities into the same common subspace, so that the data of the two modalities have the minimum difference in the common subspace, and then directly perform face matching in this subspace. For example, the document “Sharma A, Jacobs D W. Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch [C] / / CVPR 2011. IEEE, 2011: 593-600.” proposes to use the partial least squares algorithm to learn the mapping relationship of different modal images. The document “Liu D, Gao X, Wang N, et al. Coupled attribute learning for heterogeneous face recognition [J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(11): 4699-4712.” uses a deep learning model integrated with additional semantic information to learn the mapping relationship between modalities.

[0007] The feature descriptor-based method is to find local feature descriptors to represent face images, and to identify according to these descriptors. For example, the document “Lei Z, Yi D, Li S Z. Discriminant image filter learning for face recognition with local binary pattern like representation [C] / / 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012: 2512-2517.” proposes a discriminant image filter learning method for matching different modal images. The document “Liu D, Li J, Wang N, et al. Composite components-based face sketch recognition [J]. Neurocomputing, 2018, 302: 46-54.” integrates the discriminant information of different face components to improve the performance of the model.

[0008] However, the method based on face synthesis cannot generate samples closer to real images when the training samples are less, and the recognition performance is subject to the quality of the synthesized face. The public subspace mapping method always causes partial information loss when mapping images to a common subspace, thereby reducing the recognition performance. The feature descriptor-based method always has high computational complexity and takes a long time. In addition, the existing methods almost do not consider the current personal privacy protection problem. The privacy problem makes it difficult to collect a sufficient number of heterogeneous faces from different source domains for training, and it takes a lot of manpower to obtain face labels, which is costly. SUMMARY

[0009] In order to solve the above problems existing in the prior art, the present application provides a semi-supervised heterogeneous face recognition method based on modal adversarial disturbance. The technical problems to be solved by the present application are realized by the following technical solutions:

[0010] A semi-supervised heterogeneous face recognition method based on modal adversarial disturbance, comprising:

[0011] Step 1: obtaining a heterogeneous face dataset including labeled data and unlabeled data;

[0012] Step 2: constructing a semi-supervised learning heterogeneous face recognition model including a multi-collaborative face representation module and a face modal adversarial disturbance module;

[0013] The multi-collaborative face representation module utilizes the relevant interaction of multi-modal invariant discriminant knowledge at different depths in the network to extract stronger modal invariant discriminant features;

[0014] The face modal adversarial disturbance module is used to generate modal disturbance images, map data to a modal-agnostic subspace, and alleviate the data modal difference;

[0015] Step 3: training the semi-supervised learning heterogeneous face recognition model based on the heterogeneous face dataset, so as to realize heterogeneous face recognition by using the trained model.

[0016] The present application has the following advantages:

[0017] The semi-supervised heterogeneous face recognition method based on modal adversarial disturbance provided by the application can improve the robustness of the heterogeneous face recognition model by designing a multi-collaborative face representation module, mining face discrimination information of different deep networks, extracting stronger modal invariant discrimination knowledge, and reducing the requirement for labeled data.

[0018] The application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of the semi-supervised heterogeneous face recognition method based on modal adversarial disturbance provided by the embodiment of the application is shown in the figure.

[0020] Figure 2 A general framework diagram of the model method based on the multi-collaborative representation model and the modal adversarial disturbance provided by the embodiment of the application is shown in the figure.

[0021] Figure 3 A framework diagram of the face modal adversarial disturbance module provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0022] The application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0023] Embodiment one

[0024] Please refer to Figure 1 , Figure 1 A flowchart of the semi-supervised heterogeneous face recognition method based on modal adversarial disturbance provided by the embodiment of the application is shown in the figure, and the method comprises:

[0025] Step 1: Obtain a heterogeneous face dataset comprising labeled data and unlabeled data.

[0026] Specifically, the proposed heterogeneous face recognition method is described by taking near-infrared and visible light face images as examples. It can be easily understood that the method can be easily extended to other cross-modal face scenarios.

[0027] Given a representative heterogeneous face dataset, denoted as:

[0028]

[0029] where, denotes the i-th training example, M is the number of heterogeneous face images, N and V represent two different facial modalities, y i is the real identity label.

[0030] The goal of semi-supervised heterogeneous face recognition is to train a robust face representation model on the dataset D, and use as few real labeled images as possible. Therefore, the key problem is to design a face representation with stronger generalization: where, F θ is a face feature extractor, w, h and d are the width, height and channel size of the feature map.

[0031] Based on this, the embodiment designs a multi-collaborative face representation module and a face modal adversarial disturbance module, and forms a semi-supervised learning heterogeneous face recognition model through training, please refer to the following steps.

[0032] Step 2: Construct a semi-supervised learning heterogeneous face recognition model including a multi-collaborative face representation module and a face modal adversarial disturbance module.

[0033] Please refer to Figure 2 , Figure 2 The overall framework diagram of the model method based on the multi-collaborative representation model and the modal adversarial disturbance provided by the embodiment of the application. The framework is composed of two key modules: the multi-collaborative face representation module uses the relevant interaction of the diversity and modal invariant discriminant knowledge in the network of different depths to extract stronger modal invariant discriminant features; the face modal adversarial disturbance module is used to generate modal disturbance samples, and map the data to a modal-agnostic subspace to alleviate the data modal difference.

[0034] Specifically, since different depth face representation networks focus on different modal discriminant information, which is invariant in images of different modalities. Based on this, the embodiment constructs a multi-collaborative representation model, which includes several blocks, and each block contains a feature extraction layer; and in addition to the last block, the feature extraction layer in the remaining blocks is also connected to a bottleneck layer; wherein,

[0035] The feature extraction layer constitutes the backbone network of the entire model to perform feature extraction of different depths on the input data;

[0036] The bottleneck layer is used to learn the feature representation of different depths in the model training stage.

[0037] Optionally, as an implementation manner, as Figure 1The network in the literature 'Wu X, He R, Sun Z, et al. A light CNN for deep face representation with noisy labels [J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884-2896.' is used as the backbone network in this embodiment, and four blocks are divided according to the structure thereof. Each block is followed by a bottleneck layer composed of a nonlinear mapping layer and a linear layer, which is composed of 3 convolutional layers and 2 linear layers, for learning representations at different depths.

[0038] It can be understood that a classifier is further included in each block, for image classification based on the previously extracted features.

[0039] For each input of a heterogeneous face After the designed block, the output can be denoted as And the prediction probability of the i-th classifier can be denoted as The expression is as follows:

[0040]

[0041] Wherein, the parameter K is a temperature hyperparameter for adjusting the smoothness of the probability distribution, and K represents the number of face identity classes.

[0042] Further, data augmentation is an important strategy to alleviate the overfitting problem in the classification model. Common enhancement methods such as converting brightness, contrast and clarity can only obtain limited improvement in the heterogeneous face recognition task. This is because there is indeed a large modal gap between near-infrared and visible light data distribution, and existing enhancement methods cannot effectively increase the diversity of training data. Considering the special properties of cross-modal images, the face modal recognition enhancement is proposed in this embodiment, which maps unlabelled heterogeneous faces to latent modal agnostic domain through generative adversarial perturbation. The goal of the modal adversarial perturbation module is to learn a small perturbation to generate examples And This can mislead the modal classifier and at the same time preserve the identity discriminative information.

[0043] Please refer to Figure 3 , Figure 3 The framework diagram of the face modal adversarial perturbation module provided by the embodiment of the present application, the face modal adversarial perturbation module comprises a modal agnostic perturbation generator G, a face modal classifier D m And a cross-modal face label classifier D id ; wherein,

[0044] The modal agnostic perturbation generator is used to generate adversarial perturbations on input data, resulting in modal perturbed images;

[0045] The face modal classifier is used to learn to distinguish different modalities in the model training phase.

[0046] The cross-modal face label classifier is used for face classification and recognition based on original images and modal perturbed images.

[0047] Specifically, given an unlabeled input heterogeneous face and whose identity label is a pseudo label The face modal label is q i , where q i ∈{0,1}. The generator G aims to generate appropriate perturbations to construct perturbed samples and where the parameters of the face modal classifier D m are pre-trained using large-scale near-infrared and visible light images, and the modal distinguishability is strong, with a correct rate of more than 98%. The parameters of the cross-modal face identity recognition model D id are frozen in the training phase.

[0048] The modal classifier D m aims to accurately distinguish the modal q i of the input image, which can help to hide different modal information in the perturbed image and effectively reduce the modal gap. While the perturbation generator G aims to mislead the above modal classifier D m .

[0049] Step 3: Train the semi-supervised learning heterogeneous face recognition model based on the heterogeneous face dataset, so as to realize heterogeneous face recognition using the trained model.

[0050] In this embodiment, in order to train the face representation model with unlabeled images, the present application uses pseudo labels to simulate distinguishable identity information. Specifically, these unlabeled data are directly input into the pre-trained model to obtain pseudo labels. When the predicted identity probability of the input face is higher than the confidence threshold τ, the obtained prediction value is converted into a pseudo label for the following training process.

[0051] Specifically, the process of training the semi-supervised learning heterogeneous face recognition model mainly includes the following steps:

[0052] 31) Pre-train the semi-supervised learning heterogeneous face recognition model.

[0053] Considering that the heterogeneous face dataset contains labeled data (i.e. labeled data) and unlabeled data (i.e. unlabeled data), the real identity label is represented as y i , and the pseudo label is represented as The cross entropy between the real label or the pseudo label and the predicted probability is designed as follows:

[0054]

[0055] where C represents the number of classifiers, N l represents the number of labeled data, N u represents the number of unlabeled data, and K represents the number of face identity classes.

[0056] In addition, the relative entropy (KL divergence) constraint of each block classifier prediction is also used to constrain the different depths of This can help them interact with each other to mine stronger discriminative information. The loss function of the designed multi-collaboration representation is represented as:

[0057]

[0058] where N represents the number of all data participating in training, represents the predicted probability of the face recognition model at different depths, represents the final predicted probability of the face recognition model, and the overall training loss can be written as:

[0059] L total =·L cls +(1-)· col ;

[0060] where α represents the first hyperparameter, which can balance the above multi-layer classification loss L cls and the multi-collaboration loss L col .

[0061] In the present embodiment, the process of pre-training the semi-supervised learning heterogeneous face recognition model in step 31) can be summarized as follows:

[0062] 31a) Initialize the network model using the LightCNN network weight trained on the public face dataset (such as CASIA-WebFace and MS-Celeb-1M datasets).

[0063] 31b) Before each training iteration, use the current network model to make predictions on the unlabeled data. If the predicted confidence is greater than a given threshold, then the current unlabeled data is labeled with a pseudo label and used for the next training iteration process; otherwise, the current unlabeled data will not participate in the next training.

[0064] 31c) inputting the labeled data and the image data marked with pseudo labels into the network to train the network and update the network weights, wherein a loss function in the training process uses the overall training loss L total .

[0065] Optionally, in the embodiment, the process of pre-training the network model is iterated 50 times.

[0066] 32) Based on step 31), the face modal adversarial perturbation module is trained.

[0067] For the enhanced image (i.e. the modal perturbation image), the embodiment sets the same pseudo labels as the source unlabeled face, and these pseudo labels are iteratively updated throughout the training process.

[0068] In the embodiment, the modal classifier D m aims to accurately distinguish the modal q i of the input image, which can help to hide the different modal information in the perturbation image and effectively reduce the modal gap, and the face modal classifier loss can be expressed as:

[0069]

[0070] wherein q i,m represents the face modal label. Here, the cross-entropy loss is used to train the modal classifier parameters. Conversely, the perturbation generator G aims to mislead the above face modal classifier D m . Here, the modal classifier loss is maximized, and the early stopping strategy is used to maintain generalization. It is worth noting that the modal perturbation can directly use existing gradient-based attack algorithms such as PGD, etc. Therefore, the modal adversarial perturbation loss can be formulated as:

[0071]

[0072] In addition, even in the modal recognition domain, the constructed perturbation face image should contain robust discriminative identity information. Therefore, the loss function that maintains the cross-modal identity discriminative information is designed as follows:

[0073]

[0074] Therefore, the overall loss of the unknown modal perturbation generator is:

[0075]

[0076] wherein β represents a second hyperparameter, used to balance the identity discrimination ability and the performance of the modal perturbation generator. In the training process of the modal adversarial perturbation module, the perturbation generator G and the modal classifier D mAlternatively, individual optimization. Modal classifier D m Maximizing discriminability to predict accurate modal labels, the perturbation generator G aims to minimize discriminability. When the perturbation generator G is optimized, it is used to generate new samples in the modal-agnostic domain, which can be directly input into the proposed multi-collaborative face representation model to improve performance.

[0077] Then the process of step 32) training the face modal adversarial perturbation module can be summarized as follows:

[0078] 32a) Use the training set in step 31) to iteratively train the face modal classifier in the face modal adversarial perturbation module, and freeze the parameters of the remaining layers of the model during the training process, and only update the parameters of the face modal classifier.

[0079] It should be noted that the face modal classifier includes a 1-layer linear layer added after Table 1 MFM_fc1, with an output size of 2, and a binary classification for identifying modalities. After 5 iterations of the face modal classifier, the modal classification accuracy can reach 98%.

[0080] 32b) Freeze all parameters in the network model except the face modal classifier, input unlabeled data, maximize the modal classifier loss and minimize the identity discrimination loss using PGD attack, and add the perturbation generated by the PGD attack to the unlabeled data to generate modal perturbation images. During this training process, the loss function uses L G .

[0081] 32c) Input the modal perturbation image into the network for adversarial training, so that the model tries to distinguish the modal information of the modal perturbation image and update the parameters of the face modal classifier; During this training process, the loss function uses

[0082] 32d) Generate modal perturbation images and train modal classifiers alternately, and save the last generated modal perturbation image as the training data for the next stage.

[0083] 33) Based on the modal perturbation images generated by the trained face modal adversarial perturbation module, retrain the semi-supervised learning heterogeneous face recognition model.

[0084] Specifically, replace the unlabeled data with the modal perturbation images obtained in step 32), reduce the learning rate, and repeat the training process in step 31) for 100 iterations for fine-tuning training, and finally obtain the trained semi-supervised learning heterogeneous face recognition model.

[0085] Based on the trained heterogeneous face recognition model, heterogeneous face recognition can be realized.

[0086] The semi-supervised heterogeneous face recognition method based on modal adversarial disturbance provided by the application can improve the robustness of the heterogeneous face recognition model by designing a multi-collaborative face representation module, mining face discrimination information of different deep networks, extracting stronger modal invariant discrimination knowledge, and reducing the requirement for labeled data; the modal adversarial disturbance algorithm is proposed to map the image to a modal-agnostic subspace, thereby effectively alleviating the modal difference of the heterogeneous face recognition, so that the method can achieve good recognition accuracy with only a small amount of labeled data, reduce the requirement for labeled data and the algorithm complexity, and is not restricted by the quality of the synthesized face and the labeled data, and has better recognition performance.

[0087] Embodiment two

[0088] The beneficial effects of the application are verified and described below through simulation experiments.

[0089] Specifically, several public heterogeneous face databases are selected in the embodiment, and the semi-supervised heterogeneous face recognition based on the multi-collaborative representation model and the modal adversarial disturbance (MAMCO-HFR for short) is evaluated on the databases.

[0090] The selected databases include the CASIA NIR-VIS 2.0 dataset, the Tufts Face dataset and the LAMP-HQ dataset.

[0091] The CASIA NIR-VIS 2.0 dataset is a widely used near-infrared-visible light face dataset. The dataset contains 725 subjects. Each subject has 1-22 visible light and 5-50 near-infrared face images. The face images contain different poses and expressions to increase the difficulty of recognition.

[0092] The Tufts Face database is a large-scale public heterogeneous face database. It contains various types of face images, including thermal images, near-infrared images, software-generated sketch images, photo images and 3D images. For ease of comparison, the embodiment selects the thermal infrared-visible light dataset, which contains 1583 pairs of thermal infrared images from 112 subjects.

[0093] The LAMP-HQ dataset is a high-quality near-infrared-visible light face dataset, containing 56788 near-infrared and 16828 visible light images of 573 subjects. The images include 2 races, 3 expressions, 5 scenes, 3 angles and a wide range of age distribution.

[0094] Firstly, the present embodiment performs a 10-fold cross-validation experiment on the CASIA NIR-VIS 2.0 dataset, and provides the accuracy of Rank-1, the verification rate (VR) @ FAR = 1% and VR @ FAR = 0.1%. At the same time, the results of using several existing methods are compared, and the existing methods include ADFL (Adversarial Discriminative Feature Learning), W-CNN (Wasserstein Convolutional Network), PACH (Pose Aligned Cross-Domain Network), DVR (Discriminative Variant Representation), DVG (Dual Variant Generation Representation), OMDRA (Orthogonal Modal Decomposition and Representation Alignment), CAL (Coupled Adversarial Learning) and FSIAD (Face Synthesis by Identity-Attribute Decomposition), and the results are shown in Table 1.

[0095] Table 1 Comparison of Rank-1 accuracy and verification rate of FAR = 1% and 0.1% on 10-fold cross-validation experiment on CASIA NIR-VIS 2.0 dataset

[0096]

[0097]

[0098] Table 1 shows the comparison results of Rank-1 accuracy and verification rate of FAR = 1% and 0.1% on 10-fold cross-validation experiment on CASIA NIR-VIS 2.0 dataset. As can be seen from Table 1, due to the strong discriminability of the deep learning model, the existing methods all achieve an accuracy of more than 98% in Rank-1. In the case of identification through a synthesis strategy, FSIAD achieves an accuracy of 99.9% in Rank-1, which is almost the best performance that can be achieved. In the case of only 60% of the labeled face images, the MAMCO-HFR proposed in the present embodiment can achieve an accuracy of 99.9% in Rank-1. In the evaluation index of VR @ FAR = 0.1%, the method of the present embodiment is even better than the current best method (FSIAD) by 0.72%.

[0099] Secondly, Table 2 shows the comparison results of Rank-1 accuracy and verification rate of FAR = 1%, 0.1% and 0.01% on 10-fold cross-validation experiment on LAMP-HQ dataset.

[0100] Table 2 Comparison of Rank-1 accuracy and verification rate of FAR = 1%, 0.1% and 0.01% on 10-fold cross-validation experiment on LAMP-HQ dataset

[0101]

[0102]

[0103] From Table 2, the evaluation metrics of Rank-1 accuracy, VR@FAR=1%, VR@FAR=0.1%, and VR@FAR=0.01% are provided by the present application in the LAMP-HQ dataset. The HFR methods ADFL and PACH based on face synthesis are affected by the quality of the synthesized faces and only achieve about 95% in Rank-1 accuracy.

[0104] The DVG takes advantage of the unconditioned face synthesis to generate more training data to improve the recognition performance. The FSIAD further utilizes the disentangled representation to enrich the attribute diversity of the synthesized faces and achieves 98.7% in Rank-1 accuracy. With only 60% of the labeled training data, the method proposed by the present application improves the Rank-1 accuracy to 99.2%, VR@FAR=0.1% accuracy to 99.2%, and VR@FAR=0.01% accuracy to 99.2%, which is 5.3% higher than the state-of-the-art FSIAD method. This is because the method proposed by the present application can effectively utilize the mutual information between different deep networks, and the simple modal adversarial perturbation can help to reduce the modal gap.

[0105] Finally, Table 3 gives the comparison results of the area under the curve (AUC), equal error rate (EER), and verification rate at FAR=1% on the Tufts Face database dataset.

[0106] Table 3 Comparison of area under the curve (AUC), equal error rate (EER), and verification rate at FAR=1% on the Tufts Face database dataset

[0107]

[0108]

[0109] Specifically, the recognition results of Pixel2Pixel, DAL-GAN, PIM, CRN+CL, CANVFS, DA-GAN, Multi-AP GAN, CUT-ATC and MMTN are compared in this experiment. Here, the area under the curve (AUC), equal error rate (EER) and VR@FAR=1% are provided as evaluation indicators of the Tufts face dataset. Due to the large modal difference and limited training data in the thermal infrared data provided in the Tufts Face dataset, this is a more challenging problem. Although MMTN alleviates the problem of data scarcity by synthesizing a large number of heterogeneous face images, the image quality of the synthesized images will affect the recognition performance.

[0110] Unlike previous comprehensive recognition strategies, the MAMCO-HFR proposed in the present application uses a simple modal adversarial perturbation strategy to increase the scale of training data, and uses a multi-collaborative feature representation module to learn stronger modal invariant discrimination knowledge. In the case of only 60% labeled data, the method of the present application still achieves significant improvement in recognition performance. As can be seen from Table 3, the AUC accuracy of the method is improved from 92.6% to 97.2%. The method is superior to the state-of-the-art MMTN method by at least 32% in VR@FAR=1%. The experimental results prove that even in the case of limited training data and complex heterogeneous image scenarios, the method proposed in the present application can achieve satisfactory recognition performance.

[0111] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be considered as falling within the protection scope of the present application.

Claims

1. A semi-supervised heterogeneous face recognition method based on modality adversarial perturbation, characterized by: include: Step 1: Obtain a heterogeneous face dataset including labeled data and unlabeled data; Step 2: Build a semi-supervised learning heterogeneous face recognition model including a multi-collaborative face representation module and a face modality adversarial perturbation module; The multi-collaborative face representation module utilizes the interaction of diverse and modality-invariant discriminative knowledge at different depths in the network to extract stronger modality-invariant discriminative features. The face modality adversarial perturbation module is used to generate a modality perturbation image and map the data into a modality-agnostic subspace to alleviate the modality difference of the data; The multi-collaborative face representation module includes several blocks, each of which contains a feature extraction layer. Except for the last block, the feature extraction layers in the remaining blocks are followed by a bottleneck layer. The bottleneck layer includes a nonlinear mapping layer and a linear layer, specifically three convolutional layers and two linear layers. The feature extraction layer constitutes the backbone network of the entire model and is used to extract features of different depths from the input data. The bottleneck layer is used to learn feature representations of different depths during the model training phase. The face modality adversarial perturbation module includes a modality-agnostic perturbation generator, a face modality classifier, and a cross-modal face label classifier; wherein the modality-agnostic perturbation generator is used to generate adversarial perturbations on input data to obtain modality-perturbed images; the face modality classifier is used to learn to distinguish different modalities during the model training phase; the cross-modal face label classifier is used to perform face classification and recognition based on the original image and the modality-perturbed image; Step 3: Training the semi-supervised learning heterogeneous face recognition model based on the heterogeneous face dataset, so as to realize heterogeneous face recognition using the trained model.

2. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 1 is characterized in that In step 3, the semi-supervised learning heterogeneous face recognition model is trained based on the heterogeneous face dataset, including: 31) Pre-training the semi-supervised learning heterogeneous face recognition model; 32) Based on step 31), training the face modality adversarial perturbation module; 33) Retraining the semi-supervised learning heterogeneous face recognition model based on the modal perturbation image generated by the trained face modal adversarial perturbation module.

3. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 2 is characterized in that Step 31) includes: 31a) Initialize the network model using the LightCNN network weights trained on a public face dataset; 31b) Before each training iteration, the current network model is used to make predictions on the unlabeled data. If the confidence of the prediction is greater than a given threshold, the current unlabeled data is labeled with a pseudo-label and used in the next training iteration; otherwise, the current unlabeled data will not participate in the next training iteration; 31c) Input the labeled data and the image data marked with pseudo labels into the network for training and update the network weights.

4. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 3 is characterized in that In step 31c), the loss function used in the network training process is as follows: ； in, represents the overall training loss, Represents the first hyperparameter, which is used to balance the multi-layer classification loss and the multi-collaboration loss. and They represent multi-layer classification loss and multi-collaboration loss respectively, expressed as: ； ； in, C represents the number of classifiers, represents the number of labeled data, represents the amount of unlabeled data, K represents the number of face identity classes, N Indicates the number of all data involved in training, represents the predicted probability of the face recognition model, represents the true label, represents a pseudo label, Represents the prediction probability of different depth face recognition models, Represents the final predicted probability of the face recognition model.

5. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 2 is characterized in that: Step 32) includes: 32a) iteratively training the face modality classifier in the face modality adversarial perturbation module using the training set in step 31), freezing the parameters of the remaining layers of the model during training and only updating the parameters of the face modality classifier; 32b) Freeze all parameters in the network model except the face modality classifier, input unlabeled data, use a gradient-based attack algorithm to maximize the modality classifier loss and minimize the identity discrimination loss, and add the perturbation generated by the gradient-based attack algorithm to the unlabeled data to generate a modality perturbation image; 32c) inputting the modal perturbation image into the network for adversarial training, so that the model attempts to distinguish the modal information of the modal perturbation image and update the parameters of the face modality classifier; 32d) Generating modal perturbation images and training modal classifiers are performed alternately, and the modal perturbation images generated last time are saved as training data for the next stage.

6. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 5 is characterized in that In step 32b), the loss function used in the network training process is as follows: ； in, represents the overall loss of the agnostic modal perturbation generator, The second represents a hyperparameter used to balance the modality adversarial perturbation loss and the cross-modal identity discrimination loss. and They represent the modality adversarial perturbation loss and cross-modal identity discrimination loss, respectively, and are expressed as: ； ； in, represents the amount of unlabeled data, represents the face modality label, represents the face modality classifier, represents the modal perturbation image, K represents the number of face identity classes, represents a pseudo label, Represents a cross-modal face label classifier.

7. The semi-supervised heterogeneous face recognition method based on modality adversarial perturbation according to claim 5, characterized in that In step 32c), the loss function used for adversarial training of the network is as follows: ； in, represents the amount of unlabeled data, represents the face modality label, represents the face modality classifier, represents the modal perturbation image.

Images