A multimodal face recognition and prototype reconstruction method based on VAE-GAN framework

Abstract

The present invention relates to the field of data processing technology, and in particular to a multimodal face recognition and prototype reconstruction method based on a VAE-GAN framework, which aims to solve the face recognition problem in a multimodal situation, wherein the registered face comes from domain A, and the target face to be retrieved may cover domain A, domain B, and domain C. At the same time, the method has a multimodal face prototype reconstruction function, that is, it accepts "contaminated" face data (from domain A, domain B, and domain C), and can reconstruct its face prototype image in a specific domain (domain A, domain B, or domain C) according to demand. The present invention can improve the security and scope of application of the face recognition system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal face recognition and prototype reconstruction method based on a VAE-GAN framework. Background Art

[0002] Face recognition systems have been widely used in real life, however, they still face two major challenges:

[0003] ① The registered face photo and the target face photo to be retrieved may belong to different modalities (domains). For example, the registered face photo may be a high-resolution image taken by a visible light camera, while the face photo to be retrieved may come from a different domain, such as the infrared domain (near-infrared image taken by an infrared camera) or the sketch domain. Since there are huge differences between images of different modalities (domains), this cross-modal face recognition situation will significantly increase the recognition and matching difficulty of the system, resulting in a decrease in recognition accuracy. At present, although there are some advanced solutions that can handle dual-modal heterogeneous face recognition (that is, the system can simultaneously process face images of two different modalities, such as visible light and infrared light), it cannot handle more complex multimodal situations (that is, the system needs to simultaneously process face images of multiple different modalities, such as visible light, infrared light, sketches and other different data types).

[0004] ② In some scenarios, the target face photo to be retrieved may have changes in expression (smiling, opening mouth, etc.), posture (side face, looking down, etc.), lighting (brightness, contrast, etc.), and occlusion (wearing glasses, masks, etc.). These factors will affect the system's extraction and matching of facial features, thereby reducing the accuracy of face recognition.

[0005] Variational Autoencoder (VAE) is a deep learning model that consists of two main parts: encoder and decoder. The encoder maps the input data to a probability distribution in the latent space, while the decoder decodes the samples in the latent space into the original data. VAE aims to learn the probability distribution of data and achieve effective data generation and sample interpolation. Representation decoupling is an important function of the VAE model. By mapping the input data to the latent space and forcing the representation in the latent space to obey the prior distribution, such as the Gaussian distribution, the structure of the latent space is designed so that each dimension corresponds to a feature of the data, and the loss function is designed to constrain the representation learned by the model to be continuous and distributed. This enables the model to learn the important features and structure of the data and decouple it into independent parts, improving the interpretability and generalization ability of the representation. In multimodal face recognition, the representation decoupling function of VAE can eliminate the differences between different modalities, retain face ID information, and improve the accuracy and robustness of multimodal face recognition.

[0006] Generative Adversarial Network (GAN) is a deep learning model that consists of two networks: the Generator and the Discriminator. The Generator is responsible for generating data samples, while the Discriminator is responsible for distinguishing between generated samples and real data. The core idea of ​​GAN is to use adversarial training to allow the Generator to continuously generate realistic samples, while allowing the Discriminator to continuously improve its ability to distinguish between real and generated samples. During the training process, the Generator and the Discriminator compete with each other and achieve a balance by minimizing the optimization objective. In the end, the samples generated by the Generator will become closer and closer to the real data distribution, while the Discriminator will not be able to accurately distinguish between generated samples and real samples. The core idea of ​​GAN is to enable the Generator to learn the structure of the target data distribution through adversarial learning between the Generator and the Discriminator, thereby achieving the generation of high-quality target distribution data.

[0007] Based on this, the present invention proposes a method for multimodal face recognition and prototype reconstruction, which utilizes the representation decoupling capability of the VAE-GAN framework to map facial images of multiple different modalities into a shared latent space and decouple identity features that are independent of the modality. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal face recognition and prototype reconstruction method based on the VAE-GAN framework, which aims to solve the face recognition problem in a multimodal situation, where the registered face comes from domain A, and the target face to be retrieved may cover domain A, domain B, and domain C. At the same time, the method has a multimodal face prototype reconstruction function, that is, it accepts "contaminated" face data (from domain A, domain B, and domain C), and can reconstruct its face prototype image in a specific domain (domain A, domain B, or domain C) according to demand.

[0009] A multimodal face recognition and prototype reconstruction method based on a VAE-GAN framework includes the following steps:

[0010] S1. Model training set preparation:

[0011] Assume that a training set contains N identity class samples from domains A, B, and C; each image x in the training set is labeled in Indicates the domain information label of x, Indicates that the image x belongs to domain A, that is, x~P dataA , similarly A value of 2 or 3 indicates belonging to domain B or domain C, respectively; represents the identity label of x, and Indicates whether there is facial change in image x, Indicates facial changes. Indicates that image x has no facial changes and is a normal face image; then, according to The value of is used to select face samples without facial changes from the training data set to build a prototype library. The pictures in the prototype library are denoted as x rp , in Represents x rp The domain information tag, Represents x rp identity tag, Represents x rp Whether there are facial changes;

[0012] S2. Model structure:

[0013] S2.1, Generator G: G consists of an encoder G enc and a decoder G dec Composition: G enc Responsible for decoupling the modality-independent identity feature information P from the input x x ; G dec Receive P x and control mask v, concatenate them as input, and then generate output x that matches the control information in v v ;

[0014] S2.2. Discriminator D and feature vector domain classifier D c : D includes three sub-discriminators D gan , D dom and D var ;D gan It is a sub-discriminator related to GAN, used to distinguish between real prototypes and generated prototypes; D dom It is a domain-information-dependent sub-discriminator used to infer the domain to which the input image belongs; D var Used to detect whether the input image has changed; feature vector domain classifier D c For further removal of G enc The decoupled identity feature information P x The possible domain information in D c Receive P x As input, and outputs the domain it predicts;

[0015] S3. Model training:

[0016] S3.1. Stage 1: Extracting identity feature information P that is independent of modality x ; First, input the face image x into G enc Decouple the identity feature information P from it x =G enc (x), and P xInput to the feature vector domain classifier D c In, D c The goal is to predict the domain code, i.e. So D c The objective function is as follows:

[0017]

[0018] Where pre is D c The predicted domain code, is the domain code of the face image x, P is the probability distribution over the domain pre;

[0019] In contrast, encoder G enc The purpose is to confuse D c Correctly predict domain codes;

[0020] S3.2, stage 2: reconstruction learning to maintain the identity feature information of the generated prototype; decoder G dec Learn to mask v with original attributes by decoding p P x To reconstruct the input face image x, that is where v p Like the control mask v, it is also used to control the decoder G dec The generated image content, v p Value by and Determine; reconstruction learning maintains the identity feature information of the generated prototype in that it shares the decoder G with the subsequent prototype learning dec , which will enable the feature information in the input image x to be maintained in the subsequent prototype learning process of generating prototypes; the objective function of reconstruction learning is as follows:

[0021]

[0022] S3.3, Stage 3: Prototype learning generates a face prototype for a specified domain; in this stage, the generator G and the discriminator D are trained adversarially to improve each other's performance through mutual competition; this enables the generator to reconstruct a multimodal face prototype; for the discriminator D = [D gan , D dom , D var ], which has three training objectives:

[0023] 1) The fake prototype generated by the given generator G and the real prototype x in the prototype library rp ,D gan Expect to is classified as a false prototype, and x rp Classified as a true prototype, so D ganThe objective function is as follows:

[0024]

[0025] 2) Given a real prototype image x of a domain rp , D dom Expect to correctly predict its domain information label Therefore D dom The objective function is as follows:

[0026]

[0027] where p dom D dom The predicted domain code;

[0028] 3) Given an input image x, D var Expect to correctly predict its Label, to determine whether the face image has changed; therefore, D var The objective function is as follows:

[0029]

[0030] So the final objective function of the discriminator D is as follows:

[0031]

[0032] in Corresponding to the above formulas (3)(4)(5) respectively;

[0033] For the generator G, it also has three training objectives:

[0034] 1) Cheating D gan Make it generate a fake prototype Classified as a true prototype;

[0035] 2) Make D dom Will The domain label of is predicted to be the same as the domain label specified by the control mask v;

[0036] 3) Make D var Will It is judged as a normal face picture without any changes; therefore, the objective function of training the generator G is as follows:

[0037]

[0038] Among them, α1 and α2 are dynamic balance hyperparameters adjusted according to the number of training steps; Each is defined as where p dom D domThe predicted The domain label, Represents the domain label specified in the control mask v;

[0039] Preferably, in step S1, the facial changes include expression, posture or occlusion; The tag value is 0.

[0040] Preferably, in step S2, the control mask v is used to control prototype learning and designated domain conversion, and v consists of a 4-bit one-hot encoded vector, in which the first bit is used to control whether to perform prototype learning, and the last three bits are used to control designated domain information.

[0041] Preferably, in step S3.3, D dom Will The domain label prediction is the same as the domain label specified by the control mask v, that is, the domain information label specified by the last three bits in v; the initial values ​​of α1 and α2 are 0, the final value is 1, and it gradually increases with the number of training steps.

[0042] The beneficial effects of the present invention are:

[0043] 1. Improve the security and scope of application of face recognition systems. In the field of security, such as border inspection, security monitoring and other scenarios, face recognition technology is crucial. The present invention can process multimodal face images from different sensors or environments, such as visible light, near infrared, sketches, etc., thereby improving the coverage and accuracy of recognition. At the same time, the multimodal prototype reconstruction function can help restore damaged or blurred face images, improving the practicality and reliability of face recognition systems in emergency situations.

[0044] 2. The multimodal face recognition technology of the present invention can process face images of various different modalities, such as near-infrared images taken at night and sketch images based on descriptions, and accurately identify the target object. At the same time, the multimodal prototype reconstruction technology can reconstruct occluded, blurred or damaged face images of different modalities into high-quality face images in the visible light domain, improving the reliability and credibility of the evidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 The model generator G of the present invention;

[0047] Figure 2 is the model discriminator D;

[0048] Figure 3 This is the rendering of the near-infrared (NIR) multimodal prototype restoration;

[0049] Figure 4 This is the restoration rendering of the visible light domain (VIS) prototype;

[0050] Figure 5 Repair renderings for Sketch prototype;

[0051] Figure 6 This is the network structure diagram of the generator G model;

[0052] Figure 7 This is the network structure diagram of the discriminator D model;

[0053] Figure 8 is the feature vector domain classifier D c Network structure diagram. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] In the description of the embodiments of the present invention, it should be noted that if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0056] A multimodal face recognition and prototype reconstruction method based on a VAE-GAN framework includes the following steps:

[0057] S1. Model training set preparation:

[0058] Suppose a training set contains N identity class samples from domains A, B, and C. Each image x in the training set is labeled in Indicates the domain information label of x, Indicates that the image x belongs to domain A, that is, x~P dataA , similarly The value is 2 or 3, indicating that the domain belongs to domain B or domain C respectively. represents the identity label of x, and Indicates whether there are facial changes (such as expression, posture or occlusion) in image x, Indicates facial changes. Indicates that image x has no facial changes (i.e., it is a normal face image). Then, according to The value of is used to select face samples without facial changes from the training data set to build a prototype library. The pictures in the prototype library are denoted as x rp , in Represents x rp The domain information tag, Represents x rp identity tag, Represents x rp Whether it has facial changes (in the prototype library) The label value is 0).

[0059] S2. Model structure:

[0060] S2.1, Generator G: G consists of an encoder G enc and a decoder G dec Composition. enc Responsible for decoupling the modality-independent identity feature information P from the input x x . G dec Receive P x and control mask v (used to control prototype learning and specified domain conversion, v consists of a 4-bit one-hot encoded vector, the first bit is used to control whether prototype learning is performed, and the last three bits are used to control the specified domain information), concatenate them as input, and then generate an output x that matches the control information in v v .

[0061] like Figure 1 As shown, the model generator G. The input image x contains input images from the near-infrared domain, visible light domain, and sketch domain, P(x) is the identity feature information decoupled from the feature space, and v p Represents the original attribute mask, v is the control mask, which is used to control the generation of the prototype image of the specified domain. Reconstruct the generated image for the original attributes, A prototype image generated for the specified domain.

[0062] S2.2. Discriminator D and feature vector domain classifier D c : D includes three sub-discriminators D gan, D dom and D var . D gan It is a sub-discriminator related to GAN, used to distinguish between real prototypes and generated prototypes. dom It is a domain-information-dependent sub-discriminator used to infer the domain to which the input image belongs. var Used to detect whether the input image has changed. Feature vector domain classifier D c For further removal of G enc The decoupled identity feature information P x The possible domain information in D c Receive P x As input, it outputs the domain it predicts.

[0063] like Figure 2 As shown, the model discriminator D. The input image x represents the picture in the training set, Represents the prototype image of the specified domain generated by the generator G, x rp Represents a real prototype image in the prototype library. dom , D var , D gan They represent three sub-discriminators used to judge domain information, whether there is a change, and whether it is a true prototype.

[0064] S3. Model training:

[0065] S3.1. Stage 1: Extracting identity feature information P that is independent of modality x First, input the face image x into G enc Decouple the identity feature information P from it x =G enc (x), and P x Input to the feature vector domain classifier D c In, D c The goal is to predict the domain code, i.e. So D c The objective function is as follows:

[0066]

[0067] Where pre is D c The predicted domain code, is the domain code of the face image x, and P is the probability distribution over the domain pre.

[0068] In contrast, encoder G enc The purpose is to confuse D c Correctly predict field codes.

[0069] S3.2, Stage 2: Reconstruction learning to maintain the identity feature information of the generated prototype. Decoder G dec Learn to mask v with original attributes by decoding p P x To reconstruct the input face image x, that is where v p Like the control mask v, it is also used to control the decoder G dec The generated image content, v p Value by and The reconstruction learning preserves the identity information of the generated prototype in that it shares the decoder G with the subsequent prototype learning. dec , which will enable the feature information in the input image x to be maintained in the subsequent prototype learning process of generating prototypes. The objective function of reconstruction learning is as follows:

[0070]

[0071] S3.3, Stage 3: Prototype learning generates a face prototype for a specified domain. In this stage, the generator G and the discriminator D are trained adversarially to improve each other's performance through mutual competition, so that the generator can reconstruct multimodal face prototypes.

[0072] For the discriminator D = [D gan , D dom , D var ], which has three training objectives:

[0073] 1) The fake prototype generated by the given generator G and the real prototype x in the prototype library rp ,D gan Expect to is classified as a false prototype, and x rp Classified as a true prototype, so D gan The objective function is as follows:

[0074]

[0075] 2) Given a real prototype image x of a domain rp , D dom Expect to correctly predict its domain information label Therefore D dom The objective function is as follows:

[0076]

[0077] where p dom D dom The predicted field code.

[0078] 3) Given an input image x, D var Expect to correctly predict its Label, to determine whether the face image has changed. var The objective function is as follows:

[0079]

[0080] So the final objective function of the discriminator D is as follows:

[0081]

[0082] in They correspond to the above formulas (3)(4)(5) respectively.

[0083] For the generator G, it also has three training objectives:

[0084] 1) Cheating D gan Make it generate a fake prototype Classify as a true prototype. 2) Make D dom Will The domain label of is predicted to be the same as the domain label specified by the control mask v (i.e., the domain information label specified by the last three bits in v). 3) Make D var Will It is judged as a normal face picture without any changes. Therefore, the objective function of training the generator G is as follows:

[0085]

[0086] Among them, α1 and α2 are dynamic balance hyperparameters adjusted according to the number of training steps (the initial value is 0, the final value is 1, and it gradually increases with the number of training steps).

[0087] Each is defined as where p dom D dom The predicted The domain label, Indicates the control mask v specified in the field label.

[0088]

[0089] Example:

[0090] 1. Experimental Dataset

[0091] The near-infrared and visible light data used in this experiment come from the LAMP-HQ dataset, which is a large-scale, multi-pose, high-quality dataset designed for NIR-VIS face recognition. It is currently the largest and most challenging NIR-VIS dataset and was released by the Institute of Automation, Chinese Academy of Sciences in 2020. Since there is currently no large-scale public sketch dataset, the sketch data in this experiment comes from the professional image editing software FotoSketcher, which converts the NIR images in LAMP-HQ.

[0092] In this experiment, we randomly selected data from three domains, NIR, VIS, and Sketch, of 120 volunteers in the LAMP-HQ dataset for training and testing. Each volunteer provided 62 NIR images, 26 visible light images, and 62 sketch images, depicting five head changes (10 degrees and 45 degrees tilted to the left, frontal face, 10 degrees and 45 degrees tilted to the right), five scenes (indoor natural light, indoor strong light, indoor dark light, outdoor natural light, outdoor backlight), three occlusions (sunglasses, round glasses, square glasses), and three expressions (neutral, closed eyes, smiling). In the experiment, we divided the data of the 120 volunteers into a training set and a test set, where the training set contains data from 100 volunteers and the test set contains data from 20 volunteers, with no overlap between the two. Before training, all NIR and visible light images were converted to images with a size of 128×128 pixels for three channels, while the sketch data were converted to single-channel images with a size of 128×128 pixels.

[0093] 2. Model Specific Structure

[0094] The network structure of the generator model G is shown in the following figure. Figure 6 As shown. Encoder G enc It is mainly composed of convolutional layers, and each layer uses batch normalization technology, which helps to keep the distribution of input data within a relatively stable range during training. The activation function uses Leaky ReLU (Leaky Rectified Linear Unit). The input image passes through the encoder G dec After encoding, a 1024-dimensional vector containing identity feature information is generated. Decoder G dec It is mainly composed of transposed convolutional layers. The first four layers use the ReLU activation function, while the last layer uses the Tanh (hyperbolic tangent) activation function. Decoder G dec The feature vector from the encoder is received, and then concatenated with the control mask (composed of a 4-bit one-hot encoded vector) to output a 128×128 image. The network structure of the discriminator D is shown in the attached figure. Figure 7 As shown. Three sub-discriminators Dgan , D dom and D var The first 5 convolutional layers and one fully connected layer of the shared model are then added with the corresponding output layers according to their respective tasks. Feature vector domain classifier D c The network structure is as follows Figure 8 As shown in Figure 1, it is mainly composed of a fully connected layer. It receives a 1024-dimensional feature vector and outputs the predicted domain information.

[0095] 3. Model testing results

[0096] 3.1 Multimodal prototype reconstruction test

[0097] We conducted a multimodal prototype reconstruction experiment on the trained model to test its performance on face images in three different domains: near infrared, visible light, and sketch. Figure 3-5 As shown in the figure, the model's face prototype reconstruction effect in different domains is demonstrated. It is worth noting that we observed that the model showed good versatility and generalization ability on face images across multiple domains. Whether it is near-infrared, visible light or sketch images, the model can effectively process and successfully reconstruct the face prototype image of the specified domain. This shows that our model has strong adaptability and can handle data from a variety of different domains, providing an effective solution for multi-modal domain face image processing.

[0098] Table 1: Multimodal face recognition accuracy

[0099] NIR→VIS Sketch → VIS 91.2％ 86.6％

[0100] 3.2 Multimodal face recognition test

[0101] In order to simulate the face recognition task in the real scene as much as possible in the experiment, we adopted an enterprise-level face recognition API interface provided by Tencent Cloud. First, we entered the visible light pictures of the front faces of 120 volunteers into the face database and used them as registered faces. Then, we used our proposed method to process the near-infrared and sketch front face pictures of these volunteers (two near-infrared and two sketch pictures), first decoupled the identity feature information, and then decoded it into face pictures in the visible light domain for subsequent testing. This processing process is designed to eliminate the modal differences between different optical domains. We input the converted multi-optical domain face images into the face recognition system and test their face recognition accuracy. The experimental results are shown in Table 1 above, which shows that our proposed method has shown significant effectiveness in processing multi-modal face recognition tasks.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal face recognition and prototype reconstruction method based on VAE-GAN framework, characterized in that: The following steps are involved: S1. Model training set preparation: Assume that a training set contains N identity class samples from domains A, B, and C; each image x in the training set is labeled in Indicates the domain information label of x, Indicates that the image x belongs to domain A, that is, x~P dataA , similarly A value of 2 or 3 indicates belonging to domain B or domain C, respectively; represents the identity label of x, and Indicates whether there is facial change in image x, Indicates facial changes. Indicates that image x has no facial changes and is a normal face image; then, according to The value of is used to select face samples without facial changes from the training data set to build a prototype library. The pictures in the prototype library are denoted as x rp , in Represents x rp The domain information tag, Represents x rp identity tag, Represents x rp Whether there are facial changes; S2. Model structure: S2.1, Generator G: G consists of an encoder G enc and a decoder G dec Composition: G enc Responsible for decoupling the modality-independent identity feature information P from the input x x ; G dec Receive P x and control mask v, concatenate them as input, and then generate output x that matches the control information in v v ; S2.

2. Discriminator D and feature vector domain classifier D c : D includes three sub-discriminators D gan , D dom and D var ;D gan It is a sub-discriminator related to GAN, used to distinguish between real prototypes and generated prototypes; D dom It is a domain-information-dependent sub-discriminator used to infer the domain to which the input image belongs; D var Used to detect whether the input image has changed; feature vector domain classifier D c For further removal of G enc The decoupled identity feature information P x The possible domain information in D c Receive P x As input, and outputs the domain it predicts; S3. Model training: S3.

1. Stage 1: Extracting identity feature information P that is independent of modality x ; First, input the face image x into G enc Decouple the identity feature information P from it x =G enc (x), and P x Input to the feature vector domain classifier D c In, D c The goal is to predict the domain code, i.e. So D c The objective function is as follows: Where pre is D c The predicted domain code, is the domain code of the face image x, P is the probability distribution over the domain pre; In contrast, encoder G enc The purpose is to confuse D c Correctly predict domain codes; S3.2, stage 2: reconstruction learning to maintain the identity feature information of the generated prototype; decoder G dec Learn to mask v with original attributes by decoding p P x To reconstruct the input face image x, that is where v p Like the control mask v, it is also used to control the decoder G dec The generated image content, v p Value by and Determine; reconstruction learning maintains the identity feature information of the generated prototype in that it shares the decoder G with the subsequent prototype learning dec , which will enable the feature information in the input image x to be maintained in the subsequent prototype learning process of generating prototypes; the objective function of reconstruction learning is as follows: S3.3, Stage 3: Prototype learning generates a face prototype for a specified domain; in this stage, the generator G and the discriminator D are trained adversarially to improve each other's performance through mutual competition; this enables the generator to reconstruct a multimodal face prototype; for the discriminator D = [D gan , D dom , D var ], which has three training objectives: 1) The fake prototype generated by the given generator G and the real prototype x in the prototype library rp ,D gan Expect to is classified as a false prototype, and x rp Classified as a true prototype, so D gan The objective function is as follows: 2) Given a real prototype image x of a domain rp , D dom Expect to correctly predict its domain information label Therefore D dom The objective function is as follows: where p dom D dom The predicted domain code; 3) Given an input image x, D var Expect to correctly predict its Label, to determine whether the face image has changed; therefore, D var The objective function is as follows: So the final objective function of the discriminator D is as follows: in Corresponding to the above formulas (3)(4)(5) respectively; For the generator G, it also has three training objectives: 1) Cheating D gan Make it generate a fake prototype Classified as a true prototype; 2) Make D dom Will The domain label of is predicted to be the same as the domain label specified by the control mask v; 3) Make D var Will It is judged as a normal face picture without any changes; therefore, the objective function of training the generator G is as follows: Among them, α1 and α2 are dynamic balance hyperparameters adjusted according to the number of training steps; Each is defined as where p dom D dom Predicted The domain label, Represents the domain label specified in the control mask v; 2. The multimodal face recognition and prototype reconstruction method based on the VAE-GAN framework according to claim 1, characterized in that: In step S1, facial changes include expression, posture or occlusion; The tag value is 0.

3. The multimodal face recognition and prototype reconstruction method based on the VAE-GAN framework according to claim 1, characterized in that: In step S2, the control mask v is used to control prototype learning and specified domain conversion. v consists of a 4-bit one-hot encoded vector, where the first bit is used to control whether prototype learning is performed, and the last three bits are used to control the specified domain information.

4. The multimodal face recognition and prototype reconstruction method based on the VAE-GAN framework according to claim 3, characterized in that: In step S3.3, D dom Will The domain label prediction is the same as the domain label specified by the control mask v, that is, the domain information label specified by the last three bits in v; the initial values ​​of α1 and α2 are 0, the final value is 1, and it gradually increases with the number of training steps.

Images