Palm vein recognition data enhancement method and system based on adversarial learning

Abstract

The present application belongs to the technical field of palm vein recognition, and particularly relates to a palm vein recognition data enhancement method and system based on adversarial learning, which utilizes a conditional deep convolutional generative adversarial network composed of a generator and a discriminator, and synchronously trains the generator and the discriminator; step 2 selects a palm vein classifier and the trained generator to form an adversarial network, step 3 performs multiple batches of adversarial training on the palm vein classifier based on the adversarial network, generates adversarial samples for increasing the loss of the classifier and training samples based on real palm vein recognition data to form an augmented data set in each batch of training, trains the classifier, and updates the parameters of the classifier; and updates the noise vector set required for the next batch of adversarial training. The present application reduces the requirements for computing and storage resources in data enhancement, and provides more high-quality training samples for palm vein recognition model training.

Description

Technical Field

[0001] This invention belongs to the field of palm vein recognition technology, specifically relating to a palm vein recognition data enhancement method and system based on adversarial learning. Background Technology

[0002] Vein feature extraction methods can be divided into three categories: manual feature extraction, traditional machine learning-based methods, and deep learning-based methods. However, all three methods have certain drawbacks.

[0003] The vein features obtained by manual feature extraction are incomplete; vein features extracted based on traditional machine learning methods can learn robust vein features, but their representational ability is limited, resulting in unstable recognition performance; deep learning-based methods can automatically learn robust features of the original image, but require a large number of samples to train the model, while vein datasets are very limited.

[0004] To address the limited availability of vein datasets, Generative Adversarial Networks (GANs) have emerged as an effective data augmentation technique. These GANs can learn the distribution of palm vein images, thereby generating realistic images with good consistency and diversity. Currently, GANs are widely used for data augmentation and have achieved good performance in various computer vision tasks. Some current work introduces GANs to expand the training data for palm vein recognition, palmprint recognition, and finger vein recognition. However, standard GANs cannot control the data generation process and require training a separate GAN model for each class to generate samples, which increases the demands on computational and storage resources. Summary of the Invention

[0005] To address the shortcomings of the three methods mentioned above and the classic GAN, this invention proposes a palm vein recognition data augmentation method based on adversarial learning, which simultaneously optimizes the vein classifier and a set of augmented samples.

[0006] The palm vein recognition data augmentation method based on adversarial learning in this invention includes:

[0007] Step 1 uses a conditional deep convolutional generative adversarial network (cDCGAN) consisting of a generator and a discriminator to train the generator and discriminator simultaneously.

[0008] The generator takes different noise vectors and the category to be generated as input to generate enhanced training samples for palm vein recognition data classification.

[0009] The discriminator is used to distinguish between the enhanced training samples generated by the input generator and the training samples based on real palm vein recognition data.

[0010] During training, the generator is first trained to generate enhanced training samples by taking the input random noise vector and class labels. Then, the enhanced training samples or training samples with added class labels are forwarded to the discriminator to predict their true or false probabilities.

[0011] And minimize the adversarial loss to update the parameters of the discriminator and generator for the target;

[0012] Step 2: Select a palm vein classifier and the trained generator to form an adversarial network. The palm vein classifier is used to predict the probability that the input sample belongs to a certain class.

[0013] Step 3 involves performing adversarial training on the palm vein classifier using the adversarial network in multiple batches. Each batch of training includes:

[0014] The noise vector set and the categories to be generated are obtained as input to the generator to generate adversarial examples to increase the classifier loss;

[0015] An augmented dataset for classifier training is constructed using training samples based on real palm vein recognition data and the adversarial samples. The classifier is then trained and its parameters are updated.

[0016] Furthermore, after updating the parameters of the classifier by inputting adversarial examples, the noise vector corresponding to each adversarial example is updated one by one using gradient descent based on the output of the classifier; finally, the updated noise vector set is used as the noise vector set required for the next batch of adversarial training.

[0017] Furthermore, during the training process in step 1, a dual-timescale update rule and a one-sided label smoothing strategy are adopted.

[0018] Furthermore, during the training process in step 1, there is a discriminator D with parameter vector w'. w' A generator G with parameter vector w w Update w' and w in the following ways:

[0019] w n+1 =w n +a(n)(g(w,w′)+V w )

[0020] w' n+1 =w' n +b(n)(h(w,w′)+V w' ),

[0021] Where g(w,w') and h(w,w') represent the calculations respectively. and and Let V be any loss function of the discriminator D and the generator G. w and V w' Let n be a random variable, and a(n) and b(n) be configurable learning rates.

[0022] Furthermore, during the training process in step 1, the positive classification target of the discriminator D is μ, and the negative classification target is υ.

[0023] The output of discriminator D can be expressed as:

[0024]

[0025] In the formula p data (x) and p model (x) represents the actual data distribution and the model distribution, respectively.

[0026] Furthermore, during the training process in step 3, the palm vein classifier is updated in the following manner. The parameter θ:

[0027]

[0028] Where N represents the number of randomly selected training samples, N1 represents the number of adversarial samples, and N2 represents the number of training samples based on real palm vein recognition data, so N = N1 + N2.

[0029] α is the configurable learning rate, G w* (·) represents the trained generator, L ce (·) represents the classifier The loss function;

[0030] Noisy vector set Class tag set Raw palm vein recognition data sample set Where K is the number of samples generated for each class, and C is the number of classes.

[0031] Furthermore, during the training process in step 3, the noise vector set is updated in the following manner. The noise vectors in the data:

[0032]

[0033] Where β is the configurable learning rate, and cosine(·) is a function of cosine similarity;

[0034] It is a teacher model, its parameters Palm vein classifier The exponential moving average of the parameter θ, with λ1 and λ2 as given weights, has λ1 + λ2 = 1.

[0035] Furthermore, the training samples based on real palm vein recognition data include real palm vein images.

[0036] And classic data-enhanced images obtained from real palm vein images using classic data enhancement techniques.

[0037] Furthermore, the palm vein classifier employs one of the following models as the feature extraction model: ResNet, VGG, FV-CNN, PV-CNN, FVRAS-Net, Lightweight CNN, or ViT.

[0038] Furthermore, in step 3, the loss function is calculated in the following manner.

[0039]

[0040] Furthermore, in step 3, the following smoothing labels are used:

[0041]

[0042] Where ω∈[0,1) is the smoothing parameter.

[0043] The present invention also discloses a palm vein recognition data augmentation system based on adversarial learning, including a generator, a discriminator and a palm vein classifier. The generator and the discriminator form a conditional deep convolutional generative adversarial network, and the generator and the palm vein classifier form an adversarial network.

[0044] The system trains the generator and discriminator according to the aforementioned method, and further trains the palm vein classifier using the trained generator.

[0045] The method and system in this invention overcome the shortcomings of standard GANs, which cannot control the data generation process and require training a GAN model for each class to generate samples. At the same time, it optimizes the vein classifier and a set of augmented samples, reduces the requirements for computing and storage resources in data augmentation, provides higher quality training samples for the training of palm vein recognition models, and breaks through the limitation of the number of real vein datasets. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the palm vein recognition data enhancement method based on adversarial learning in an embodiment of the present invention. Detailed Implementation

[0047] The palm vein recognition data enhancement method based on adversarial learning in this embodiment is basically as follows: Figure 1As shown, a cDCGAN consisting of a generator and a discriminator is first used for data augmentation. To improve performance, a dual-time-scale update rule (TTUR) and one-sided label smoothing are used to train the cDCGAN, and WGAN-GP loss is used to stabilize the training. After training, different noise vectors and conditional information are used as inputs to the generator to generate data augmentation samples.

[0048] Secondly, the trained generator is combined with the vein classifier to obtain an adversarial network consisting of two adversarial modules. One is a palm vein classifier used to predict the probability that an input sample belongs to a class, and the other is a set of noise vectors used as input to the trained generator to generate adversarial examples, thereby increasing the loss of the classifier. The two modules are updated alternately during training.

[0049] More specifically, firstly, due to the limited size of the palm vein dataset, directly training cDCGAN is prone to overfitting. To overcome this problem, classic geometric transformations, such as rotation, scaling, flipping, cropping, and translation, and color enhancement transformations, such as contrast, brightness, hue, equalization, or saturation, can be used to expand the palm vein dataset. However, some enhancement techniques, such as cropping, are not suitable for palm vein image enhancement because the enhanced data distribution differs from the original image. Furthermore, large rotation angles alter the data distribution because black areas exist in the rotated or cropped image, making them unsuitable for palm vein image enhancement. To effectively expand the training data, this embodiment uses classic data augmentation approaches, such as translating brightness, contrast, hue, and saturation in different directions, generating approximately 50 images per class to form classic data augmented images. These are then combined with the original palm vein image (Original Palm-vein example) to obtain a new training dataset (Augmented palm-vein dataset), such as... Figure 1 As shown, the actual image data used for training is the region of interest (ROI) in the original palm vein image.

[0050] Conditional deep convolutional generative adversarial networks (cDCGANs) can generate training samples for each class by adding additional information. Therefore, in this embodiment, a conditional deep convolutional generative adversarial network (cDCGAN) for palm vein data augmentation is disclosed.

[0051] The image generation process is controlled by adding additional information to the model input, which corresponds to class labels. The generator takes a random noise vector and class labels as input to output a generated image. It then forwards either the generated image or a real image with class labels added to it to the discriminator to predict its probability of being real or fake. To improve the model's stability, a cDCGAN with stochastic gradient descent is trained on any GAN loss function using the dual time-scale update rule (TTUR).

[0052] Meanwhile, a one-sided label smoothing mechanism is introduced into cDCGANs to improve model performance. One-sided label smoothing refers to replacing the 0 and 1 targets of the classifier with smoothed values ​​to reduce the discriminator's vulnerability to adversarial examples. Therefore, in this embodiment, the positive classification target is replaced with μ, and the negative classification target is replaced with υ, resulting in the following discriminator.

[0053]

[0054] In the formula p data (x) and p model (x) represents the real data distribution and the model distribution, respectively, with the variables μ and υ set to 0.9 and 0, respectively.

[0055] Let z be the noise vector and y be the conditional information, such as the class label. The discriminator and generator are denoted as D and G, respectively. The generator takes z and y as input and outputs the generated image x' = G(z|y). The discriminator receives the generated image as input and classifies it as false or true. The objective function of the conditional GAN ​​is defined as Equation (2):

[0056]

[0057] Classical GANs are difficult to train because the divergence they typically minimize may be independent of the generator's parameters. To accelerate convergence, the WGAN-GP loss is used as an adversarial loss, defined as Equation (3):

[0058]

[0059] in Indicates data distribution, Indicates by The implicitly defined generator distribution represented by z ~ p(z), where z is sampled from some simple noise distribution. Defined as along and The sampling points are uniformly sampled along a straight line between the sampling points, and λ is the penalty coefficient for the unit gradient norm constraint.

[0060] During training, the discriminator D is first trained to distinguish between generated samples and real data, and then the generator G is trained to deceive the discriminator. To improve the stability of the GAN, a dual time-scale update rule (TTUR) is used to update the weights of the discriminator and the generator.

[0061] Here we consider a discriminator D with parameter vector w'. w' A generator G with parameter vector w w .set up and Let g(w,w') and h(w,w') be the loss functions for the discriminator D and the generator G, respectively, and calculate them as follows: and Then merge the random variables V w and V w' The approximate value of the true gradient is calculated using the following method: password and password Furthermore, the gradient approximations are stochastic because they are calculated based on mini-batches of m real-world samples x and synthetic samples x', which are randomly selected. The resulting dual-timescale stochastic approximation, namely the dual-timescale update rule (TTUR), is used to train the GAN. In the TTUR, the discriminator and generator are updated with learning rates a(n) and b(n) according to equations (4) and (5), respectively.

[0062] w n+1 =w n +a(n)(g(w,w′)+V w (4)

[0063] w' n+1 =w' n +b(n)(h(w,w′)+V w' (5)

[0064] After training, the generator learns the true data distribution, thus enabling it to produce high-quality samples for data augmentation.

[0065] Therefore, assuming It is a set of noise vectors. Let K be the class label set, where K is the number of samples generated for each class, and C is the number of classes. After training the GAN, a noise vector is sampled from the noise distribution space. The obtained vector z and its corresponding label As a trained generator G w* The input is used to generate K synthetic samples x through equation (6). ck :

[0066] x ck =Gw* (z,y). (6)

[0067] In existing technologies, various deep learning classifiers are used for vein recognition. It is the training set, where x cl Let represent the l-th image from class c, where L is the number of samples in each class. Let For feature extraction models, these models can be, but are not limited to, ResNet, VGG, FV-CNN, PV-CNN, FVRAS-Net, Lightweight CNN, or ViT, where θ are trainable weights. Example of a classifier map. To tag Deep learning-based classifiers are... This is achieved through a combination of parameters and a softmax regression layer, which predicts probabilities using equation (7):

[0068]

[0069] Where w c This is a vector of classification parameters for the c-th class;

[0070] Given a training set {x c ,y c}, parameters θ and By minimizing the loss function of equation (8), we obtain:

[0071]

[0072] In this embodiment, the trained generator in equation (6) is used to generate palm vein images, and a deep learning-based classifier is effectively trained on this basis for palm vein recognition. However, while the generator can generate palm vein samples based on different input noise vectors z, it is impossible to train the classifier using all synthetic palm vein samples in the data distribution space. Typically, a finite number of samples are generated for classifier training, but such a finite number of samples is unlikely to guarantee effective training of the target classifier. To overcome this problem, an adversarial framework is proposed in this example to jointly optimize the training of the classifier and the generation of augmented datasets. Setting It is a set of noise vectors, where z ck It is the k-th noise vector of class c. It is related to tag sets The relevant training set. Based on generator G w A composite image set can be obtained. Then the two sets and The images are merged to form an augmented dataset Ω. In this dataset, each class has (K+L) images, resulting in (K+L)×C images. An object classifier is trained based on the augmented dataset Ω. To minimize the loss in equation (9). Generator G w (·) By searching the noise vector set As its input, it attempts to generate an enhanced image set. Reduce the target network through adversarial learning The training loss is considered. Finally, the goal is to reach a balance point where the learned representation achieves maximum performance.

[0073] Consider target classifier The loss function is L ce (·). The trained generator G w* (·) Transform the random noise space into the sample data space; thus, the adversarial loss definition for joint training is minimized as shown in equation (9).

[0074]

[0075] Here, the predicted label for each query in the current mini-batch is a function of the classifier weights, thus optimizing θ through them. Similarly, The noise vector in the equation is considered as a composite function. The free variables can be optimized for a given θ. In Equation (9), the first term on the right is the loss calculated based on the generated image, and the second term is the loss calculated based on the original image. During training, the generated image and the original image are mixed and input into the target classifier for training. For ease of description, this is referred to as two terms here. To optimize the parameter θ, G w* (·) Generating images with given noise vectors to challenge the classifier may result in the loss of the inherent meaning of the images. To address this issue, cosine similarity and the teacher model are introduced as two regularization terms to control the selection of noise vectors in the set Z, and the loss minimization problem is transformed into Equation (10).

[0076]

[0077] Where cosine(·) is a function of cosine similarity, where It is a teacher model, which is the exponential moving average (EMA) of the target model, for example:

[0078] As described by many existing adversarial training algorithms, it is difficult to find a saddle point in equation (10). Solution. Generally, a pair of descending and ascending gradients are used to alternately update θ and θ. For example, the optimization process of θ can be transformed into the following minimization problem (11).

[0079]

[0080] Where λ1=λ2=0.5. The problem in equation (11) is usually solved using vanilla SGD with a learning rate of α and a batch size of N, where N=N1+N2, and N1 and N2 correspond to the number of generated images and the number of original images in the current batch, respectively. The training process for each batch can be expressed as:

[0081]

[0082] Since the cosine similarity and the teacher model are independent of the variable θ, the above equation is changed to equation (13):

[0083]

[0084] The training process uses gradient calculations on average over N instances to reduce gradient variance, which leads to faster convergence of the target network. However, based on limited training data, overfitting is also prone to occur. To overcome this problem, the training sample set generated by the GAN-based augmentation network serves as an adversary to the target network, leading to a minimax problem for the training network. Such a self-supervised target may be challenging enough to prevent the learned representation from overfitting to the target. Therefore, the object can be mathematically defined as the maximization problem in equation (14) as follows:

[0085]

[0086] To address the above issues, a gradient ascent method is employed for updating the parameters, with a learning rate of β. The parameter update rule is defined as shown in equation (15):

[0087]

[0088] Intuitively, the optimization of equation (14) can be interpreted as for adversarial training When L ce (φ θ (G w* (z ck ,y c )),y c Maximize and Minimize, the classifier's loss in Equation (8) is maximized. In other words, this tends to push the generated samples far. Samples from the same class, within the range that the teacher model can recognize and maintaining a constrained distance from the original image, generate challenging samples along the trajectory of the classifier update.

[0089] In this embodiment, the adversarial learning method for searching the noise set of the target network training and augmentation data is summarized as Algorithm 1:

[0090]

[0091] As can be seen from the algorithm above, the training of the target network and the search for the noise set of augmented data are performed alternately.

[0092] like Figure 1 As shown, in each training round, the current noise vector set is utilized. and tag set And the original training sample set is used to generate an augmented sample dataset Ω for training the target network;

[0093] During each training iteration, the parameters θ (Updatedtarget classifier) ​​of the target network are updated using the gradient descent method in formula (13), while the noise vector set is... Remains unchanged (Fixed Latent variable);

[0094] Next, in the noise set of the search augmented data, the noise vector is updated using formula (15) during each training iteration, thereby updating the noise vector set. (Updated Latent variable), during which the parameter θ of the noisy target network remains unchanged (Fixed target classifier).

[0095] It is worth noting that tag sets It remains unchanged.

[0096] The training process of the model is similar to that of Generative Adversarial Networks (GANs). To improve performance, two improvements are introduced into the adversarial model: non-saturation loss and label smoothing.

[0097] Non-saturation loss: Most classification tasks use cross-entropy loss as the loss function, i.e., equation (16).

[0098]

[0099] Based on this loss function, when the target model's predictions are highly confident, the gradient of the first term in equation (10) tends to saturate in the problem of maximizing Z. To overcome this problem, instead of using equation (16), the non-saturating loss defined in equation (17) is used to train the augmentation model.

[0100]

[0101] Note that the above formula shows The calculation method, in some embodiments The calculation can be obtained in the same way.

[0102] Label smoothing: Label smoothing techniques are applied to various classification tasks and are defined as...

[0103]

[0104] Where ω∈[0,1) is the smoothing parameter. According to equation (18), one-hot label y c Converted into smooth labels Label smoothing techniques, under non-saturation loss, can prevent gradient explosion, especially for simple tasks or strong target models, when the classifier's decisions are very confident.

Claims

1. A palm vein recognition data augmentation method based on adversarial learning, characterized in that, The palm vein recognition data augmentation system based on adversarial learning comprises a generator, a discriminator and a palm vein classifier, the generator and the discriminator constitute a conditional deep convolutional generative adversarial network, and the generator and the palm vein classifier constitute an adversarial network. Step 1: The conditional deep convolutional generative adversarial network composed of the generator and the discriminator is used for synchronous training of the generator and the discriminator. The generator generates enhanced training samples for palm vein recognition data classification by taking different noise vectors and the category to be generated as input. The discriminator is used for distinguishing the enhanced training samples generated by the generator and the training samples based on real palm vein recognition data. In the training, the generator generates enhanced training samples by inputting random noise vectors and category labels, and then forwards the enhanced training samples or the training samples with additional category labels to the discriminator to predict the probability of authenticity. The parameters of the discriminator and the generator are updated with the goal of minimizing the adversarial loss. Step 2: A palm vein classifier is selected to constitute an adversarial network with the trained generator, and the palm vein classifier is used to predict the probability that the input sample belongs to a certain category. Step 3: The palm vein classifier is subjected to multiple batches of adversarial training based on the adversarial network, and each batch of training includes: Obtaining a set of noise vectors and categories to be generated as input of the generator to generate adversarial samples for increasing the classifier loss; Training the classifier using the augmented data set composed of the training samples based on real palm vein recognition data and the adversarial samples, updating the parameters of the classifier, and inputting the adversarial samples into the classifier with updated parameters to update the noise vectors corresponding to each adversarial sample according to the output of the classifier using the gradient descent method; and finally obtaining the updated noise vector set as the noise vector set required for the next batch of adversarial training. The output of the discriminator D can be expressed as: parameters of the palm vein classifier are updated during training in the following manner :​ ； Wherein, N represents the number of randomly selected training samples, represents the number of adversarial samples therein, represents the number of training samples based on real palm vein recognition data therein, and N= + ; a settable learning rate, a trained generator, a classifier a loss function; There are , class label set , original palm vein recognition data sample set , where K is the number of samples generated for each class, and C is the number of classes. The training process updates the parameters of the model in the following way each noise vector in ， wherein, is a settable learning rate, (·) is a function of the similarity; is a teacher model, whose parameters are the parameters of the palm vein classifier are the parameters of the palm vein classifier is an exponential moving average of is a given weight, and = 1.

2. The method of claim 1, wherein, During the training process in step 1, there is a parameter vector as Discriminator and parameter vector are generator Update via the following methods and : ， ， wherein, and respectively represent a calculation and and is an arbitrary loss function of a discriminator and a generator , and are random variables, and are settable learning rates.

3. The method of claim 1, wherein, During the training process of Step 1, the positive classification target of the discriminator D is , and the negative classification target is , The training samples based on real palm vein recognition data include real palm vein images, ， wherein and respectively represent the real data distribution and the model distribution.

4. The method of claim 1, wherein, and classic data augmentation images obtained by classic data augmentation means based on real palm vein images. One of ResNet, VGG, FV-CNN, PV-CNN, FVRAS-Net, Lightweight CNN or ViT is used as a feature extraction model in the palm vein classifier.

5. The method of claim 1, wherein, In step 3, the loss function is calculated as follows:

6. The method of claim 1, wherein, In step 3, the following smooth label is used: 。 7. The method of claim 1, wherein, 8. A palm vein recognition data augmentation system based on adversarial learning, comprising a generator, a discriminator and a palm vein classifier, the generator and the discriminator constitute a conditional deep convolutional generative adversarial network, and the generator and the palm vein classifier constitute an adversarial network; ； wherein ∈ [0, 1) is a smoothing parameter. The system trains the generator and the discriminator according to the method of any one of claims 1-7, and further trains the palm vein classifier using the trained generator. ​

Images