Iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge driven

Abstract

The present invention discloses an iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge driving, the method comprising: step 1, establishing a data set and dividing it into a training set and a test set; step 2, obtaining a model architecture; step 3, training the model architecture; step 4, obtaining a convolutional coding part; step 5, positioning the target detection model in the iris and periorbital area; step 6, after the training is completed, removing the last fully connected layer to obtain a periorbital feature extraction model E per ; Step 7, use the iris visible light image training set and personnel identity label training; Step 8, align the feature distributions of different modalities in the low-dimensional subspace; Step 9, set two learnable parameters; Step 10, obtain the model feature extractor; Step 11, determine the decision mechanism of the dual-modal fusion recognition model. Beneficial effects: Improve the reliability and security of the system; Improve the recognition performance on a small sample data set; Can improve the final recognition performance.

Description

Technical Field

[0001] The present invention relates to an iris and periorbital adversarial adaptive fusion recognition method, and in particular to an iris and periorbital adversarial adaptive fusion recognition method driven by contrast knowledge. Background Art

[0002] At present, as information security plays an increasingly important role in the information society, biometric technologies such as iris, fingerprint, and face, and behavioral characteristics such as gait, signature, and keystrokes have solved the problems of easy loss and forgery of traditional identity authentication, and are being widely used in access control, identity recognition, and other scenarios. Among them, iris recognition, as a non-contact biometric technology, is used in military, finance, banking, smart construction sites, smart mines, smart homes, and other fields due to its uniqueness, stability, difficulty in theft, ease of use, and its accuracy rate is second only to DNA in biometric technology.

[0003] Since the iris is located in the internal organ area of ​​the eye and has a small area, the user needs to cooperate at a small distance to effectively collect the iris image. In addition, since the iris color of Asians is dark brown, iris collection mostly uses more expensive near-infrared light collection equipment, which restricts the civilian promotion of iris recognition. Single iris identity recognition technology is susceptible to noise, low upper limit of recognition rate, and presentation attacks. In addition, previous multimodal solutions such as iris and face fusion have reduced recognition rates due to the face being blocked by a mask. In addition, heterogeneous modal fusion has inconsistent feature distribution, which leads to a decrease in recognition rate after fusion.

[0004] Therefore, how to solve key problems such as the small iris recognition acquisition distance, the expensive acquisition method cost, the low robustness of a single iris recognition system, and how to improve the recognition rate of heterogeneous modal fusion is a focus of current iris recognition and multimodal biometrics technology. Summary of the invention

[0005] The main purpose of the present invention is to solve the problems of high acquisition conditions, low security and reliability of existing single-modality biometric iris recognition;

[0006] Another object of the present invention is to improve the performance of multimodal biometrics technology that integrates iris with other modalities;

[0007] Another object of the present invention is to provide a training paradigm to solve the overfitting problem of deep neural networks when a large-scale iris database is not available, thereby improving the recognition performance of biometric features such as iris.

[0008] In order to solve the above problems and achieve the above objectives, the present invention provides an iris and periorbital adversarial adaptive fusion recognition method driven by contrast knowledge.

[0009] The present invention provides an iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge driving, and the method comprises the following steps:

[0010] Step 1: Establish eye visible light image dataset, iris and periocular target detection dataset, iris visible light image dataset and periocular visible light image dataset, and divide them into training set and test set. The training set is used to train the iris and periocular area detection network model and the iris and periocular feature fusion model, and the test set is used to evaluate the accuracy of the model.

[0011] Step 2: Initialize three MobileNetV3 deep convolutional neural networks. For each network model, remove the last fully connected layer of MobileNetV3 as the encoding layer, add a projection layer after the encoding layer, and add a prediction layer after the projection layer to obtain three initialized model architectures.

[0012] Step 3: Use the eye visible light image training set collected in step 1 to train the three initialization model architectures obtained in step 2 using a comparative knowledge-driven algorithm;

[0013] Step 4: Remove the projection layer and prediction layer of the three trained model architectures to obtain the convolutional coding parts of the three trained MobileNetV3s;

[0014] Step 5: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4 and use it as the backbone to build the IrisPer_YoloV3 target detection model, freeze the MobileNetV3 convolutional coding part, and use the iris and periorbital target detection training set to train the IrisPer_YoloV3 target detection model to locate the iris and periorbital area;

[0015] Step 6: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4, add a multi-layer perceptron as a classifier, freeze the MobileNetV3 convolutional coding part, use the periocular visible light image training set and personnel identity label training, and remove the last fully connected layer after the training to obtain the periocular feature extraction model E. per ;

[0016] Step 7: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4, add a multi-layer perceptron as a classifier, freeze the MobileNetV3 convolutional coding part, use the iris visible light image training set and personnel identity label training, and remove the last fully connected layer after the training to obtain the iris feature extraction model E. iris ;

[0017] Step 8: Build a fully connected neural network C that integrates the classifier and discriminator, and use the iris feature extraction model E obtained in steps 6 and 7 iris and eye contour extraction model E per Together with the fully connected neural network C integrated with the classifier and discriminator, the iris visible light training set and the eye periocular visible light training set are used for joint adversarial training with the identity category labels, with the aim of aligning the feature distributions of different modalities in a low-dimensional subspace.

[0018] Step 9: Set two learnable parameters and use the iris feature extractor E obtained through adversarial training in step 8. iris and the eye periphery feature extractor E per , then add a multi-layer perceptron as the fusion classifier F to build the iris and periorbital fusion model, freeze the iris feature extraction model and the periorbital feature extraction model, use the iris visible light training set and the periorbital visible light training set to train the entire fusion recognition model, and optimize the multi-layer perceptron of the fusion classifier F;

[0019] Step 10: The iris and periocular fusion model obtained in step 9 is jointly trained with the iris and periocular synchronous detection model IrisPer_YoloV3 target detection model obtained in step 5, and the last layer of the neural network is removed to obtain the final iris and periocular area detection and fusion recognition model feature extractor;

[0020] Step 11: Use a multi-stage decision strategy collaborative solution as the decision mechanism of the trained iris and periorbital dual-modal fusion recognition model.

[0021] The specific process of establishing the eye visible light image dataset, iris and periorbital target detection dataset, iris visible light image dataset and periorbital visible light image dataset in the above step 1 is as follows:

[0022] The first step is to use a high-definition camera to collect 2,500 visible light images of the template tester's template eyes, of which 2,000 are divided into a training set and 500 are divided into a test set, as the visible light image dataset of the eyes;

[0023] Step 2: Use the labelme tool to annotate the iris area and periocular area of ​​each sample in the eye visible light image training set and test set to obtain the target detection position label of the iris area and periocular area. The label is combined with the eye visible light image dataset to obtain the iris and periocular target detection dataset.

[0024] The third step is to crop each sample in the eye visible light image dataset to obtain iris visible light images and periocular visible light images according to the labels in the iris and periocular target detection dataset obtained in the second step, and assign an identity category label to each image. The identity labels and iris visible light images and periocular visible light images construct the iris visible light dataset and periocular visible light dataset respectively.

[0025] The specific process of constructing the projection layer and prediction layer in the above step 2 is as follows:

[0026] The first step is to build a 3-layer multi-layer perceptron as the projection layer. Each fully connected layer has 1024 neurons, followed by a BN layer or batch normalization layer;

[0027] The second step is to build a 2-layer multilayer perceptron as the prediction layer. The prediction layer is a 2-layer multilayer perceptron. Except for the output layer, other layers are followed by a BN layer. The dimension of the hidden layer is 256, and the dimension of other layers is 1024.

[0028] The comparative knowledge-driven training process in step 3 above is as follows:

[0029] The first step is to perform random data enhancement on samples in the eye visible light image training set data. The eye visible light image is randomly cropped according to the ratio of the original image size [0.5, 2.0], and then the image size is adjusted using the bilinear interpolation algorithm. Then, the image is randomly flipped left and right, and the image contrast, brightness, hue and saturation are randomly adjusted. Then, the image is randomly grayed, and finally, the image is randomly blurred using the mean filter, Gaussian blur, median blur, bilateral filter and box blur.

[0030] In the second step, the random enhancement operation in the first step is used to perform two random enhancement operations on each image X to obtain images X1 and X2. The two images X1 and X2 are respectively input into a convolutional coding part f, and the output is expressed as m1=f(X1) and m2=f(X2). m1 and m2 are input into the projection layer g, and the output is expressed as n1=g(m1) and n2=g(m2). n1 and n2 are input into the prediction layer h, and the output is expressed as v1=h(n1) and v2=h(n2). The architecture idea of ​​the twin neural network is adopted, and the weights of f, g, and h are shared to construct a comparative knowledge-driven training paradigm.

[0031] Step 3: Take the eye visible light image training set with a BatchSize of 512, the optimizer is SGD, the initial learning rate is 0.1, and the contrast knowledge driven learning architecture is input for training. The contrast knowledge driven loss function is formula (1);

[0032]

[0033] Where: L contrast represents the contrast knowledge-driven loss function, n1 represents the output of image X1 after f and g, n2 represents the output of image X2 after f, v1 represents the output of n1 after h, v2 represents the output of n2 after h, stopgrad(n2) means to stop calculating the gradient of vector n2 during training.

[0034] The specific process of training the IrisPer_YoloV3 object detection model in step 5 above to locate the iris and periorbital area is as follows:

[0035] The first step is to perform data enhancement on the iris and periorbital target detection dataset by preprocessing it. Each sample in the training set of the target detection dataset is randomly changed in brightness, contrast, color, padding, cropping, flipping, and rotation.

[0036] The second step is to mark the training labels of the target detection model. According to the downsampling scale of MobileNetV3, the single sample image in the training set is divided into several scale×scale grids. Three anchor frames are generated in each grid. The coordinates of the center point of the anchor frame are the same as the coordinates of the center point of the grid. The size of the anchor frame is as consistent as possible with the size of the iris and periocular real frame in the training sample. According to the position information of the iris and periocular real frame in the training sample, the periocular real frame and the iris real frame are traversed. Assume that the horizontal coordinate of the center point coordinate of the real frame is i ceneter , the horizontal axis is j ceneter , then its grid position is (i ceneter / scale,j ceneter / scale), use formula (2) to calculate the IOU of these three anchor boxes and the real box;

[0037]

[0038] Where: S truth represents the area of ​​the real box, S anchor represents the area of ​​the anchor box;

[0039] Take the anchor box with the largest IOU, and set the objectness label of the corresponding prediction box to 1. Objectness indicates the confidence, that is, the possibility that the prediction box has iris or periorbital area. For other anchor boxes in the grid, if its IOU is not the maximum value but exceeds the set threshold, the objectness training label of its corresponding prediction box is set to -1. For all other prediction boxes, their objectness training labels are 0. For the prediction box with objectness label 1, its position and size should be marked in the training label, and the vector (t x, t y , t h , t w ) indicates the position and size, t x is the central abscissa of the iris or periorbital area, t y is the central ordinate of the iris or eye periphery, t h is the height of the iris or eye periphery, t w is the width of the iris or eye periphery. The goal of training the network is to make the position and size of the output prediction box as close to the real box as possible. The position and size relationship between the prediction box and the real box must satisfy the following equations (3), (4), (5), (6):

[0040] σ(t x )+c x =gt x #(3)

[0041] σ(t y )+c y =gt y #(4)

[0042]

[0043]

[0044] Where: c x is the horizontal coordinate of the small grid where the prediction box is located, c y is the vertical coordinate of the small grid where the prediction box is located, a w is the width of the anchor box corresponding to the prediction box, a h is the height of the anchor box corresponding to the predicted box, gt x Represents the horizontal coordinate of the real box, gt y Represents the ordinate of the real box, gt y Represents the ordinate of the real box, gt w Indicates the width of the real box, gt h Indicates the height of the real frame;

[0045] The third step is to build the IrisPer_YoloV3 target detection model based on MobileNetV3. The detection model is composed of a three-branch convolutional detection head network added after the convolution part of MobileNetV3. Assume that the kernel size and number of the convolutional layer are (W kernal ×H kernal , N kernal ) indicates that W kernal Represents the width of the convolution kernel, H kernal Represents the height of the convolution kernel, N kernalRepresents the number of convolution kernels. The first branch is to add 7 convolution layers after the fifth downsampling convolution layer of MobileNetV3. The size and number of convolution kernels in each layer are (1×1, 512), (3×3, 1024), (1×1, 512), (3×3, 1024), (1×1, 512), (3×3, 1024), (1×1, 21). The second branch is to first pass the output of the fifth convolution layer of the first branch through a (1×1, 256) convolution layer and an upsampling operation, and then make a channel-wise feature map splicing with the output of the fourth downsampling of MobileNetV3. After the splicing operation, 7 convolution layers are added. The size and number of convolution kernels in each layer are (1×1, 256), (3×3, 512), (1×1, 256), (3×3, 512), (1×1, 256), (3×3, 512), (1×1, 21), the third branch is to first pass the output of the fifth convolutional layer of the first branch through a (1×1, 256) convolutional layer and an upsampling operation, and then concatenate the output of the third downsampling of MobileNetV3 with a feature map in the channel direction. After the concatenation operation, 7 convolutional layers are added, and the size and number of convolution kernels of each layer are (1×1, 128), (3×3, 256), (1×1, 128), (3×3, 256), (1×1, 128), (3×3, 256), (1×1, 21);

[0046] Step 4: Feed the data-enhanced training set from the first step into the IrisPer_YoloV3 target detection model based on MobileNetV3 established in the third step. According to the labeling rules of the training labels in the second step, freeze the convolution part of MobileNetV3 and perform supervised training. The three-branch target detection head constructed in the third step, since the prediction box output by the three-branch target detection head network has three scales, it is necessary to calculate the loss function for the prediction box corresponding to each scale, and then take their average. The target detection loss function for iris and periorbital is as follows (7) and (8):

[0047]

[0048] Loss mean =(Loss1+Loss2+Loss3) / 3#(8)

[0049] Among them, Loss k represents the loss of the k-th branch output feature map, where S k represents the size of the k-th branch output feature map, i represents the i+1-th grid, j represents the j+1-th prediction box, and λ coord Represents the weight coefficient of position loss, If the j+1 prediction box in the i+1th grid contains iris or periorbital area, the value is 1, otherwise it is 0. ij is the central horizontal coordinate of the iris or eye surrounding real frame, is the central horizontal coordinate of the prediction box, y ij is the central ordinate of the iris or eye surrounding real frame, is the central ordinate of the prediction box, w ij is the width of the real frame of the iris or eye, is the width of the prediction box, h ij is the height of the real frame of the iris or eye, is the height of the prediction box, c ij is the category of the real box, is the category of the prediction box, λ noobj Represents the weight coefficient of the category loss without objects, If there is no iris or periorbital area in the j+1 prediction box in the i+1 grid, the value is 1, otherwise it is 0. ij represents the confidence of the true box, Indicates the confidence of the predicted box.

[0050] The eye periphery feature extraction model E in the above steps 6 and 7 per and iris feature extraction model E iris The specific process of training is as follows:

[0051] The first step is to expand the data of the periocular visible light image training set and the iris membrane visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, random rotation, random adjustment of brightness, hue, saturation and contrast. The size of each image in the expanded periocular image training set and each image in the expanded iris image training set is adjusted to 224×224.

[0052] Step 2: A three-layer fully connected neural network is constructed for the periocular features and iris features as a multi-layer perceptron, with 960 neurons in the input layer, 1280 neurons in the hidden layer, and k neurons in the output layer corresponding to the total number of identity labels in the periocular and iris training sets;

[0053] Step 3: Use the expanded periocular visible light image training set or the iris visible light image training set as input to the model for supervised training. The training loss function is formula (9);

[0054]

[0055] Among them: when modal is iris, it means that the loss function is used to train the iris feature extractor, when modal is per, it means that the loss function is used to train the periocular feature extractor. represents the cross entropy loss of the classification model for training the periorbital or iris, n is the number of BatchSize, is the soft output of the jth neuron after the i-th periocular image or iris image passes through the last layer of the model, is the jth value in the Onehot encoding of the identity label of the current input periocular image or iris image.

[0056] The specific process of adversarial training in step eight above is as follows:

[0057] The first step is to construct a classification and discrimination integrator neural network, an iris feature extraction network, and an eye periphery feature extraction network to form an adversarial modality adaptive architecture, where the integrated neural network is a fully connected neural network with an input layer of 1280 neurons and an output layer of k+1 neurons;

[0058] The second step is to label the samples from the iris visible light training set as in represents the i-th iris image in the training set, Represents the identity label in the form of a Onehot vector corresponding to the i-th iris image, n iris Represents the total number of samples in the iris training set. Similarly, the images from the periocular visible light training set are labeled Modal labels are set for different modalities of iris and periorbital. The modal label of samples from iris is 0, and the modal label of samples from periorbital is 1.

[0059] Step 3: Input the iris sample into the iris feature extractor E iris , input the eye periphery sample into the feature extractor E per , input the obtained iris and eye periphery feature vectors into the classification and discrimination integrator C to obtain the classification output vector, and use express That is, the output vector of the classification and discrimination integrator, where item = iris, per, Represents the output vector of the classifier and the discriminant ensemble The k-th value of , where k = 1, 2, ..., K, calculates the conditional probability of the output vector, and the k-th element value of the conditional probability is as shown in formula (10):

[0060]

[0061] in: express The k′th element value of the modal prediction vector of the kth class identity label is calculated by the kth element value of the output conditional probability of the classifier and the discriminant integrator as shown in formula (11):

[0062]

[0063] in: express The k′th element value of the modal prediction vector of the kth class identity label output by the classifier and the discriminant ensemble;

[0064] Step 4: In order to use adversarial training to align the feature distribution of the iris and periocular modalities, we train the classification and discrimination integrator to correctly classify the identity labels of the iris and periocular, and correctly classify the modal labels of the iris and periocular. When training the iris feature extractor and the periocular feature extractor, the generated modal features make it impossible for the classification and discrimination integrator to distinguish the modal labels, so as to achieve the purpose of aligning the feature distribution of the iris and periocular modalities. The loss function L associated with the iris feature extractor is calculated. iris (E iris , C),

[0065] Loss function L iris (E iris , C) is designed to ensure that the iris feature extractor has a mutually antagonistic and inhibiting relationship between the prediction of the iris category and the prediction of the modality type. The loss function is shown in formula (12):

[0066]

[0067] Where: L iris (E iris , C) represents the rainbow model feature extractor E iris of confrontation losses;

[0068] Loss Function The design of is to ensure that the type of feature modality extracted by the iris feature extractor remains discriminative while ensuring the discriminability of the iris category of the feature vector, so as to achieve the purpose of joint probability distribution registration of iris category and modality prediction. The loss function is shown in formula (13):

[0069]

[0070] in: represents the joint probabilistic registration loss of the iris feature extractor;

[0071] Compute the joint probabilistic registration loss of the classification and ensemble discriminator views associated with the iris data The formula is shown in (14):

[0072]

[0073] Calculate the loss function L associated with the periocular feature extractor per(E per , C),

[0074] Loss function L per (E per , C) is designed to ensure a mutually antagonistic inhibition relationship between the prediction of the periocular category and the prediction of the modality type by the periocular feature extractor. The loss function is shown in formula (15):

[0075]

[0076] Where: L per (E per , C) represents the eye periphery feature extractor E per of confrontation losses;

[0077] Loss Function The design of is to ensure that the type of feature modality extracted by the periocular feature extractor remains discriminative while ensuring the discriminability of the periocular category of the feature vector, so as to achieve the purpose of joint probability distribution registration of the periocular category and modality prediction. The loss function is shown in formula (16):

[0078]

[0079] in: represents the joint probabilistic registration loss of the periocular feature extractor;

[0080] Compute the joint probabilistic registration loss of the classification and ensemble discriminator viewpoints associated with the periocular data The formula is shown in (17):

[0081]

[0082] In order to estimate the clustering with lower error, a minimization entropy principle is calculated as shown in formula (18):

[0083]

[0084] in: represents the minimum entropy loss of the iris or periorbital area;

[0085] Step 5: Unfreeze the classification and discrimination integrator to make the weights learnable, freeze the iris feature extractor to make the weights unlearnable, and freeze the periorbital feature extractor to make the weights unlearnable. Training classification and discrimination ensembles, Indicates training C so that Take the minimum value;

[0086] Step 6: Freeze the classification and discrimination integrators to make the weights unlearnable, unfreeze the iris feature extractor to make the weights learnable, and unfreeze the periorbital feature extractor to make the weights learnable. Train an iris feature extractor, where Indicates training E iris make Take the maximum value to learn the goal Train the periocular feature extractor, where Indicates training E per make Take the maximum value;

[0087] Step 7: Repeat steps 3, 4, 5, and 6 until the model converges or the maximum number of cycles is reached.

[0088] The specific process of step nine above is as follows:

[0089] The first step is to expand the data of the iris visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After the expansion, the size of each image in the iris visible light image training set is adjusted to 224×224. The data expansion of the periocular visible light image training set is carried out. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After the expansion, the size of each image in the periocular visible light image training set is adjusted to 224×224.

[0090] The second step is to construct an iris and periorbital fusion classifier F. The fusion classifier is a three-layer perceptron. The number of neurons in the input layer is 2560, the number of neurons in the hidden layer is 2560, and the number of neurons in the output layer is K.

[0091] The third step is to input the expanded iris visible light image training set and the expanded periocular visible light training set into the iris feature extractor and the periocular feature extractor respectively. The output of the iris feature extractor is the iris feature vector, and the output of the periocular extractor is the periocular feature vector. Two learnable parameters are set as the contribution weight coefficients of the iris feature vector and the periocular feature vector respectively. The iris feature vector and the periocular feature vector are multiplied by their respective weight coefficients respectively, and then the two vectors are spliced ​​in the feature direction to obtain a 2560-dimensional fusion feature vector. The entire fusion process can be expressed as V fusion =[γ iris V iris , γ per V per ], where V fusion represents the final fusion feature vector of iris and periorbital area, γ irisrepresents the iris contribution trade-off coefficient, V iris represents the iris feature vector output by the iris feature extractor, γ per represents the periocular contribution trade-off coefficient, V per represents the periocular feature vector output by the periocular feature extractor;

[0092] Step 4: Input the fused feature vector into the iris and periocular fusion classifier F, calculate the cross entropy loss, freeze the iris feature extractor and the periocular feature extractor, and perform error back propagation training to update the iris and periocular fusion classifier F and the trade-off coefficient γ per and γ iris .

[0093] The specific process of training the feature extractor of the iris and periorbital area detection and fusion recognition model in the above step 10 is as follows:

[0094] The first step is to expand the data of the eye visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After expansion, the size of each image in the eye visible light image training set is adjusted to 224×224;

[0095] Step 2: Input the visible light image of the eye into the lightweight IrisPer_YoloV3 iris and periocular target detection model. The detection model outputs the coordinates of the iris region and the periocular region. The iris region image and the periocular region image are cropped out by the coordinates. The iris image and the periocular region image are resized to 224×224. A fusion recognition sample is constructed. The fusion recognition sample consists of the cropped iris image, the periocular image and the identity label. The fusion recognition sample is input into the iris and periocular feature extraction and feature fusion classification model. The output of the fusion classification model is recorded as r.

[0096] Step 3: Calculate the loss function L of the fusion classification fusion , freeze the iris and eye area detection model IrisPer_YoloV3, minimize the loss L fusion For the purpose of training the fusion classification model F, the loss function of the fusion classification is shown in formula (19):

[0097]

[0098] Where: n eye represents the total number of fusion recognition samples, r j Represents the jth element value of the soft output vector of the fusion classification model, t j Represents the j-th element value of the identity label Onehot vector of the fused recognition sample.

[0099] The specific process of the multi-stage decision-making in step 11 above is as follows:

[0100] The first step is to register. A random image in each category of the eye visible light image test set is extracted through the iris and periorbital dual-modal recognition model to obtain the iris and periorbital fusion feature vector. All feature vectors are stored in the feature vector database and the identity category information of the feature vector is marked.

[0101] The second step is to perform first-order authentication. In the eye light image test set, a random image is selected and extracted through the iris and periocular dual-modal recognition model to obtain the iris and periocular fusion feature vector. The feature vector and the iris and periocular fusion feature vectors registered in the database are calculated one by one by the Euclidean distance decision matching function, and the iris and periocular fusion feature vector with the smallest distance to the feature vector to be authenticated is selected. The calculation formula of the Euclidean decision matching function is (20), as shown below:

[0102]

[0103] Where: euclideandistance(X prob , Y gallery ) indicates X prob With Y gallery The Euclidean distance between prob Represents the iris and eye periphery fusion feature vector to be authenticated, Represents X prob The i-th element in Y gallery represents the fusion feature vector of iris and periorbital area in the registered database, Y gallery represent The i-th element in , L represents the dimension of the feature vector;

[0104] The third step is to perform second-order authentication, compare the Euclidean distance between the feature vector to be authenticated and the feature vector registered in the database, select the registered feature vector with the smallest Euclidean distance, and calculate the cosine similarity between the feature vector to be authenticated and the registered feature vector with the smallest Euclidean distance. If the cosine similarity value is greater than the threshold value 0.7, it is considered that the category of the feature vector to be identified is the same as the category of the iris and periorbital feature vector in the registration library. Otherwise, the identity category of the feature vector to be identified is not in the system database and the identification fails. The calculation formula (21) of the cosine similarity matching decision function is:

[0105]

[0106] Where: cosinedistance(X prob , Y chosen ) indicates X prob With Y chosen The cosine similarity between chosenIndicates that the second step selects prob The eigenvector with the smallest Euclidean distance is Represents Y chosen The value of the i-th element of .

[0107] Beneficial effects of the present invention:

[0108] The iris and periocular adversarial adaptive fusion recognition method based on contrast knowledge driven provided by the present invention fuses the new biometric periocular features with the iris features, and avoids the stringent requirements for acquisition equipment by utilizing the advantages of the larger recognizable size of the periocular and the high accuracy of the small-sized iris. At the same time, the multimodal biometric fusion improves the reliability and security of the system.

[0109] The present invention designs a multi-stage training algorithm based on contrast knowledge-driven and data-driven. First, contrast training is used to fully explore the spatial relationship of discriminable feature encoding of iris and periocular datasets, and then fine-tunes on the downstream tasks of iris and periocular area detection and iris and periocular classification. This effectively avoids the overfitting problem of training deep neural networks when the number of eye biometric datasets is limited, and improves the recognition performance on small sample datasets.

[0110] The present invention designs a method for modal adversarial adaptive fusion, which uses adversarial learning to align the feature distributions of different biological modalities in a low-dimensional feature subspace, and fuses the aligned feature vectors to improve the final recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 The figure is a flow chart of the identification method described in the present invention. DETAILED DESCRIPTION

[0112] Please refer to the flowchart of the present invention Figure 1 As shown:

[0113] Embodiment 1:

[0114] The whole process of operation on a certain person (named Tom, Tom's information has not been registered before, the test eye image and the eye image in the database for generating the iris and eye periorbital fusion feature vector are performed using the same iris acquisition device) under the framework of right 1:

[0115] 1) Use any high-definition image acquisition device on the market to collect eye images of 100 people for model training, 15 images per person, and a total of 1,500 eye images for training, to establish a training dataset of original eye visible light images, a training dataset of iris visible light, and a training dataset of periocular visible light.

[0116] 2) Initialize three MobileNetV3 deep convolutional neural networks. For each network model, remove the last fully connected layer of MobileNetV3 as the encoding layer, add a projection layer after the encoding layer, and add a prediction layer after the projection layer to obtain three initialized model architectures;

[0117] 3) Using the raw visible light image training set collected from 1), a contrast knowledge-driven learning algorithm is used to train the three initialized model architectures obtained from 2).

[0118] 4) Remove the projection layers and prediction layers of the three model architectures to obtain the convolutional coding parts of the three trained MobileNetV3s.

[0119] 5) In 4), the convolutional coding part of MobileNetV3 obtained by knowledge-driven training is used as the backbone to build the IrisPer_YoloV3 target detection model, freeze the convolutional coding part of MobileNetV3, and use the iris and periocular target detection training set to train the IrisPer_YoloV3 target detection model to locate the iris and periocular area.

[0120] 6) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Freeze the MobileNetV3 convolutional coding part, use the periocular visible light image training set and person identity label training, remove the last fully connected layer to obtain the periocular feature extraction model E per .

[0121] 7) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Use the iris visible light image training set and the person identity label training to remove the last fully connected layer to obtain the iris feature extraction model E iris

[0122] 8) Construct a fully connected neural network C that integrates the classification and discriminator. The iris feature extraction model E obtained in 6) and 7) iris and eye contour extraction model E per Together with the fully connected neural network C integrated with the classifier and discriminator, the iris visible light training set and the periocular visible light training set are used with the identity category labels for adversarial training, aiming to align the feature distributions of different modalities in a low-dimensional subspace.

[0123] 9) Set two learnable parameters, use the iris extractor and periocular feature extractor obtained through adversarial training in 8), add a multi-layer perceptron as the fusion classifier F to build the iris and periocular fusion model, freeze the iris feature extraction model and the periocular feature extraction model, use the iris visible light training set and the periocular visible light training set to train the entire fusion recognition model, and optimize the multi-layer perceptron of the fusion classifier F.

[0124] 10) The iris and periocular fusion model obtained in 9) is jointly trained with the iris and periocular synchronization detection model IrisPer_YoloV3 target detection model obtained in 5), and the last layer of the neural network is removed to obtain the final iris and periocular area detection and fusion recognition model feature extractor.

[0125] 11) Take any eye image of each person in the eye dataset among the 100 people in the test set, for a total of 100 images.

[0126] 12) Input the 100 images in 11) into the feature extractor of the iris and periorbital area detection and fusion recognition model obtained in 10), output 100 template feature vectors, and store them in the database.

[0127] 13) Use the acquisition device to collect Tom's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), output Tom's template feature vector, and store it in the database.

[0128] 14) Use the collector to collect Tom's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), and output Tom's test feature vector.

[0129] 15) A two-stage decision strategy is used to compare the test feature vector to be authenticated with the template feature vectors in the database one by one, and the template feature vector with the minimum Euclidean matching decision function value calculated with the test feature vector is screened out. The cosine matching decision function value is calculated between the template feature vector and the test feature vector. If the cosine matching decision function value is greater than the threshold value of 0.7, the template feature vector successfully matches the test feature vector.

[0130] 16) The identity label of the template feature vector is Tom, so we can conclude that the template feature vector and the test feature vector are of the same category and both belong to Tom.

[0131] Example 2:

[0132] The whole process of operation on two persons (named Betty and John, whose information has not been registered before, and the test eye image and the eye image in the database for generating the iris and periorbital fusion feature vector are performed using the same iris acquisition device) under the framework of right 1:

[0133] 1) Use any high-definition image acquisition device on the market to collect eye images of 100 people for model training, 15 images per person, and a total of 1,500 eye images for training, to establish a training dataset of original eye visible light images, a training dataset of iris visible light, and a training dataset of periocular visible light.

[0134] 2) Initialize three MobileNetV3 deep convolutional neural networks. For each network model, remove the last fully connected layer of MobileNetV3 as the encoding layer, add a projection layer after the encoding layer, and add a prediction layer after the projection layer to obtain three initialized model architectures;

[0135] 3) Using the raw visible light image training set collected from 1), a contrast knowledge-driven learning algorithm is used to train the three initialized model architectures obtained from 2).

[0136] 4) Remove the projection layers and prediction layers of the three model architectures to obtain the convolutional coding parts of the three trained MobileNetV3s.

[0137] 5) In 4), the convolutional coding part of MobileNetV3 obtained by knowledge-driven training is used as the backbone to build the IrisPer_YoloV3 target detection model, freeze the convolutional coding part of MobileNetV3, and use the iris and periocular target detection training set to train the IrisPer_YoloV3 target detection model to locate the iris and periocular area.

[0138] 6) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Freeze the MobileNetV3 convolutional coding part, use the periocular visible light image training set and person identity label training, remove the last fully connected layer to obtain the periocular feature extraction model E per .

[0139] 7) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Use the iris visible light image training set and the person identity label training to remove the last fully connected layer to obtain the iris feature extraction model E iris

[0140] 8) Construct a fully connected neural network C that integrates the classification and discriminator. The iris feature extraction model E obtained in 6) and 7) iris and eye contour extraction model E perTogether with the fully connected neural network C integrated with the classifier and discriminator, the iris visible light training set and the periocular visible light training set are used with the identity category labels for adversarial training, aiming to align the feature distributions of different modalities in a low-dimensional subspace.

[0141] 9) Set two learnable parameters, use the iris extractor and periocular feature extractor obtained through adversarial training in 8), add a multi-layer perceptron as the fusion classifier F to build the iris and periocular fusion model, freeze the iris feature extraction model and the periocular feature extraction model, use the iris visible light training set and the periocular visible light training set to train the entire fusion recognition model, and optimize the multi-layer perceptron of the fusion classifier F.

[0142] 10) The iris and periocular fusion model obtained in 9) is jointly trained with the iris and periocular synchronization detection model IrisPer_YoloV3 target detection model obtained in 5), and the last layer of the neural network is removed to obtain the final iris and periocular area detection and fusion recognition model feature extractor.

[0143] 11) Take any eye image of each person in the eye dataset of the 100 people in the test set, a total of 100 images

[0144] 12) Input the 100 images in 11) into the feature extractor of the iris and periorbital area detection and fusion recognition model obtained in 10), output 100 template feature vectors, and store them in the database.

[0145] 13) Use a data acquisition device to collect Betty's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), output Betty's template feature vector, and store it in the database.

[0146] 14) Use the acquisition device to collect John's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), and output John's test feature vector.

[0147] 15) A two-stage decision strategy is used to compare the test feature vector to be authenticated with the template feature vectors in the database one by one, and the template feature vector with the minimum Euclidean matching decision function value calculated with the test feature vector is screened out. The cosine matching decision function value is calculated between the template feature vector and the test feature vector. If the cosine matching decision function value is less than a threshold, the template feature vector fails to match the test feature vector.

[0148] 16) There is no template feature vector in the database that satisfies the minimum Euclidean matching decision function value with the test feature vector and the cosine matching function value is greater than the threshold 0.7. Therefore, it is concluded that the template feature vector and the test feature vector are of different categories, and Betty's template feature vector and John's test feature vector are not of the same category.

[0149] Example 3:

[0150] The whole process of operation on two persons (named Martin and Robert, whose information has not been registered before, and the test eye image and the eye image in the database for generating the iris and periorbital fusion feature vector) under the framework of right 1 are as follows:

[0151] 1) Use any high-definition image acquisition device on the market to collect eye images of 100 people for model training, 15 images per person, and a total of 1,500 eye images for training, to establish a training dataset of original eye visible light images, a training dataset of iris visible light, and a training dataset of periocular visible light.

[0152] 2) Initialize three MobileNetV3 deep convolutional neural networks. For each network model, remove the last fully connected layer of MobileNetV3 as the encoding layer, add a projection layer after the encoding layer, and add a prediction layer after the projection layer to obtain three initialized model architectures;

[0153] 3) Using the raw visible light image training set collected from 1), a contrast knowledge-driven learning algorithm is used to train the three initialized model architectures obtained from 2).

[0154] 4) Remove the projection layers and prediction layers of the three model architectures to obtain the convolutional coding parts of the three trained MobileNetV3s.

[0155] 5) In 4), the convolutional coding part of MobileNetV3 obtained by knowledge-driven training is used as the backbone to build the IrisPer_YoloV3 target detection model, freeze the convolutional coding part of MobileNetV3, and use the iris and periocular target detection training set to train the IrisPer_YoloV3 target detection model to locate the iris and periocular area.

[0156] 6) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Freeze the MobileNetV3 convolutional coding part, use the periocular visible light image training set and person identity label training, remove the last fully connected layer to obtain the periocular feature extraction model E per .

[0157] 7) Add a multi-layer perceptron as a classifier after the MobileNetV3 convolutional coding part obtained by the comparative knowledge-driven training in 4). Use the iris visible light image training set and the person identity label training to remove the last fully connected layer to obtain the iris feature extraction model Eiris

[0158] 8) Construct a fully connected neural network C that integrates the classification and discriminator. The iris feature extraction model E obtained in 6) and 7) iris and eye contour extraction model E per Together with the fully connected neural network C integrated with the classifier and discriminator, the iris visible light training set and the periocular visible light training set are used with the identity category labels for adversarial training, aiming to align the feature distributions of different modalities in a low-dimensional subspace.

[0159] 9) Set two learnable parameters, use the iris extractor and periocular feature extractor obtained through adversarial training in 8), add a multi-layer perceptron as the fusion classifier F to build the iris and periocular fusion model, freeze the iris feature extraction model and the periocular feature extraction model, use the iris visible light training set and the periocular visible light training set to train the entire fusion recognition model, and optimize the multi-layer perceptron of the fusion classifier F.

[0160] 10) The iris and periocular fusion model obtained in 9) is jointly trained with the iris and periocular synchronization detection model IrisPer_YoloV3 target detection model obtained in 5), and the last layer of the neural network is removed to obtain the final iris and periocular area detection and fusion recognition model feature extractor.

[0161] 11) Take any eye image of each person in the eye dataset of the 100 people in the test set, a total of 100 images

[0162] 12) Input the 100 images in 11) into the feature extractor of the iris and periorbital area detection and fusion recognition model obtained in 10), output 100 template feature vectors, and store them in the database.

[0163] 13) Use a data acquisition device to collect Martin's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), output Martin's template feature vector, and store it in the database.

[0164] 14) Use a data acquisition device to collect Robert's eye image and input it into the iris and eye area detection and fusion recognition model extractor obtained in 10), and output Robert's test feature vector.

[0165] 15) A two-stage decision strategy is used to compare the test feature vector to be authenticated with the template feature vectors in the database one by one, and the template feature vector with the minimum Euclidean matching decision function value calculated with the test feature vector is screened out. The cosine matching decision function value is calculated between the template feature vector and the test feature vector. If the cosine matching decision function value is less than a threshold, the template feature vector fails to match the test feature vector.

[0166] 16) There is no template feature vector in the database that satisfies the minimum Euclidean matching decision function value with the test feature vector and the cosine matching function value is greater than the threshold 0.7. Therefore, it is concluded that the template feature vector and the test feature vector are of different categories, and the Martin template feature vector and the Robert test feature vector are not of the same category.

Claims

1. An iris and periorbital adversarial adaptive fusion recognition method driven by contrast knowledge, characterized by: The method comprises the following steps: Step 1: Establish eye visible light image dataset, iris and periocular target detection dataset, iris visible light image dataset and periocular visible light image dataset, and divide them into training set and test set. The training set is used to train the iris and periocular area detection network model and the iris and periocular feature fusion model, and the test set is used to evaluate the accuracy of the model. Step 2: Initialize three MobileNetV3 deep convolutional neural networks. For each network model, remove the last fully connected layer of MobileNetV3 as the encoding layer, add a projection layer after the encoding layer, and add a prediction layer after the projection layer to obtain three initialized model architectures. Step 3: Use the eye visible light image training set collected in step 1 to train the three initialization model architectures obtained in step 2 using a comparative knowledge-driven algorithm; Step 4: Remove the projection layer and prediction layer of the three trained model architectures to obtain the convolutional coding parts of the three trained MobileNetV3s; Step 5: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4 and use it as the backbone to build the IrisPer_YoloV3 target detection model, freeze the MobileNetV3 convolutional coding part, and use the iris and periorbital target detection training set to train the IrisPer_YoloV3 target detection model to locate the iris and periorbital area; Step 6: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4, add a multi-layer perceptron as a classifier, freeze the MobileNetV3 convolutional coding part, use the periocular visible light image training set and personnel identity label training, and remove the last fully connected layer after the training to obtain the periocular feature extraction model E. per ; Step 7: Compare the MobileNetV3 convolutional coding part obtained by knowledge-driven training in step 4, add a multi-layer perceptron as a classifier, freeze the MobileNetV3 convolutional coding part, use the iris visible light image training set and personnel identity label training, and remove the last fully connected layer after the training to obtain the iris feature extraction model E. iris ; Step 8: Build a fully connected neural network C that integrates the classifier and discriminator, and use the iris feature extraction model E obtained in steps 6 and 7 iris and eye contour extraction model E per Together with the fully connected neural network C integrated with the classifier and discriminator, the iris visible light training set and the eye periocular visible light training set are used for joint adversarial training with the identity category labels, with the aim of aligning the feature distributions of different modalities in a low-dimensional subspace. Step 9: Set two learnable parameters and use the iris feature extractor E obtained through adversarial training in step 8. iris and the eye periphery feature extractor E per , then add a multi-layer perceptron as the fusion classifier F to build the iris and periorbital fusion model, freeze the iris feature extraction model and the periorbital feature extraction model, use the iris visible light training set and the periorbital visible light training set to train the entire fusion recognition model, and optimize the multi-layer perceptron of the fusion classifier F; Step 10: The iris and periocular fusion model obtained in step 9 is jointly trained with the iris and periocular synchronous detection model IrisPer_YoloV3 target detection model obtained in step 5, and the last layer of the neural network is removed to obtain the final iris and periocular area detection and fusion recognition model feature extractor; Step 11: Use a multi-stage decision strategy collaborative solution as the decision mechanism of the trained iris and periorbital dual-modal fusion recognition model.

2. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of establishing the eye visible light image dataset, the iris and periorbital target detection dataset, the iris visible light image dataset, and the periorbital visible light image dataset in the above step 1 is as follows: The first step is to use a high-definition camera to collect 2,500 visible light images of the template tester's template eyes, of which 2,000 are divided into a training set and 500 are divided into a test set, as the visible light image dataset of the eyes; Step 2: Use the labelme tool to annotate the iris area and periocular area of ​​each sample in the eye visible light image training set and test set to obtain the target detection position label of the iris area and periocular area. The label is combined with the eye visible light image dataset to obtain the iris and periocular target detection dataset. The third step is to crop each sample in the eye visible light image dataset to obtain iris visible light images and periocular visible light images according to the labels in the iris and periocular target detection dataset obtained in the second step, and assign an identity category label to each image. The identity labels and iris visible light images and periocular visible light images construct the iris visible light dataset and periocular visible light dataset respectively.

3. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of constructing the projection layer and prediction layer in the above step 2 is as follows: The first step is to build a 3-layer multi-layer perceptron as the projection layer. Each fully connected layer has 1024 neurons, followed by a BN layer or batch normalization layer; The second step is to build a 2-layer multilayer perceptron as the prediction layer. The prediction layer is a 2-layer multilayer perceptron. Except for the output layer, other layers are followed by a BN layer. The dimension of the hidden layer is 256, and the dimension of other layers is 1024.

4. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The comparative knowledge-driven training process in step 3 above is as follows: The first step is to perform random data enhancement on samples in the eye visible light image training set data. The eye visible light image is randomly cropped according to the ratio of the original image size [0.5, 2.0], and then the image size is adjusted using the bilinear interpolation algorithm. Then, the image is randomly flipped left and right, and the image contrast, brightness, hue and saturation are randomly adjusted. Then, the image is randomly grayed, and finally, the image is randomly blurred using the mean filter, Gaussian blur, median blur, bilateral filter and box blur. In the second step, the random enhancement operation in the first step is used to perform two random enhancement operations on each image X to obtain images X1 and X2. The two images X1 and X2 are respectively input into a convolutional coding part f, and the output is expressed as m1=f(X1) and m2=f(X2). m1 and m2 are input into the projection layer g, and the output is expressed as n1=g(m1) and n2=g(m2). n1 and n2 are input into the prediction layer h, and the output is expressed as v1=h(n1) and v2=h(n2). The architecture idea of ​​the twin neural network is adopted, and the weights of f, g, and h are shared to construct a comparative knowledge-driven training paradigm. Step 3: Take the eye visible light image training set with a BatchSize of 512, the optimizer is SGD, the initial learning rate is 0.1, and the contrast knowledge driven learning architecture is input for training. The contrast knowledge driven loss function is formula (1); Where: L contrast It represents the contrast knowledge driven loss function, n1 represents the output of image X1 after passing through f and g, n2 represents the output of image X2 after passing through f, v1 represents the output of n1 after passing through h, v2 represents the output of n2 after passing through h, and stopgrad(n2) means to stop calculating the gradient of vector n2 during the training process.

5. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of training the IrisPer_YoloV3 object detection model in step 5 above to locate the iris and periocular area is as follows: The first step is to perform data enhancement on the iris and periorbital target detection dataset by preprocessing it. Each sample in the training set of the target detection dataset is randomly changed in brightness, contrast, color, padding, cropping, flipping, and rotation. The second step is to mark the training labels of the target detection model. According to the downsampling scale of MobileNetV3, the single sample image in the training set is divided into several scale×scale grids. Three anchor frames are generated in each grid. The coordinates of the center point of the anchor frame are the same as the coordinates of the center point of the grid. The size of the anchor frame is as consistent as possible with the size of the iris and periocular real frame in the training sample. According to the position information of the iris and periocular real frame in the training sample, the periocular real frame and the iris real frame are traversed. Assume that the horizontal coordinate of the center point coordinate of the real frame is i ceneter , the horizontal axis is j ceneter , then its grid position is (i ceneter / scale,j ceneter / scale), use formula (2) to calculate the IOU of these three anchor boxes and the real box; Where: S truth Represents the area of ​​the real box, S anchor represents the area of ​​the anchor box; Take the anchor box with the largest IOU, and set the objectness label of the corresponding prediction box to 1. Objectness indicates the confidence, that is, the possibility that the prediction box has the iris or eye periphery. For other anchor boxes in the grid, if its IOU is not the maximum value but exceeds the set threshold, the objectness training label of its corresponding prediction box is set to -1. For all other prediction boxes, their objectness training labels are 0. For the prediction box with objectness marked as 1, its position and size should be marked in the training label, and the vector (t x , t y , t h , t w ) indicates the position and size, t x is the central abscissa of the iris or periorbital area, t y is the central ordinate of the iris or eye periphery, t h is the height of the iris or eye periphery, t w is the width of the iris or eye periphery. The goal of training the network is to make the position and size of the output prediction box as close as possible to the real box. The position and size relationship between the prediction box and the real box must satisfy the following equations (3), (4), (5), (6): σ(t x )+c x =gt x #(3) σ(t y )+c y =gt y #(4) Where: c x is the horizontal coordinate of the small grid where the prediction box is located, c y is the vertical coordinate of the small grid where the prediction box is located, a w is the width of the anchor box corresponding to the prediction box, a h is the height of the anchor box corresponding to the prediction box, gr x Represents the horizontal coordinate of the real box, gt y Represents the ordinate of the real box, gt y Represents the ordinate of the real box, gt w Indicates the width of the real box, gt h Indicates the height of the real frame; The third step is to build the IrisPer_YoloV3 target detection model based on MobileNetV3. The detection model is composed of a three-branch convolutional detection head network added after the convolution part of MobileNetV3. Assume that the kernel size and number of the convolutional layer are (W kernal ×H kernal , N kernal ) indicates that W kernal Represents the width of the convolution kernel, H kernal Represents the height of the convolution kernel, N kernal Represents the number of convolution kernels. The first branch is to add 7 convolution layers after the fifth downsampling convolution layer of MobileNetV3. The size and number of convolution kernels of each layer are (1×1, 512), (3×3, 1024), (1×1, 512), (3×3, 1024), (1×1, 512), (3×3, 1024), (1×1, 21). The second branch is to first pass the output of the fifth convolution layer of the first branch through a (1×1, 256) convolution layer and an upsampling operation, and then splice the output of the fourth downsampling of MobileNetV3 with a feature map in the channel direction. After the splicing operation, 7 convolution layers are added. The size and number of convolution kernels of each layer are (1 ×1, 256), (3×3, 512), (1×1, 256), (3×3, 512), (1×1, 256), (3×3, 512), (1×1, 21). The third branch is to first pass the output of the fifth convolutional layer of the first branch through a (1×1, 256) convolution layer and an upsampling operation, and the output of the third downsampling of MobileNetV3 to make a channel-wise feature map splicing. After the splicing operation, 7 convolutional layers are added. The size and number of convolution kernels in each layer are (1×1, 128), (3×3, 256), (1×1, 128), (3×3, 256), (1×1, 128), (3×3, 256), (1×1, 21); Step 4: Feed the data-enhanced training set from the first step into the IrisPer_YoloV3 target detection model based on MobileNetV3 established in the third step. According to the labeling rules of the training labels in the second step, freeze the convolution part of MobileNetV3 and perform supervised training. The three-branch target detection head constructed in the third step, since the prediction box output by the three-branch target detection head network has three scales, it is necessary to calculate the loss function for the prediction box corresponding to each scale, and then take their average. The target detection loss function for iris and periorbital is as follows (7) and (8): Loss mean =(Loss1+Loss2+Loss3) / 3#(8) Among them, Loss k represents the loss of the k-th branch output feature map, where S k represents the size of the k-th branch output feature map, i represents the i+1-th grid, j represents the j+1-th prediction box, and λ coord Represents the weight coefficient of position loss, If the j+1 prediction box in the i+1th grid contains iris or periorbital area, the value is 1, otherwise it is 0. ij is the central horizontal coordinate of the iris or eye surrounding real frame, is the central horizontal coordinate of the prediction box, y ij is the central ordinate of the iris or eye surrounding real frame, is the central ordinate of the prediction box, w ij is the width of the real frame of the iris or eye, is the width of the prediction box, h ij is the height of the real frame of the iris or eye, is the height of the prediction box, c ij is the category of the real box, is the category of the prediction box, λ noobj Represents the weight coefficient of the category loss without objects, If there is no iris or periorbital area in the j+1 prediction box in the i+1 grid, the value is 1, otherwise it is 0. ij represents the confidence of the real box, Indicates the confidence of the predicted box.

6. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The eye periphery feature extraction model E in the above steps 6 and 7 per and iris feature extraction model E iris The specific process of training is as follows: The first step is to expand the data of the periocular visible light image training set and the iris membrane visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, random rotation, random adjustment of brightness, hue, saturation and contrast. The size of each image in the expanded periocular image training set and each image in the expanded iris image training set is adjusted to 224×224. The second step is to build a three-layer fully connected neural network as a multi-layer perceptron for periocular features and iris features, with 960 neurons in the input layer, 1280 neurons in the hidden layer, and k neurons in the output layer corresponding to the total number of identity labels in the periocular and iris training sets; Step 3: Use the expanded periocular visible light image training set or the iris visible light image training set as input to the model for supervised training. The training loss function is formula (9); Among them: when modal is iris, it means that the loss function is used to train the iris feature extractor, when modal is per, it means that the loss function is used to train the periocular feature extractor. represents the cross entropy loss of the classification model for training the periorbital or iris, n is the number of BatchSize, is the soft output of the jth neuron after the i-th periocular image or iris image passes through the last layer of the model, is the jth value in the Onehot encoding of the identity label of the current input periocular image or iris image.

7. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of adversarial training in step eight above is as follows: The first step is to construct a classification and discrimination integrator neural network, an iris feature extraction network, and an eye periphery feature extraction network to form an adversarial modality adaptive architecture, where the integrated neural network is a fully connected neural network with an input layer of 1280 neurons and an output layer of k+1 neurons; The second step is to label the samples from the iris visible light training set as in represents the i-th iris image in the training set, Represents the identity label in the form of a Onehot vector corresponding to the i-th iris image, n iris Represents the total number of samples in the iris training set. Similarly, the images from the periocular visible light training set are labeled Modal labels are set for different modalities of iris and periorbital. The modal label of samples from iris is 0, and the modal label of samples from periorbital is 1. Step 3: Input the iris sample into the iris feature extractor E iris , input the eye periphery sample into the feature extractor E per , input the obtained iris and eye periphery feature vectors into the classification and discrimination integrator C to obtain the classification output vector, and use express That is, the output vector of the classification and discrimination integrator, where item = iris, per, Represents the output vector of the classifier and the discriminant ensemble The k-th value of , where k = 1, 2, ..., K, calculates the conditional probability of the output vector, and the k-th element value of the conditional probability is as shown in formula (10): in: express The k′th element value of the modal prediction vector of the kth class identity label is calculated by the kth element value of the output conditional probability of the classifier and the discriminant integrator as shown in formula (11): in: express The k′th element value of the modal prediction vector of the kth class identity label output by the classifier and the discriminant ensemble; Step 4: In order to use adversarial training to align the feature distribution of the iris and periocular modalities, we train the classification and discrimination integrator to correctly classify the identity labels of the iris and periocular, and correctly classify the modal labels of the iris and periocular. When training the iris feature extractor and the periocular feature extractor, the generated modal features make it impossible for the classification and discrimination integrator to distinguish the modal labels, so as to achieve the purpose of aligning the feature distribution of the iris and periocular modalities. The loss function L associated with the iris feature extractor is calculated. iris (E iris ,C) Loss function L iris (E iris ,C) is designed to ensure a mutually antagonistic and inhibitory relationship between the prediction of the iris category and the prediction of the modality type by the iris feature extractor. The loss function is shown in formula (12): Where: L iris (E iris ,C) represents the rainbow model feature extractor E iris of confrontation losses; Loss Function The design of is to ensure that the type of feature modality extracted by the iris feature extractor remains discriminative while ensuring the discriminability of the iris category of the feature vector, so as to achieve the purpose of joint probability distribution registration of iris category and modality prediction. The loss function is shown in formula (13): in: represents the joint probabilistic registration loss of the iris feature extractor; Compute the joint probabilistic registration loss of the classification and ensemble discriminator views associated with the iris data The formula is shown in (14): Calculate the loss function L associated with the periocular feature extractor per (E per ,C), Loss function L per (E per ,C) is designed to ensure that the periocular feature extractor has a mutually antagonistic inhibitory relationship between the prediction of the periocular category and the prediction of the modality type. The loss function is shown in formula (15): Where: L per (E per ,C) represents the eye periphery feature extractor E per of confrontation losses; Loss Function The design of is to ensure that the type of feature modality extracted by the periocular feature extractor remains discriminative while ensuring the discriminability of the periocular category of the feature vector, so as to achieve the purpose of joint probability distribution registration of the periocular category and modality prediction. The loss function is shown in formula (16): in: represents the joint probabilistic registration loss of the periocular feature extractor; Compute the joint probabilistic registration loss of the classification and ensemble discriminator viewpoints associated with the periocular data The formula is shown in (17): In order to estimate the clustering with lower error, a minimization entropy principle is calculated as shown in formula (18): in: represents the minimum entropy loss of the iris or periorbital area; Step 5: Unfreeze the classification and discrimination integrator to make the weights learnable, freeze the iris feature extractor to make the weights unlearnable, and freeze the periorbital feature extractor to make the weights unlearnable. Training classification and discrimination ensembles, Indicates training C so that Take the minimum value; Step 6: Freeze the classification and discrimination integrators to make the weights unlearnable, unfreeze the iris feature extractor to make the weights learnable, and unfreeze the periorbital feature extractor to make the weights learnable. Train an iris feature extractor, where Indicates training E iris make Take the maximum value to learn the goal Train the periocular feature extractor, where Indicates training E per make Take the maximum value; Step 7: Repeat steps 3, 4, 5, and 6 until the model converges or the maximum number of cycles is reached.

8. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of step nine above is as follows: The first step is to expand the data of the iris visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After the expansion, the size of each image in the iris visible light image training set is adjusted to 224×224. The data expansion of the periocular visible light image training set is carried out. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After the expansion, the size of each image in the periocular visible light image training set is adjusted to 224×224. Step 2: Construct an iris and periorbital fusion classifier F. The fusion classifier is a three-layer perceptron. The number of neurons in the input layer is 2560, the number of neurons in the hidden layer is 2560, and the number of neurons in the output layer is K. The third step is to input the expanded iris visible light image training set and the expanded periocular visible light training set into the iris feature extractor and the periocular feature extractor respectively. The output of the iris feature extractor is the iris feature vector, and the output of the periocular extractor is the periocular feature vector. Two learnable parameters are set as the contribution weight coefficients of the iris feature vector and the periocular feature vector respectively. The iris feature vector and the periocular feature vector are multiplied by their respective weight coefficients respectively, and then the two vectors are spliced ​​in the feature direction to obtain a 2560-dimensional fusion feature vector. The entire fusion process can be expressed as V fusion =[γ iris V iris ,γ per V per ], where V fusion represents the final fusion feature vector of iris and periorbital area, γ iris represents the iris contribution trade-off coefficient, V iris represents the iris feature vector output by the iris feature extractor, γ per represents the periocular contribution trade-off coefficient, V per represents the periocular feature vector output by the periocular feature extractor; Step 4: Input the fused feature vector into the iris and periocular fusion classifier F, calculate the cross entropy loss, freeze the iris feature extractor and the periocular feature extractor, and perform error back propagation training to update the iris and periocular fusion classifier F and the trade-off coefficient γ per and γ iris .

9. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of training the feature extractor of the iris and periorbital area detection and fusion recognition model in the above step 10 is as follows: The first step is to expand the data of the eye visible light image training set. The data expansion strategy is random flipping in the horizontal direction, random flipping in the vertical direction, and random rotation. After expansion, the size of each image in the eye visible light image training set is adjusted to 224×224; Step 2: Input the visible light image of the eye into the lightweight IrisPer_YoloV3 iris and periocular target detection model. The detection model outputs the coordinates of the iris region and the periocular region. The iris region image and the periocular region image are cropped out by the coordinates. The iris image and the periocular region image are resized to 224×224. A fusion recognition sample is constructed. The fusion recognition sample consists of the cropped iris image, the periocular image and the identity label. The fusion recognition sample is input into the iris and periocular feature extraction and feature fusion classification model. The output of the fusion classification model is recorded as r. Step 3: Calculate the loss function L of the fusion classification fusion , freeze the iris and eye area detection model IrisPer_YoloV3, minimize the loss L fusion For the purpose of training the fusion classification model F, the loss function of the fusion classification is shown in formula (19): Where: n eye represents the total number of fusion recognition samples, r j Represents the jth element value of the soft output vector of the fusion classification model, t j Represents the j-th element value of the identity label Onehot vector of the fused recognition sample.

10. The iris and periorbital adversarial adaptive fusion recognition method based on contrast knowledge drive according to claim 1 is characterized by: The specific process of the multi-stage decision-making in step 11 above is as follows: The first step is to register. A random image in each category of the eye visible light image test set is extracted through the iris and periorbital dual-modal recognition model to obtain the iris and periorbital fusion feature vector. All feature vectors are stored in the feature vector database and the identity category information of the feature vector is marked. The second step is to perform first-order authentication. In the eye light image test set, a random image is selected and extracted through the iris and periocular dual-modal recognition model to obtain the iris and periocular fusion feature vector. The feature vector and the iris and periocular fusion feature vectors registered in the database are calculated one by one by the Euclidean distance decision matching function, and the iris and periocular fusion feature vector with the smallest distance to the feature vector to be authenticated is selected. The calculation formula of the Euclidean decision matching function is (20), as shown below: Where: euclideandistance(X prob ,Y gallery ) indicates X prob With Y gallery The Euclidean distance between prob Represents the iris and eye periphery fusion feature vector to be authenticated, Represents X prob The i-th element in Y gallery represents the fusion feature vector of iris and periorbital area in the registered database, Y gallery represent The i-th element in , L represents the dimension of the feature vector; The third step is to perform second-order authentication, compare the Euclidean distance between the feature vector to be authenticated and the feature vector registered in the database, select the registered feature vector with the smallest Euclidean distance, and calculate the cosine similarity between the feature vector to be authenticated and the registered feature vector with the smallest Euclidean distance. If the cosine similarity value is greater than the threshold value of 0.7, it is considered that the category of the feature vector to be identified is the same as the category of the iris and periorbital feature vector registered in the database. Otherwise, the identity category of the feature vector to be identified is not in the database and the identification fails. The calculation formula of the cosine similarity matching decision function is (21): Where: cosinedistance(X prob ,Y chosen ) indicates X prob With Y chosen The cosine similarity between chosen Indicates that the second step selects prob The eigenvector with the smallest Euclidean distance is Represents Y chosen The value of the i-th element of .

Images