Feature clustering face fraud detection method and system based on prototype learning and cosine similarity

Abstract

The application discloses a feature clustering face fraud detection method and system based on prototype learning and cosine similarity, which comprises the following steps: video preprocessing, extracting latent distribution features based on a latent distribution backbone network, normalizing the latent distribution features and prototype centers and calculating the cosine similarity between them; weighting and summing the cosine similarities of all prototype centers of the same category to obtain the classification prediction angle cosine of the corresponding category; calculating an additive angle loss function with an edge penalty coefficient; calculating the distance of each prototype center based on the inter-class loss function and the intra-class loss function; training the prediction model based on the constructed total loss function; adjusting the number of prototype centers according to the sample density around each prototype center and optimizing the prediction model; and outputting the test face fraud detection result based on the optimized prediction model. The application guarantees the feature extraction ability and discrimination ability of the model for face fraud videos and improves the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of face fraud detection, and in particular to a feature clustering face fraud detection method and system based on prototype learning and cosine similarity. Background Art

[0002] Nowadays, the use of facial biometrics in enterprises and industries has increased dramatically. For example, facial unlocking technology can be used to protect personal privacy in electronic devices, and facial biometrics can be used to authenticate payments. However, using the face as a biometric feature for authentication is unsafe. Facial biometric systems may be vulnerable to fraud attacks. Face fraud attacks can generally be divided into four categories: 1) photo attacks; 2) video replay attacks; 3) face mask attacks; and 4) adversarial sample attacks. These face fraud attacks are not only low-cost but can also deceive the system, seriously affecting and threatening the application of face recognition systems.

[0003] Existing face fraud detection algorithms primarily focus on two areas. On the one hand, they employ handcrafted features such as LBP, HOG, and rPPG to capture fraud clues. Other methods focus on temporal cues, such as blinking and lip movements. On the other hand, with the advancement of deep learning, many methods have begun to use convolutional neural networks (CNNs) to extract discriminative cues. These methods primarily involve extracting more representative features and designing more advanced network architectures to improve classification performance. Despite this, most of these face fraud detection algorithms treat them as classification problems. The classification results are optimized using the SoftMax loss function, which achieves good detection results within a database. However, cross-database detection results experience a sharp drop in accuracy, leading to insufficient generalization and poor transferability. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a feature clustering face fraud detection method and system based on prototype learning and cosine similarity. The present invention models complex data through a clustering method, introduces a prototype learning network, uses multiple prototype centers to represent the local cluster centers of each class, extracts latent distribution features through a latent distribution backbone network, clusters by calculating the cosine similarity between sample features and prototype centers, and performs final classification by weighted summation of the cosine similarities of the same category to obtain the category prediction angle. The total loss function is obtained by weighted summation of the additive angle loss function, the inter-class loss function, and the intra-class loss function. The prediction model is trained based on the total loss function to optimize the angle between the sample and the prototype center, and the distance between the prototype center within and between classes. This ensures the model's ability to extract and identify face fraud video features. While ensuring good detection results within the library, it effectively reduces the average error rate of the model across library tests and improves the generalization ability of the model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a feature clustering face fraud detection method based on prototype learning and cosine similarity, comprising the following steps:

[0007] Divide the divided data sets into frames and extract the face area of ​​each frame image as the area to be detected;

[0008] Build a potential distribution backbone network based on ResNet-18 and initialize the prototype center;

[0009] Input the image to be detected into the pre-trained latent distribution backbone network to obtain the latent distribution features, and normalize the latent distribution features and prototype centers;

[0010] Calculate the cosine similarity between the normalized latent distribution features and all prototype centers;

[0011] The weighted sum of the cosine similarities of all prototype centers of the same category is used to obtain the classification prediction angle cosine of the corresponding category;

[0012] Calculate the additive angle loss function with a marginal penalty coefficient for the classification prediction angle;

[0013] Calculate the distance between each prototype center based on the inter-class loss function and the intra-class loss function;

[0014] The total loss function is obtained by weighted summation of the additive angle loss function, the inter-class loss function, and the intra-class loss function, and the prediction model is obtained by training based on the total loss function;

[0015] Construct an adaptive prototype center adjustment algorithm to adjust the number of prototype centers according to the sample density around each prototype center and optimize the prediction model;

[0016] Based on the optimized prediction model, feature extraction and prediction classification are performed to output the test face fraud detection results.

[0017] As a preferred technical solution, the potential distributed backbone network is constructed based on ResNet-18 and the prototype center is initialized, specifically including:

[0018] Pre-train ResNet-18 and add fully connected layers as the potential distribution backbone network;

[0019] Two types of multiple prototype center vectors are randomly initialized in the sample space as the initial prototype centers.

[0020] As a preferred technical solution, the potential distribution characteristics and prototype centers are normalized, specifically expressed as follows:

[0021]

[0022]

[0023]

[0024]

[0025] where f represents the latent distribution feature, f i represents the L2 normalized latent distribution feature, c1 represents the dimension of the feature vector, ctotal1 represents the total number of dimensions of the feature vector, represents the prototype center, represents the L2 normalized prototype center, c2 represents the dimension of the prototype center vector, ctotal2 represents the total number of dimensions of the prototype center vector.

[0026] As a preferred technical solution, the cosine similarity of the normalized latent distribution feature and all prototype centers is calculated, which is specifically represented as:

[0027]

[0028] where f i is the normalized latent distribution feature of the i-th batch data, represents each normalized prototype center, j∈{0,1} represents the class to which it belongs, r∈{1,2,…,K} represents the r-th prototype center in the corresponding class, is the angle between the latent distribution feature and the prototype center in space.

[0029] As a preferred technical solution, the cosine similarity of all prototype centers of the same class is weighted and summed to obtain the classification prediction angle cosine of the corresponding class, which is specifically represented as:

[0030]

[0031] where θ j is the classification prediction angle size of class j, τ is a scaling factor, f i is the normalized latent distribution feature of the i-th batch data, represents each normalized prototype center, j∈{0,1} represents the class to which it belongs, r∈{1,2,…,K} represents the r-th prototype center in the corresponding class.

[0032] As a preferred technical solution, the classification prediction angle is calculated with an additive angle loss function with an edge penalty coefficient, which is specifically represented as:

[0033]

[0034] Among them, y i is the label vector of the i-th batch, Indicates the angle corresponding to the wrong label, m is the edge penalty coefficient, s is the scaling factor, Loss data represents the additive angular loss function.

[0035] As a preferred technical solution, the inter-class loss function is specifically expressed as:

[0036]

[0037] Among them, the first item represents the minimum distance between prototype centers of different categories, the second item represents the maximum distance among prototype centers of the same category, δ1 is the correction coefficient, and + means that the value in the brackets is greater than 0.

[0038] As a preferred technical solution, the intra-class loss function is specifically expressed as:

[0039]

[0040] in, represents the different prototype centers of category j, δ2 is the correction index, and r∈{1,2,…,K} represents the rth prototype center in the corresponding category.

[0041] As a preferred technical solution, the self-adaptive prototype center adjustment algorithm is constructed to adjust the number of prototype centers according to the sample density around each prototype center and optimize the prediction model, specifically including:

[0042] Traverse all trained prototype centers, calculate the cosine similarity between the sample and the selected prototype center as the distance metric, set the distance judgment threshold, when the calculated cosine similarity is less than the distance judgment threshold, that is, the sample belongs to this prototype center, the corresponding prototype center sample density is increased by 1, and stop after traversing all prototype centers or all samples;

[0043] Set a density determination threshold. When the sample density around the prototype center is less than the density determination threshold, remove the prototype center and save the best prediction model and the best weight.

[0044] The present invention also provides a feature clustering face fraud detection system based on prototype learning and cosine similarity, comprising: a video preprocessing module, a potential distribution backbone network construction module, a prototype center initialization module, a potential distribution feature extraction module, a normalization module, a cosine similarity calculation module, a classification prediction angle cosine calculation module, an additive angle loss calculation module, a prototype center distance calculation module, a total loss function construction module, a training module, an adjustment and optimization module, and a prediction module;

[0045] The video preprocessing module is used for framing the divided data set video, and extracting a face region of each frame image as a detection region;

[0046] The latent distribution backbone network construction module is used for constructing a latent distribution backbone network based on ResNet-18;

[0047] The prototype center initialization module is used for initializing a prototype center;

[0048] The latent distribution feature extraction module is used for inputting a detection image into a pre-trained latent distribution backbone network to extract a latent distribution feature;

[0049] The normalization module is used for normalizing the latent distribution feature and the prototype center;

[0050] The cosine similarity calculation module is used for calculating the cosine similarity of the normalized latent distribution feature and all prototype centers;

[0051] The classification prediction angle cosine calculation module is used for weighted summing the cosine similarity of all prototype centers of the same class to obtain a classification prediction angle cosine of the corresponding class;

[0052] The additive angle loss calculation module is used for calculating an additive angle loss function with an edge penalty coefficient for the classification prediction angle;

[0053] The prototype center distance calculation module is used for calculating the distance of each prototype center based on an inter-class loss function and an intra-class loss function;

[0054] The total loss function construction module is used for weighted summing the additive angle loss function, the inter-class loss function and the intra-class loss function to obtain a total loss function;

[0055] The training module is used for training a prediction model based on the total loss function;

[0056] The adjustment optimization module is used for constructing an adaptive adjustment prototype center algorithm, adjusting the number of prototype centers according to the sample density around each prototype center, and optimizing the prediction model;

[0057] The prediction module is used for performing feature extraction and prediction classification based on the optimized prediction model, and outputting a test face fraud detection result.

[0058] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0059] (1) The present invention regards the face fraud detection problem as a clustering problem with multiple cluster centers. Based on prototype learning, the prototype center is used instead of the cluster center to cluster the features obtained by the feature extraction network. During the network training process, the prototype center is used as a learnable parameter and is continuously optimized through training. It can adapt to different sample distributions in different scenarios and has good adaptability. It provides an effective way for face fraud detection. While maintaining a good detection effect within the database, it effectively reduces the average error rate of the model cross-database test and improves the generalization ability of the model.

[0060] (2) The present invention uses cosine similarity as the clustering metric, and expresses the distance by calculating the normalized angle between the feature and the prototype center. An additive angle loss function with a penalty coefficient is designed to constrain the final binary classification prediction result. The prototype center intra-class and inter-class loss functions are designed to constrain the position of the prototype center, effectively preventing the confusion of prototype centers of different categories and the overlap of prototype centers of the same category, improving the compactness of the prototype center class and the difference between classes, and thus improving the robustness of the model.

[0061] (3) After the model training is completed, the present invention can flexibly adjust the number of prototype centers through an adaptive prototype center adjustment algorithm according to the sample density around the prototype center, and use fewer prototype centers to cluster data, so that the model can reduce the computing cost and parameter amount without reducing performance, and can automatically complete the process without retraining the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the overall implementation framework of the feature clustering face fraud detection method based on prototype learning and cosine similarity of the present invention;

[0063] Figure 2 This is an overall training flow chart of the feature clustering face fraud detection method based on prototype learning and cosine similarity of the present invention;

[0064] Figure 3 This is a diagram showing the calculation process of the additive angle loss function of the feature clustering face fraud detection method based on prototype learning and cosine similarity of the present invention;

[0065] Figure 4 Schematic diagram of the calculation process of the inter-class loss function and the intra-class loss function of the feature clustering face fraud detection method based on prototype learning and cosine similarity of the present invention;

[0066] Figure 5 This is a graph showing comparative experimental results in the central library of prototypes with different numbers of initializations of the present invention;

[0067] Figure 6 (a) is a sample visualization diagram without prototype center clustering;

[0068] Figure 6 (b) a schematic diagram for visualization of the ABC categories that are difficult to classify in the previous classification model;

[0069] Figure 6 (c) a sample visualization schematic diagram of the prototype center clustering for different attack types corresponding to the ABC categories;

[0070] Figure 7 A prototype center similarity heat map obtained by training under different loss functions of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0072] Example 1

[0073] In this embodiment, five face-swapping video databases, CelebA-spoof, Replay_Attack, CASIA_MFSD, MSU_MFSD and OULU-NPU, are used for training and testing. CelebA-spoof contains 625537 photos of 10177 individuals, and the fraud images are taken from 8 scenes using more than 10 camera sensors. The CelebA-Spoof database is divided into training set, validation set and test set according to the ratio of 7:2:1; the Replay_Attack dataset contains 1200 videos, which are collected from 50 testers using a MacBook camera with a resolution of 320x240 pixels, and the fraud faces are generated accordingly, and are divided into training set, validation set and test set according to the ratio of 3:3:4; the CASIA_MFSD dataset contains 600 videos, which are collected from 50 testers using three cameras with resolutions of 640x480 pixels, 480x640 pixels and 1920x1080 pixels, and the fraud faces are generated accordingly, and are divided into training set and test set according to the ratio of 2:3; the MSU_MFSD dataset contains 280 videos, which are collected from 35 testers, and the fraud faces are generated accordingly, and 15 people are used for the training set and 20 people are used for the test set; the OULU-NPU dataset contains 1800 videos, which are collected from 55 testers using 6 kinds of mobile phones, and the fraud faces are generated accordingly, and are divided into training set, validation set and test set according to the ratio of 4:3:4.

[0074] This embodiment conducts two experiments. The first is to use the training set and validation set of the CelebA-spoof database for model training, and perform in-library testing on the test set of the CelebA-spoof database; the second is to select the training set and validation set of three of the four databases, Replay_Attack (hereinafter referred to as the I database), CASIA_MFSD (hereinafter referred to as the C database), MSU_MFSD (hereinafter referred to as the M database), and OULU-NPU (hereinafter referred to as the O database), for model training, and perform testing on the test set of the remaining database.

[0075] This embodiment is mainly implemented based on the deep learning framework Pytorch. The graphics card used in the experiment is RTX 3090, the system is Ubuntu 18.04, the CUDA version is 10.1.105, the cudnn version is 7.4.1, and the OpenCV version is 4.1.2. The detailed software and hardware information is shown in Table 1 below:

[0076] Table 1 Experimental deployment environment

[0077]

[0078] like Figure 1 、 Figure 2 As shown, this embodiment provides a feature clustering face fraud detection method based on prototype learning and cosine similarity, including the following steps:

[0079] S1. Divide the dataset to be tested into a training set, a validation set, and a test set, divide the video of each dataset into frames, and extract the face area of ​​each frame image as the area to be detected;

[0080] In this embodiment, the FFmpeg tool library is used to frame the video samples of each data set, and a frame is extracted every k frames to obtain a video frame sequence {f0, f1, ..., f N-1}, where N is the total number of frames extracted. Next, the face recognition algorithm of the Dlib tool library is used to extract the rectangular face area of ​​each frame in the video frame sequence as the detection area, and the Rezise() function of the Opencv tool library is used to scale it to a size of 224×224. In this embodiment, the value of k is 5;

[0081] S2. Build a potential distribution backbone network and initialize two types of K prototype centers

[0082] In this embodiment, the potential distribution backbone network selects ResNet-18 pre-trained on ImageNet with an output dimension of 2048, and adds a fully connected layer to obtain an output dimension of 512; 2×10 512-dimensional (K=10) prototype center vectors are randomly initialized in the sample space as the initial prototype center

[0083] S3, input the image to be detected into the pre-trained ResNet-18 to obtain the potential distribution feature f, and compare the potential distribution feature f and the prototype center Perform L2 normalization;

[0084] In this implementation, the extracted face detection area image is input into ResNet-18 to obtain the 512-dimensional potential distribution feature f, the prototype center Initialize K = 10 512-dimensional feature vectors for each of the two categories. First, calculate the sum of the second-order norms of all dimensions of the two vectors, and then divide the original vector by the sum of the norms to obtain the normalized vector. The specific formula for L2 normalization is as follows:

[0085]

[0086]

[0087] Among them, ff represents the potential distribution characteristics, f i represents the normalized potential distribution feature, c1 represents the dimension of the feature vector, and ctotal1 represents the total number of dimensions of the feature vector, which is 512 in this embodiment;

[0088]

[0089]

[0090] in, Represents the initialized prototype center, represents the normalized prototype center, c2 represents the dimension of the prototype center vector, and ctotal2 represents the total number of dimensions of the prototype center vector, which is 512 in this embodiment;

[0091] S4. Calculate the normalized feature f i With all Prototype Centers The cosine similarity Sim;

[0092] In this embodiment, the cosine similarity is calculated by performing a dot product operation on the vectors of the normalized features of the same dimension and the prototype center. The formula is as follows:

[0093]

[0094] Among them, f i is the normalized feature of the i-th batch data, Represents each normalized prototype center, where j∈{0,1} represents the category it belongs to (real / fraud), and r∈{1,2,…,K} represents the rth prototype center in the corresponding category. is the angle between the feature and the prototype center in space;

[0095] After the potential distribution features and prototype centers are normalized, the cosine similarity is expressed as the cosine of the angle between two unit vectors, as shown in the following formula:

[0096]

[0097] S5. Take the weighted sum of the cosine similarities Sim of all prototype centers of the same category to obtain the classification prediction angle cosine cosθ of category j j ;

[0098] In this embodiment, the weighted summation formula is as follows:

[0099]

[0100] Among them, θ j is the predicted angle size for category j, and τ is the scaling factor;

[0101] The arc cosine function is used to obtain the predicted angles of the two categories j. The specific formula is as follows:

[0102] θ j =arccos(cosθ j )

[0103] Here we get θ j is the final predicted angle size when the feature belongs to two categories, where j∈{0,1} represents the two categories of real / fraud.

[0104] S6. Calculate the additive angle loss function Loss with an edge penalty coefficient for the classification prediction angle data ;

[0105] like Figure 3 As shown, the specific approach is to add an additive edge penalty coefficient m to the angle of the correct category, where the angle of the correct category refers to the θ obtained above j The value of j is determined according to the correct sample label of the training data set, and then the angle SoftMax function is calculated. The formula is as follows:

[0106]

[0107] Among them, y i is the label vector of the i-th batch, with a dimension of 1×batch, and each label value ∈{0,1}, which is 1 if it is the correct label. Indicates the angle corresponding to the wrong label, m is the edge penalty coefficient, and s is the scaling factor;

[0108] S7. Use inter-class loss function Lossinter And the intra-class loss function Loss intra Calculate the distance between the centers of each prototype to obtain the loss function of the entire model;

[0109] like Figure 4 As shown, the specific approach is to use cosine similarity to represent the distance between prototype centers, calculate the maximum cosine similarity between prototype centers of different categories, and the cosine similarity between prototype centers of the same category. The larger the cosine similarity, the more similar the two are, and the smaller the distance in space, as shown in the following formula:

[0110]

[0111] In the above formula, the first term represents the minimum distance between prototype centers of different categories, the second term represents the maximum distance between prototype centers of the same category, δ1 is the correction coefficient, and + indicates that the value in the brackets is greater than 0;

[0112] Next, we calculate the distances between all prototype centers in the same category, and the results are expressed in matrix form as follows:

[0113]

[0114] in, represents the different prototype centers of category j, δ2 is the correction index;

[0115] The loss function of the entire model is obtained by summing the above three loss functions:

[0116] Loss=Loss data +λ1Loss inter +λ2Loss intra

[0117] Among them, λ1 and λ2 are balance coefficients.

[0118] S8. Train the above model, calculate the loss function and back-propagate to update the network weight coefficient, and stop training when the set epoch is reached;

[0119] In this embodiment, the training process is as follows: Figure 2 As shown in the figure, the cropped face images are input into the built prototype learning clustering model for end-to-end training, with the maximum number of epochs set to 50, the batch size to 512, the SGD optimizer with momentum of 0.9, and the initial learning rate and learning rate decay parameters to 1×10 -3and 0.95, and a step learning rate adjustment strategy is adopted, and the learning rate is attenuated to 0.95 of the original learning rate every 10 continuous training cycles, the prototype center is initialized to 10 per class, the inter-class loss penalty coefficient δ1 is 0.25, the intra-class loss function penalty coefficient δ2 is 0.25, the angle loss function scaling factor τ is 2, the edge coefficient m is 0.25, the balance coefficient λ1 is 0.1, and λ2 is 0.1;

[0120] When the data training of each epoch is completed, the current model is verified using the verification set, the loss function value is returned, and the best result is compared, if it is greater than the best result, the best result is updated, and if it is less than the best result, the training is continued until the maximum set epoch is reached and the training is stopped;

[0121] S9, an adaptive adjustment prototype center algorithm is constructed, the number of prototype centers is adjusted according to the sample density around each prototype center, the model is optimized, and the best network model and the best weight are saved;

[0122] In this embodiment, all the trained prototype centers are traversed first The cosine similarity between the sample and the selected prototype center is calculated as a distance measure, and a distance judgment threshold t is set to 0.25. When the calculated cosine similarity is less than the threshold, the sample belongs to the prototype center, and the corresponding prototype center sample density is increased by 1. After traversing all the prototype centers or all the samples, the process is stopped. A density judgment threshold d is set to 100. When the sample density around the prototype center is less than d, the prototype center is removed. The best network model and the best weight are saved. That is, the sample density around the prototype center is represented by the number of samples within a given radius.

[0123] S10, using the trained model to extract features and predict classification, outputting the test face fraud detection result;

[0124] In this embodiment, when the library experiment is performed, the trained weight in CelabA library is used to test the test set and return the test index. When the cross-library experiment is performed, three databases in C, I, M and O are selected as the training and verification set, and the remaining one library is selected as the test database. There are four groups of cross-library experiments, and the test index is returned.

[0125] False Acceptance Rate (FAR), False Rejection Rate (FRR) and Half Total Error Rate (HTER) are used as test indexes. The smaller the above index value is, the better the classification performance of the model in the data set is. The calculation formulas of FAR, FRR and HTER are as follows:

[0126]

[0127]

[0128]

[0129] Wherein, the false acceptance FA refers to the number of false face samples being misjudged as true face samples, the false rejection FR refers to the number of true face samples being misjudged as false face samples, the true acceptance number TA refers to the number of true face samples being correctly judged as true face samples, and the true rejection number TR refers to the number of false face samples being correctly judged as false face samples;

[0130] The in-database results of the present example in the CelebA-Spoof dataset are shown in the following table:

[0131] Table 2 In-database experimental results of CelebA-Spoof

[0132]

[0133] As can be seen from Table 2, the false acceptance rate of the model trained on the CelebA-Spoof database in the present example is 0.58%, the false rejection rate is 1.17%, and the semi-error rate is 0.87%, which shows good in-database test results, and verifies the effectiveness of the method of the present example.

[0134] The method of the present example tests different numbers of prototype centers on the basis of the above in-database experiment, and the specific method is to take different values of K when initializing the prototype center in the network training stage, which represents different numbers of prototype centers of the model, retrain the network and perform in-database test experiment.

[0135] The experimental results are shown in Figure 5 The broken line chart is the final network in-database test error rate when initializing different numbers of prototype centers, and the other three points are the experimental results of the current mainstream algorithm, it can be seen that the increase of the prototype center improves the classification ability of the model, compared with K = 2, the classification results are improved, which confirms the feasibility of the multi-prototype center adopted in the present example, when the optimal number of prototype centers is reached, the experimental results start to fluctuate up and down, which shows that there is an optimal number of prototype centers.

[0136] The present example performs t-SNE visualization experiment on the final features of the model, and uses the visualization toolkit t-SNE to visualize the sample features after clustering by the model for further analysis of the features of the samples around the prototype center.

[0137] like Figure 6 (a)- Figure 6 As shown in (c), we can see that the model can divide each category of the model into different clusters at the center of the prototype, and the sample distribution is more compact. When visualizing the clusters, we can see that samples with the same attributes can be clustered. Figure 6 (a) is the sample visualization without prototype center clustering. Figure 6 (b) is the feature distribution of the model samples in this embodiment. We select the ABC category, which is more difficult to classify in the previous classification model, for visualization. We can see Figure 6 In (c), A, B, and C correspond to three different attack types: paper attack, tablet replay attack, and hole-punched paper photo. In this model, they can be well clustered by the prototype centers of each category, indicating that this model can well distinguish data with different attributes through prototype center training and has good generalization.

[0138] In addition, this example method also conducted cross-database experiments on multiple databases. The experimental results are shown in Table 3 below:

[0139] Table 3 CMIO multi-library cross-library experiment

[0140]

[0141] From the results in Table 3, we can see that the proposed model performs well on unseen databases, with the best performance of 6.58% achieved in the OCI→M database experiment, demonstrating good cross-database performance.

[0142] This example explores the effectiveness of the three proposed loss functions and also conducts a heat map of the impact of different loss functions on the similarity of the model prototype centers. Here, four prototype centers are initialized for each of the two categories, and the similarity between each prototype center is calculated and represented by different colors. Figure 7 As shown, similarity = 1 indicates that the two prototype center vectors have the same spatial position. The leftmost figure shows that when only the angle loss function is used, the similarity between the prototype centers of the two categories is not much different. After adding the inter-class prototype center loss function, the cosine similarity of different categories becomes smaller, that is, they become more different, while the similarity of the same category becomes larger, which may be due to the problem of prototype centers being too similar and overlapping. This is consistent with the above analysis. Finally, the intra-class prototype center loss function is used to ensure the difference between classes while ensuring the difference between each prototype center within the class. It can be seen that the combination of the three loss functions makes the prototype centers of the same category in the model more compact, and the prototype centers of different categories more different. It proves that the three proposed loss functions are effective in constraining the inter-class and intra-class distribution of prototype centers.

[0143] This example uses clustering to model complex data, introducing a prototype learning network. Specifically, the last layer is replaced with a network consisting of 2×K prototype centers. The previous classification network directly mapped 512-dimensional features to two fully connected classes. This example replaces this with a 2×K prototype center vector initialization for clustering. Multiple prototype centers are used to represent the local cluster centers of each class. Latent distribution features are extracted using a latent distribution backbone network. The cosine similarity between sample features and prototype centers is calculated to determine the prototype center to which the sample belongs, and thus the class to which it belongs. Three loss functions are used to optimize the angle between the sample and the prototype center, as well as the distance between the prototype center within and between classes. This ensures the model's ability to extract and identify facial spoofing videos. While maintaining good detection performance within the database, it also effectively reduces the average error rate in cross-database testing, improving the model's generalization. Experimental results demonstrate that this example performs well in both within-database testing on the CelebA-Spoof database and in cross-database testing on the CMIO database, effectively improving the model's generalization.

[0144] Example 2

[0145] This embodiment provides a feature clustering face fraud detection system based on prototype learning and cosine similarity, which is used to implement the feature clustering face fraud detection method based on prototype learning and cosine similarity of the above-mentioned embodiment 1. The system includes: a video preprocessing module, a potential distribution backbone network construction module, a prototype center initialization module, a potential distribution feature extraction module, a normalization module, a cosine similarity calculation module, a classification prediction angle cosine calculation module, an additive angle loss calculation module, a prototype center distance calculation module, a total loss function construction module, a training module, an adjustment and optimization module, and a prediction module.

[0146] In this embodiment, the video preprocessing module is used to divide the divided data sets into frames and extract the face area of ​​each frame image as the area to be detected;

[0147] In this embodiment, the potential distributed backbone network construction module is used to construct a potential distributed backbone network based on ResNet-18;

[0148] In this embodiment, the prototype center initialization module is used to initialize the prototype center;

[0149] In this embodiment, the latent distribution feature extraction module is used to input the image to be detected into the pre-trained latent distribution backbone network to extract the latent distribution features;

[0150] In this embodiment, the normalization module is used to normalize the potential distribution features and prototype centers;

[0151] In this embodiment, the cosine similarity calculation module is used to calculate the cosine similarity between the normalized potential distribution features and all prototype centers;

[0152] In this embodiment, the classification prediction angle cosine calculation module is used to obtain the classification prediction angle cosine of the corresponding category by weighted summing up the cosine similarities of all prototype centers of the same category;

[0153] In this embodiment, the additive angle loss calculation module is used to calculate the additive angle loss function with an edge penalty coefficient for the classification prediction angle;

[0154] In this embodiment, the prototype center distance calculation module is used to calculate the distance between each prototype center based on the inter-class loss function and the intra-class loss function;

[0155] In this embodiment, the total loss function construction module is used to obtain the total loss function based on the weighted sum of the additive angle loss function, the inter-class loss function, and the intra-class loss function;

[0156] In this embodiment, the training module is used to obtain a prediction model based on the total loss function training;

[0157] In this embodiment, the adjustment and optimization module is used to construct an adaptive prototype center adjustment algorithm, adjust the number of prototype centers according to the sample density around each prototype center, and optimize the prediction model;

[0158] In this embodiment, the prediction module is used to perform feature extraction and prediction classification based on the optimized prediction model, and output the test face fraud detection result.

[0159] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A feature clustering face fraud detection method based on prototype learning and cosine similarity, characterized in that: The steps include: Divide the divided data sets into frames and extract the face area of ​​each frame image as the area to be detected; Build a potential distribution backbone network based on ResNet-18 and initialize the prototype center; Input the image to be detected into the pre-trained latent distribution backbone network to obtain the latent distribution features, and normalize the latent distribution features and prototype centers; Calculate the cosine similarity between the normalized latent distribution features and all prototype centers; The weighted sum of the cosine similarities of all prototype centers of the same category is used to obtain the classification prediction angle cosine of the corresponding category; Calculate the additive angle loss function with a marginal penalty coefficient for the classification prediction angle; Calculate the distance between each prototype center based on the inter-class loss function and the intra-class loss function; The total loss function is obtained by weighted summation of the additive angle loss function, the inter-class loss function, and the intra-class loss function, and the prediction model is obtained by training based on the total loss function; Construct an adaptive prototype center adjustment algorithm to adjust the number of prototype centers according to the sample density around each prototype center and optimize the prediction model; Based on the optimized prediction model, feature extraction and prediction classification are performed to output the test face fraud detection results.

2. The method for face fraud detection based on prototype learning and cosine similarity by feature clustering according to claim 1, characterized in that: The potential distribution backbone network is constructed based on ResNet-18 and the prototype center is initialized, specifically including: Pre-train ResNet-18 and add fully connected layers as the potential distribution backbone network; Two types of multiple prototype center vectors are randomly initialized in the sample space as the initial prototype centers.

3. The method for face fraud detection based on prototype learning and cosine similarity by feature clustering according to claim 1, characterized in that: The potential distribution characteristics and prototype centers are normalized as follows: Among them, f represents the potential distribution characteristics, f i Represents the potential distribution characteristics of the normalized i-th batch data, c1 represents the dimension of the feature vector, and ctotal1 represents the total number of dimensions of the feature vector. Represents the prototype center, Represents the prototype center after L2 normalization, c2 represents the dimension of the prototype center vector, and ctotal2 represents the total number of dimensions of the prototype center vector.

4. The method for face fraud detection based on prototype learning and cosine similarity by feature clustering according to claim 1, characterized in that: The cosine similarity between the normalized potential distribution features and all prototype centers is specifically expressed as: Among them, f i is the potential distribution feature of the normalized i-th batch data, Represents each normalized prototype center, j∈{0,1} represents the category it belongs to, r∈{1,2,…,K} represents the rth prototype center in the corresponding category, is the angle between the potential distribution feature and the prototype center in space.

5. The method for face fraud detection based on feature clustering based on prototype learning and cosine similarity according to claim 1, characterized in that: The weighted sum of the cosine similarities of all prototype centers of the same category is used to obtain the classification prediction angle cosine of the corresponding category, which is specifically expressed as: Among them, θ j is the classification prediction angle size of category j, τ is the scaling factor, f i is the potential distribution feature of the normalized i-th batch data, Represents each normalized prototype center, j∈{0,1} represents the category it belongs to, and r∈{1,2,…,K} represents the rth prototype center in the corresponding category.

6. The method for face fraud detection based on prototype learning and cosine similarity by feature clustering according to claim 1, characterized in that: The additive angle loss function with an edge penalty coefficient for the classification prediction angle calculation is specifically expressed as: Among them, y i is the label vector of the i-th batch, Indicates the angle corresponding to the wrong label, m is the edge penalty coefficient, s is the scaling factor, Loss data represents the additive angular loss function.

7. The method for face fraud detection based on feature clustering based on prototype learning and cosine similarity according to claim 1, characterized in that: The inter-class loss function is specifically expressed as: Among them, the first item represents the minimum distance between prototype centers of different categories, the second item represents the maximum distance between prototype centers of the same category, δ1 is the correction coefficient, and + means that the value in the brackets is greater than 0.

8. The method for face fraud detection based on feature clustering based on prototype learning and cosine similarity according to claim 1, characterized in that: The intra-class loss function is specifically expressed as: in, represents the different prototype centers of category j, δ2 is the correction index, and r∈{1,2,…,K} represents the rth prototype center in the corresponding category.

9. The method for face fraud detection based on feature clustering based on prototype learning and cosine similarity according to claim 1, characterized in that: The method of constructing an adaptive prototype center adjustment algorithm adjusts the number of prototype centers according to the sample density around each prototype center and optimizes the prediction model, specifically including: Traverse all trained prototype centers, calculate the cosine similarity between the sample and the selected prototype center as the distance metric, set the distance judgment threshold, when the calculated cosine similarity is less than the distance judgment threshold, that is, the sample belongs to this prototype center, the corresponding prototype center sample density is increased by 1, and stop after traversing all prototype centers or all samples; Set a density determination threshold. When the sample density around the prototype center is less than the density determination threshold, remove the prototype center and save the best prediction model and the best weight.

10. A feature clustering face fraud detection system based on prototype learning and cosine similarity, characterized in that: include: Video preprocessing module, potential distribution backbone network construction module, prototype center initialization module, potential distribution feature extraction module, normalization module, cosine similarity calculation module, classification prediction angle cosine calculation module, additive angle loss calculation module, prototype center distance calculation module, total loss function construction module, training module, adjustment optimization module and prediction module; The video preprocessing module is used to divide the divided data sets into frames and extract the face area of ​​each frame image as the area to be detected; The potential distribution backbone network construction module is used to construct a potential distribution backbone network based on ResNet-18; The prototype center initialization module is used to initialize the prototype center; The latent distribution feature extraction module is used to input the image to be detected into the pre-trained latent distribution backbone network to extract the latent distribution features; The normalization module is used to normalize the potential distribution features and the prototype center; The cosine similarity calculation module is used to calculate the cosine similarity of the normalized potential distribution features and all prototype centers; The classification prediction angle cosine calculation module is used to obtain the classification prediction angle cosine of the corresponding category by weighted summation of the cosine similarities of all prototype centers of the same category; The additive angle loss calculation module is used to calculate the additive angle loss function with an edge penalty coefficient for the classification prediction angle; The prototype center distance calculation module is used to calculate the distance between each prototype center based on the inter-class loss function and the intra-class loss function; The total loss function construction module is used to obtain the total loss function based on the weighted summation of the additive angle loss function, the inter-class loss function, and the intra-class loss function; The training module is used to obtain a prediction model based on the total loss function training; The adjustment and optimization module is used to construct an adaptive prototype center adjustment algorithm, adjust the number of prototype centers according to the sample density around each prototype center, and optimize the prediction model; The prediction module is used to perform feature extraction and prediction classification based on the optimized prediction model, and output the test face fraud detection result.

Images