A fraud detection method and system for makeup faces

Abstract

The present invention discloses a fraud detection method and system for faces wearing makeup. The method comprises the following steps: decoding videos from various datasets into frame sequences; cropping facial regions and detecting facial landmarks; selecting a portion of real face images without makeup to generate a set of augmented images with makeup; extracting features from different databases using a ResNet18 network improved with a batch channel normalization module; obtaining the optimal true / false classification hyperplanes from different databases and converging them to a global true / false classification hyperplane, calculating the global cross-entropy loss, and outputting binary true / false prediction labels; calculating the K most similar samples in a cache to a new sample, calculating the nearest neighbor supervised contrast loss between the selected anchor points and the new sample; constructing virtual triples and calculating the virtual triple loss; constructing a total loss function and training a prediction model; and obtaining binary prediction probabilities based on the prediction model. The present invention effectively improves the generalization and detection performance of faces wearing makeup.

Description

Technical Field

[0001] The present invention relates to the technical field of face fraud detection, and in particular to a fraud detection method and system for faces wearing makeup. Background Art

[0002] Face spoofing refers to the act of forging, simulating, or using non-authentic facial information to deceive facial recognition systems, enabling illegal access, identity theft, or bypassing authentication. Key attack methods include photo printing, video replay, and facial masks. Existing face spoofing detection technologies often perform well within a database, but perform poorly across databases. This is primarily due to the fact that data within and outside the database is often collected under different conditions, such as different camera equipment, ambient lighting, and presentation devices. This results in domain shift between the two data sets. Insufficient training data diversity can lead to overfitting during in-database learning, resulting in poor generalization performance.

[0003] In addition, with the emergence of various commercial beauty products, makeup has become increasingly common for modern people, covering a variety of makeup methods such as foundation, eye shadow, lipstick and blush. Most made-up faces in real life are mainly light makeup. Although there are changes in facial texture and color of some areas, the identity of the face is not changed, so it should still be judged as a real face. However, existing technologies have poor generalization effects on made-up faces and it is difficult to meet the requirements of detection accuracy. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a fraud detection method and system for made-up faces. The present invention uses only a small number of made-up face images to perform makeup augmentation on the input image, enriching the samples of the input feature extraction network. The makeup face feature learning generated based on makeup augmentation improves the generalization for made-up faces. The batch channel normalization method is dynamically adjusted by adaptive parameters to better adapt to different data sets and models. By calculating the nearest neighbor supervision contrast loss, the similarity of samples in the same database and the same category is maximized, the intra-class compactness of features is improved, and the detection performance is effectively improved. By constructing two fixed orthogonal vectors to replace the positive and negative examples in the classic triplet, the distance between similar samples can be shortened and the distance between heterogeneous samples can be extended, thereby improving the detection performance. Compared with the classic ternary loss, the virtual ternary loss has lower computational complexity.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a fraud detection method for a face wearing makeup, comprising the following steps:

[0007] Divide the data set and decode the video of each data set into a frame sequence;

[0008] Crop out the face regions of each dataset frame sequence and the selected makeup reference image set, and perform facial landmark detection;

[0009] Select some real face images without makeup from the training set with cropped face areas, and use the makeup reference face images after cropping the face areas as references to perform makeup augmentation to generate a makeup augmented image set;

[0010] Select the feature extraction area, use the ResNet18 network improved by the batch channel normalization module as the backbone network, input the feature extraction areas of different databases into the backbone network in batches for feature extraction, and extract the features of different databases;

[0011] Based on the features extracted from different databases, the binary classifiers are trained separately to obtain the global optimal true and false classification hyperplanes for different databases. The global cross entropy loss is calculated. The optimal true and false classification hyperplanes for different databases are optimized by the projected gradient method and converged to the global true and false classification hyperplane. The binary classification true and false prediction labels are output.

[0012] Establish a cache area to store sample feature representations. For new sample features extracted by the backbone network, calculate the top-K samples in the cache area that are most similar to the current sample. Select samples from the same database and of the same type as the current sample as anchor points, and calculate the nearest neighbor supervision contrast loss between the anchor points and the new sample.

[0013] Construct two fixed orthogonal vectors, form a virtual triplet with the features extracted by the backbone network, and calculate the virtual triplet loss of the virtual triplet;

[0014] The total loss function is obtained by weighted summation of global cross entropy loss, neighbor supervision contrast loss and virtual ternary loss, and the prediction model is trained based on the total loss function;

[0015] Input the face image test set into the trained prediction model, obtain the binary classification prediction probability, and output the final prediction result.

[0016] As a preferred technical solution, makeup augmentation is performed to generate a makeup augmented image set, specifically including:

[0017] Perform TPS transformation on the makeup reference face image, using the facial landmarks of the real face image without makeup as the reference points of the TPS transformation, and the facial landmarks of the makeup reference face image as the control points of the TPS transformation. The control points are mapped to the positions of the reference points through affine transformation, and other pixels are corrected based on TPS interpolation;

[0018] Substitute the coordinates of the control points after affine transformation into the interpolation function of TPS to obtain the mapping of other pixels of the makeup reference face image to the natural face image;

[0019] The makeup reference face image after TPS transformation and the real face image without makeup are Poisson fused to generate the makeup augmented image.

[0020] As a preferred technical solution, the Poisson fusion of the makeup reference face image after TPS transformation and the real face image without makeup to generate the makeup augmented image specifically includes:

[0021] Generate an eye region mask and a lip region mask, and perform a bitwise OR operation on the eye region mask and the lip region mask to obtain an eye and lip region mask;

[0022] Determine whether the face image is a real face based on the true and false labels, and determine whether the face image is a face wearing glasses based on the glasses label;

[0023] Randomly generate a probability value and set a probability value threshold, and perform lip makeup augmentation, lip and eye makeup augmentation, and eye makeup augmentation based on the probability value threshold and the face image type;

[0024] Based on different makeup augmentations, the corresponding region mask is selected and the makeup reference face image and the real face image without makeup are Poisson fused. The specific calculation formula of the objective function is:

[0025]

[0026] Among them, I mr_TPS (x, y) is the pixel value of the makeup reference face image, I r (x,y) is the pixel value of the plain face image, represents the gradient operation on the image, and offset(x,y) is the offset of the makeup reference face image relative to the real face image without makeup.

[0027] As a preferred technical solution, the ResNet18 network improved by the batch channel normalization module is used as the backbone network. The feature extraction areas of different databases are input into the backbone network in batches for feature extraction. Specifically, it includes:

[0028] Improve the ResNet18 network by replacing the batch normalization layer in ResNet18 with the batch channel normalization module. The batch channel normalization module normalizes the input data along the channel dimension and batch dimension respectively, and calculates the mean μ1 and variance of the layer input along the (N, H, W) axis. Calculate the mean μ2 and variance along the (C, H, W) axis Specifically expressed as:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] Among them, q r For input, are respectively μ1, and μ2, Normalized, ∈ is a constant, N represents the batch, C represents the channel, H represents the feature height, and W represents the feature width;

[0036] The two normalized outputs are adaptively weighted as the final output. The specific calculation formula is:

[0037]

[0038] Among them, Y represents the extracted features, τ, γ and is a learnable parameter, τ is used to adaptively balance the normalized output along the (N, H, W) and (C, H, W) axes, and γ is used to scale the normalized values. Used for translation.

[0039] As a preferred technical solution, the calculation of the global cross entropy loss is specifically expressed as:

[0040]

[0041]

[0042] Among them, L cls represents the global cross entropy loss function, L cls_e represents the cross entropy loss function, e represents the database, E represents the number of databases, and p z Represents a predicted probability value in a batch, y z Indicates the value of the true or false label, p z (y z ) indicates the prediction is label y z probability.

[0043] As a preferred technical solution, the projected gradient method is used to optimize the optimal true and false classification hyperplanes of different databases and converge to a global true and false classification hyperplane, specifically including:

[0044] Calculate the gradient of the global cross entropy loss for different database hyperplanes. For each database e, select the hyperplane β that is consistent with the current update. eThe farthest hyperplane Based on the interpolation method, the currently updated hyperplane β e and the farthest hyperplane Perform interpolation update and use the gradient descent method to update the feature space parameters, which can be expressed as:

[0045]

[0046]

[0047] Among them, θ is the target point, is the loss function, ξ is the learning rate, P Ω () is the Euclidean projection, Ω is the feasible set, which is used to describe the value range of the hyperplane β, and η is the alignment parameter used to adjust the current updated hyperplane β e Perform interpolation updates.

[0048] As a preferred technical solution, a buffer area for storing sample feature representations is established, specifically including:

[0049] At the beginning of training, an empty buffer is initialized; during the training process, the network Φ is updated using the EMA method. EMA Extract features from samples and transform the network Φ EMA The extracted sample features are stored in the cache, Φ EMA The network architecture is the same as the backbone network, and the EMA method is used to update the parameters. The specific calculation formula is as follows:

[0050] Φ EMA ←(1-θ)Φ EMA_old +θΦ grad

[0051] Among them, Φ EMA_old is the network whose parameters are updated in EMA mode after the last update, Φ grad is the backbone network, which uses the gradient descent method to update parameters, Φ EMA is the network updated by EMA, θ is the momentum coefficient;

[0052] The buffer is a queue data structure. New features enter the buffer from the end of the queue. If the buffer is full, the head feature leaves the buffer.

[0053] As a preferred technical solution, for the new sample features extracted by the backbone network, the top-K most similar samples in the cache are calculated. From these samples, samples of the same type and in the same database as the current sample are selected as anchor points. The nearest neighbor supervision contrast loss between the anchor points and the new sample is calculated, specifically including:

[0054] Calculate the dot product between the new sample feature representation and the sample feature representation in the cache area to obtain the similarity matrix;

[0055] Based on the KNN algorithm, the first K samples that are most similar to the new sample are selected from the cache, and all elements in the similarity matrix except the elements corresponding to these K samples are set to 0;

[0056] After selecting K samples, the similarity matrix is ​​normalized. According to the true and false category labels and database labels, N samples of the same type as the new sample and in the same database are selected as anchor points. The nearest neighbor supervision contrast loss is calculated. The specific calculation formula is:

[0057]

[0058] Among them, g u is the weight of the anchor point in the similarity matrix, and u is the index value of the anchor point.

[0059] As an optimal technical solution, two fixed orthogonal vectors are constructed to form a virtual triplet with the features extracted by the backbone network, and the virtual triplet loss of the virtual triplet is calculated, which is specifically expressed as:

[0060] Construct two fixed orthogonal vectors d c and d 1-c , d c and d 1-c The length is consistent with the length of the feature vector extracted from a single sample;

[0061] The specific calculation formula of the virtual ternary loss is:

[0062] L dmyt (F z ,c)=max(2F z ·d 1-c -2F z ·d c +ρ,0)

[0063] Among them, F z Represents the features extracted from a single sample, c represents the true and false categories, and ρ is the marginal interval of the loss function.

[0064] The present invention also provides a fraud detection system for makeup faces, comprising: data set division, video decoding module, face region extraction module, face marking module, makeup augmentation module, feature extraction module, different database hyperplane alignment module, buffer area construction module, nearest neighbor supervision comparison module, virtual triple construction module, virtual triple loss calculation module, total loss function construction module, model training module, and prediction module;

[0065] The data set partitioning is used to partition the data set;

[0066] The video decoding module is used to decode the video of each data set into a frame sequence;

[0067] The face region extraction module is used to crop the face regions of each data set frame sequence and the selected makeup reference image set;

[0068] The face marking module is used to detect face landmarks on the cropped face area;

[0069] The makeup augmentation module is used to perform makeup augmentation, selecting some real face images without makeup from the training set with cropped face areas, and using the makeup reference face images after cropping the face areas as references to generate a makeup augmented image set through makeup augmentation;

[0070] The feature extraction module is used for feature extraction, selecting a feature extraction area, using an improved ResNet18 network with batch channel normalization as a backbone network, and inputting the feature extraction areas of different databases into the backbone network in batches for feature extraction to obtain features of different databases;

[0071] The different database hyperplane alignment module is used to construct a global optimal true and false classification hyperplane, train binary classifiers based on the features extracted from different databases, obtain the optimal true and false classification hyperplanes for different databases, calculate the global cross entropy loss, optimize the optimal true and false classification hyperplanes for different databases through the projected gradient method and converge to the global true and false classification hyperplane, and output the binary classification true and false prediction labels;

[0072] The buffer area construction module is used to establish a buffer area for storing sample feature representations;

[0073] The nearest neighbor supervision comparison module is used to calculate the nearest neighbor supervision comparison loss between the anchor point and the new sample. For the new sample features extracted by the backbone network, the top-K samples in the cache area that are most similar to it are calculated, and samples of the same database and type as the current sample are selected as anchor points to maximize the similarity between the current sample and the anchor point.

[0074] The virtual triplet construction module is used to construct a virtual triplet based on the two constructed fixed orthogonal vectors and the features extracted by the backbone network;

[0075] The virtual ternary loss calculation module is used to calculate the virtual ternary loss of the virtual triplet;

[0076] The total loss function construction module is used to obtain the total loss function according to the weighted sum of the global cross entropy loss, the nearest neighbor supervision contrast loss and the virtual ternary loss;

[0077] The model training module is used to obtain a prediction model based on total loss function training;

[0078] The prediction module is used to input the face image test set into the trained prediction model, obtain the binary classification prediction probability, and output the final prediction result.

[0079] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0080] (1) The present invention uses only a small number of makeup reference images to enhance the set of real face images without makeup and generate a set of augmented face images with makeup, which enriches the samples of the input feature extraction network and improves the generalization of the network for makeup faces.

[0081] (2) The present invention normalizes the input data along the channel dimension and the batch dimension respectively, and uses adaptive parameters to dynamically adjust the normalization method, so as to better adapt to different data sets and models and improve the generalization of the model.

[0082] (3) The present invention calculates the K samples in the cache area that are most similar to the current sample, selects samples from the same database and the same category as the current sample as anchor points, calculates the neighbor supervision comparison loss between the anchor points and the new samples, maximizes the similarity of samples from the same database and the same category, improves the compactness of the feature class, and effectively improves the detection performance.

[0083] (4) The present invention constructs two fixed orthogonal vectors to replace the positive and negative examples in the classic triplet, which can shorten the distance between similar samples and increase the distance between heterogeneous samples, thereby improving the detection performance. Compared with the classic ternary loss, the virtual ternary loss has lower computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Schematic diagram of the process of the fraud detection method for a face wearing makeup according to the present invention;

[0085] Figure 2 A schematic diagram of the implementation process of makeup augmentation according to the present invention;

[0086] Figure 3 Schematic diagram of the implementation process of batch channel normalization in the present invention;

[0087] Figure 4 Schematic diagram of the implementation process of hyperplane alignment of different databases in the present invention;

[0088] Figure 5 Schematic diagram of the implementation process of the invented nearest neighbor supervised comparative calculation.

[0089] Figure 6 This is a schematic diagram of the main architecture of the fraud detection system for makeup faces of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0091] Example 1

[0092] This embodiment uses the face fraud detection databases Oulu-NPU, CASIA-FASD, Replay attack, and MSU-MFSD, the public database MT-Dataset in the field of makeup detection and migration, and a library of makeup face images generated by migrating makeup from real bare faces in Oulu-NPU, CASIA-FASD, Replayattack, and MSU-MFSD. The Oulu-NPU database contains 55 subjects, collected in three scenarios, with a total of 990 real videos and 3960 fraud attack videos. The attack methods are paper printing attack and replay attack, shot with 6 types of mobile phones; the CASIA-FASD database contains 300 real videos and 300 fraud attack videos, collected from 50 subjects, shot with a high-resolution Sony NEX-5 camera and a low-quality USB camera, the attack methods include distorted photos, cut photos and video replay, including a variety of postures and expression changes; the Replay attack database contains 50 subjects, with a total of 200 real videos and 1000 fraud attack videos, the attack methods are A4 paper printing attack, iPhone replay attack and iPad replay attack, shot with Mac front camera; the MSU-MFSD database is collected from 35 subjects, including 70 real videos and 210 fraud videos, shot with MacBook Air The images were taken with the built-in camera of a 13-inch laptop and the front camera of a Google Nexus 5 Android phone. MT-Dataset is a database for makeup transfer, containing 2,719 makeup-enhanced face images and 1,114 non-makeup face images. The images were collected using a web crawler. Due to the current lack of publicly available makeup face libraries, this example uses two makeup transfer methods: SpMT (semi-parametric makeup transfer via semantic-aware correspondence) and EleGANt (exquisite and locally editable GAN for makeup transfer). Using high-definition makeup face images collected online as a reference, makeup transfer is performed on real bare-faced faces from the Oulu-NPU, CASIA-FASD, Replay attack, and MSU-MFSD databases, yielding a set of makeup-enhanced face images generated by makeup transfer. This example was performed on a Linux system, primarily based on the deep learning framework Pytorch, using a GTX1080Ti graphics card and CUDA version 11.4.

[0093] like Figure 1As shown, this embodiment provides a fraud detection method for a face wearing makeup, comprising the following steps:

[0094] S1: Divide the dataset into a training set, a validation set, and a test set. Decode the videos of each dataset into frame sequences. Select n images from the MT-Dataset database as makeup reference image sets. Crop out the face area from the frame sequences of each dataset and the makeup reference image sets and save them. Perform makeup transfer on the validation set and test set after cropping the face area.

[0095] In this embodiment, the dataset is first divided into a training set, a validation set, and a test set. The Oulu-NPU database is divided into training, validation, and test sets in a ratio of 4:3:4, the CASIA-FASD database is divided into training, validation, and test sets in a ratio of 2:1:2, the Replay attack database is divided into training, validation, and test sets in a ratio of 4:3:4, and the MSU-MFSD database is divided into training, validation, and test sets in a ratio of 3:2:2.

[0096] Five makeup images are selected from the public dataset MT-Dataset in the field of makeup migration as the makeup reference image set.

[0097] In this embodiment, the VideoCapture class in the OpenCV open source software library is used to decode the videos of each data set, and the video is decoded into a frame sequence. The Dlib library face detector get_frontal_face_detector is used to perform face recognition on each frame of the makeup reference image and the video, and the four coordinate values ​​of the face area box are extracted. The video frame is cropped with the face area box and resized to 256×256. The cropped face image set of the makeup reference image set is saved in the same folder, and the cropped face image set of each video is saved in a separate folder to prevent interference between different videos.

[0098] For the bare-faced real face images in the Oulu-NPU database's validation and test sets, makeup was transferred using the SpMT and EleGANt methods, referencing high-definition makeup-appearing face images collected online. The generated makeup-appearing face images were resized to 256×256. The makeup-appearing face image sets generated for each video were saved in separate folders and added to the validation and test sets of the Oulu-NPU database. The same process was performed on the CASIA-FASD, Replayattack, and MSU-MFSD databases, using different high-definition makeup-appearing face images as reference. The number of videos processed for makeup transfer for each database is shown in Table 1.

[0099] Table 1 Number of videos processed for makeup migration in each database

[0100]

[0101]

[0102] In this embodiment, the training set only includes real faces without makeup (defined in this embodiment as real faces without makeup) and fraudulent faces, and the validation set and test set include real faces without makeup, fraudulent faces and makeup faces.

[0103] S2: Detect facial landmarks in the makeup reference image set and the training set after cropping the face area, and save each image as a facial landmark file;

[0104] In this embodiment, the makeup reference image set after the face area is cropped and the image path in the training set are traversed and read, and the face is recognized on the image through the Dlib library face detector get_frontal_face_detector, and the face landmarks are extracted. In this embodiment, the number of face landmarks is selected as 68, and the coordinates of the face landmarks of each image are written into a separate npy file. The face landmark files of all makeup reference images are saved in the same folder, and the face landmark files of each video face image in the training set are saved as separate folders;

[0105] Write the video folder path where facial images are stored, the video folder path where facial landmark files are stored, the video true or false label, and the label of whether the video is wearing glasses into the json file.

[0106] S3: Read the data and perform operations such as augmentation, random cropping, scaling, and normalization on the data;

[0107] S31: Read the training set image after cropping the face area and its corresponding face landmark point file;

[0108] In this embodiment, the json file is first read, and the face images in the training set video folder after the face area is cropped are read according to the video folder path where the face images are stored. In order to ensure a balanced ratio of true and false category samples and samples from different databases, the number of face images read from each video folder in the true and false categories of different databases is different. First, the category with the most videos in the database with the most videos is selected. In each iteration, only one face image is read from each video folder of this category in this database; if the number of videos in the category with the most videos is H, most , use H most Divide by the number of videos in some other category H other , and get the quotient H quotient Sum remainder H remainder, for each iteration, first read H quotient face images from each video folder of this other category, and then randomly select H remainder videos from the videos of this other category, and read 1 face image from the H remainder video folders randomly selected. After each face image is read, according to the video folder path where the face landmark file is stored and the name of the sampled face image, read the corresponding face landmark file.

[0109] S32: As Figure 2 shown, select some real samples from the training set with the face region cropped out, and use the makeup reference image after cropping out the face region as a reference to perform makeup augmentation to generate a makeup-augmented image set;

[0110] Judge whether the face image in the training set after cropping out the face region is a real face according to the true and false labels, and judge whether the face image is a face wearing glasses according to the whether-wearing-glasses label, and perform an enhancement operation on the face image with probability t;

[0111] Randomly generate a probability value t, 0 < t < 1. If it is a real face and t < 0.1, then perform lip makeup augmentation on the real face image I r of natural face; if it is a real face without glasses and 0.1 ≤ t < 0.35, then perform lip and eye makeup augmentation on the real face image I r of natural face; if it is a real face without glasses and 0.35 ≤ t < 0.6, then perform eye makeup augmentation on the real face image I r of natural face.

[0112] For each real face image I r of natural face selected for makeup augmentation, select a makeup reference face image I mk_refer (hereinafter referred to as I mr ) from the makeup reference image set after cropping out the face region as a reference to perform makeup augmentation.

[0113] First perform TPS (Thin Plate Spline) transformation on the makeup reference face image I mr . TPS transformation is a non-rigid deformation model. Its goal is to solve a function f such that the control point coordinates are transformed to the reference point coordinates, and the bending energy function is minimized. At the same time, other pixel points on the image can also be well corrected through interpolation. The specific steps are as follows:

[0114] In this invention, the face landmarks of I [[ID=​​The facial landmarks are used as the control points of TPS transformation (hereinafter referred to as control points). The control points are mapped to the positions of the reference points through affine transformation. The specific calculation formula of affine transformation is:

[0115]

[0116] Where m is the number of facial landmarks. In this embodiment, m is 68, (x′ i ,y′ i ) represents the coordinates of the control points after affine transformation, (x i ,y i ) represents the coordinates of the control points, (a1, a2, a3, a4, a5, a6) are the parameters of the affine transformation, which can be obtained by least squares fitting. The objective function formula of the least squares method is as follows:

[0117]

[0118] in, Indicates the reference point coordinates.

[0119] Then, other pixels are corrected by interpolation. The specific calculation formula of the TPS interpolation function is:

[0120]

[0121] Among them, b1, b2, b3, w i is the parameter to be sought, is the radial basis function, where r ij Represents the Euclidean distance between the control point and other pixel points, i is the index value of the control point, and j is the index value of other pixel points.

[0122] The coordinates of the control points after affine transformation (x′ i ,y′ i ) into the standard interpolation function of TPS, we can solve b1, b2, b3, w i , thereby obtaining the mapping of the makeup reference face image to other pixels of the plain face image.

[0123] Next, according to the probability t and the label of whether wearing glasses, the makeup reference face image I after TPS transformation is transformed mr_TPS and real face images without makeup I r Perform Poisson fusion to generate makeup augmented image I mk_aug , the specific steps are as follows:

[0124] Generate the mask used for Poisson fusion, the mask size is the same as I mr_TPS Same, it is a binary mask. mr_TPSThe minimum external matrix of the left eye marker set of the face markers is taken, the center position of the external matrix frame is kept unchanged, the minimum external matrix frame of the left eye is enlarged to 2 times the original, and then a matrix is ​​created with I mr_TPS A completely black image of the same size (defined in this embodiment as an image with all pixels (0, 0, 0)) is used as the initial mask, and the pixels in the left eye minimum bounding matrix frame are set to (1, 1, 1) to generate the left eye bounding matrix area mask M. left_eye_rect ; Connect the left eye marker point set to obtain an irregular left eye marker point connection area, and create a mr_TPS The black image of the same size is used as the initial mask, and the pixels in the left eye marker connection area are set to (1, 1, 1) to generate the left eye marker connection area mask M left_eye_in ; for M left_eye_rect and M left_eye_in Perform bitwise XOR operation to obtain the left eye area mask M left_eye Similarly, we get the right eye area mask M right_eye , M left_eye and M right_eye Perform a bitwise OR operation to obtain the eye area mask M eyes . to I mr_TPS The mouth marker set of the face markers takes the minimum circumscribed matrix, takes the center of the circumscribed matrix as the center of the ellipse, takes the length and width of the circumscribed matrix as the major axis and minor axis of the ellipse, and makes an ellipse. Create a mr_TPS The black image of the same size is used as the initial mask, and the pixels in the elliptical area are set to (1, 1, 1) to generate the mask M of the elliptical area outside the mouth. mouth_ellipse ; Connect the internal point set of the mouth to obtain the irregular internal mark point connection area of ​​the mouth, and create a mr_TPS The black image of the same size is used as the initial mask, and the pixels in the area where the internal marking points of the mouth are connected are set to (1, 1, 1), generating the mask M of the area where the internal marking points of the mouth are connected. mouth_in ; M mouth_ellipse and M mouth_in Perform bitwise XOR operation to obtain the lip area mask M lips 。 eyes and M lips Perform bitwise OR operation to obtain the eye and lip area mask M eyes_lips .

[0125] According to the probability t and the label of whether to wear glasses, from M eyes 、M lips and M eyes_lips Select a mask and apply the makeup reference face image I mr_TPS and real face images without makeup I r Poisson fusion is an image processing technique used to adjust Imr_TPS The pixel value of , so that the edge of the fusion area maintains a smooth transition, the specific calculation formula of the objective function is:

[0126]

[0127] Among them, I mr_TPS (x,y) is I mr_TPS The pixel value, I r (x,y) is the pixel value of the plain face image, Represents the gradient operation on the image, ofset(x,y) is I mr_TPS Relative to I r offset.

[0128] S33: Randomly crop, scale, randomly flip, and normalize the un-makeup-augmented images and the makeup-augmented images in the training set to obtain feature extraction regions. The specific steps include:

[0129] The images without makeup augmentation and the images with makeup augmentation in the training set are randomly cropped, with the cropping scale ranging from 20% to 100% of the original image, and the aspect ratio of the cropping frame is kept at 1 to obtain a cropped image; the cropped image is scaled and uniformly adjusted to an H×W×3 image, where H is the height of the uniformly adjusted image and W is the width of the uniformly adjusted image. In this embodiment, the value of H is preferably 256 and the value of W is preferably 256; the uniformly adjusted image is randomly flipped and normalized to obtain a feature extraction area.

[0130] S4: The feature extraction regions of different databases are input into the ResNet18 network improved by the batch channel normalization module in batches for feature extraction. The specific steps are as follows:

[0131] Replace the batch normalization layer in the ResNet18 network with the batch channel normalization module to improve the ResNet18 network. The batch channel normalization module is as follows: Figure 3 As shown in Figure 1, the input data is normalized along the channel dimension and the batch dimension, and the two normalized outputs are adaptively weighted as the final output.

[0132] During training, we first calculate the mean μ1 and variance of the layer input along the (N, H, W) axis. Next, calculate the mean μ2 and variance along the (C, H, W) axis The specific calculation formula is:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] Among them, q r For input, are respectively μ1, and μ2, The normalized value ∈ is a small constant used to maintain numerical stability. In this embodiment, ∈ is set to 0.00001.

[0140] Then the two normalized outputs are adaptively weighted as the final output. The specific calculation formula is:

[0141]

[0142] Among them, Y is the final output, τ, γ and is a learnable parameter, τ is used to adaptively balance the normalized output along the (N, H, W) and (C, H, W) axes, and γ is used to scale the normalized values. For translation, gamma and The learning of allows the model to learn how to better adapt to different input distributions.

[0143] The improved ResNet18 network is used as the backbone network. The feature extraction areas of different database data are input into the backbone network in batches for feature extraction. The output tensor of the last layer of the average pooling layer of the network is selected as the extracted features F1, F2, F3...F E .

[0144] S5: Use the features extracted from different databases to train the binary classifiers respectively, obtain the best true and false classification hyperplanes for different databases, and calculate the global cross entropy loss L cls , optimize the best true and false classification hyperplane of different databases through the projected gradient method, make it converge to a global true and false classification hyperplane, and output the binary true and false prediction labels;

[0145] In this embodiment, different database data are represented as ε={e (1) ,e (2) ,...,e (E)}, use the features extracted from different database data to train the binary classifiers respectively, and obtain the best true and false classification hyperplane for each database

[0146] like Figure 4As shown, for each database e∈ε, the feature F extracted from the database data is used e and true and false labels to train a binary classifier and obtain the best true and false classification hyperplane β for the database e , for each database e∈ε, calculate its cross entropy loss function L cls_e , the specific calculation formula is:

[0147]

[0148] Among them, p z Represents a predicted probability value in a batch, y z Indicates the value of the true or false label, p z (y z ) indicates the prediction is label y z probability.

[0149] The cross entropy loss function of different databases is averaged to obtain the global cross entropy loss function L cls , and its specific calculation formula is:

[0150]

[0151] Where E represents the number of databases.

[0152] Calculate the global cross entropy loss L cls For β e To ensure that the true and false hyperplanes of each updated database are still consistent with the global hyperplane, this embodiment adopts the projected gradient method to update the hyperplane parameters. For each database e∈ε, the hyperplane β that is consistent with the current update is selected. e The farthest hyperplane Use the interpolation method to update the current hyperplane β e and the farthest hyperplane Interpolation updates are performed to push the hyperplanes closer to each other, and the feature space parameters are updated using the gradient descent method. The specific calculation formula is:

[0153]

[0154]

[0155] Among them, θ is the target point, is the loss function, ξ is the learning rate, P Ω () is the Euclidean projection, Ω is the feasible set, which represents the value range of the hyperplane β, ensuring that the true and false hyperplanes of each database after the update are still consistent with the global hyperplane, and η is the alignment parameter used to adjust the current updated hyperplane β e Perform interpolation updates.

[0156] After optimizing the hyperplanes of different databases, the converged global true and false classification hyperplane is used to output the true and false binary classification labels.

[0157] S6: Establish a cache area to store sample feature representations. In order to make the intra-class features of different databases and their true and false categories more compact, for the new sample features extracted by the backbone network, use the KNN (K-Nearest Neighbors) algorithm to select the top-K samples most similar to it from the cache area, and select samples from the same database and the same category as the current sample as anchor points to calculate the neighbor supervision contrast loss L supin , maximize the similarity between the anchor point and the current sample. The specific steps include:

[0158] S61: Establish a buffer area to store sample feature representation;

[0159] like Figure 5 As shown in FIG, at the beginning of training, an empty buffer is initialized to store sample feature representations. In this embodiment, the buffer can store 1080 sample feature representations. During the training process, in order to ensure that the sample feature representations entering the buffer are all high-quality sample feature representations, the network Φ is updated using the EMA (Exponential Moving Average) method. EMA Perform feature extraction and transform the network Φ EMA The extracted sample features are stored in the cache, Φ EMA The network architecture is the same as the backbone network, and the EMA method is used to update the parameters. That is, each time the parameters are updated, the model parameters will be processed by EMA, thereby reducing the fluctuation of each update and making the model more stable. The specific calculation formula is as follows:

[0160] Φ EMA ←(1-θ)Φ EMA_old +θΦ grad

[0161] Among them, Φ EMA_old is the network whose parameters are updated in EMA mode after the last update, Φ grad is the backbone network, which uses the gradient descent method to update parameters, Φ EMA is the network updated by EMA, θ is the momentum coefficient, and in this embodiment, θ is 0.996.

[0162] The buffer is a queue data structure. New features enter the buffer from the end of the queue. If the buffer is full at this time, the head feature leaves the buffer.

[0163] S62: For each new sample, use the KNN algorithm to select the top-K samples most similar to each sample from the cache, and select samples of the same database and type as the current sample as anchor points to calculate the nearest neighbor supervision contrast loss L supin , maximize the similarity between the anchor point and the current sample;

[0164] First, the dot product of the feature representation of the sample and the feature representation of the samples in the cache area is calculated to obtain a similarity matrix. Next, based on the similarity matrix, the top K samples most similar to the new sample are selected from the cache area, and all elements in the similarity matrix except the elements corresponding to these K samples are set to 0. In this embodiment, K is 256.

[0165] After selecting K samples, the similarity matrix is ​​normalized. According to the true and false category labels and database labels, N samples of the same type as the new sample and in the same database are selected as anchor points. The nearest neighbor supervision contrast loss is calculated. The specific calculation formula is:

[0166]

[0167] Among them, g u is the weight of the anchor point in the similarity matrix, and u is the index value of the anchor point.

[0168] S7: To make the intra-class features of different true and false categories more compact and to increase the distance between features of different categories, two fixed orthogonal vectors are constructed to form a virtual triplet together with the features extracted by the backbone network. The triplet loss L of the virtual triplet is calculated. dmyt , the specific steps include:

[0169] Construct two fixed vectors d c and d 1-c , the two are orthogonal, d c and d 1-c The length is consistent with the length of the feature vector extracted from a single sample and remains unchanged throughout the training process. c and d 1-c Instead of the positive and negative examples in the classic triples, virtual triples are constructed together with the extracted features. c The first half of the elements are set to 0, the second half are set to 1, and d 1-c The first half of the elements are set to 1, and the second half are set to 0. If d c and d 1-c If the length of is odd, then d c and d 1-c The middle element is set to 0.

[0170] The specific calculation formula for virtual ternary loss is:

[0171] L dmyt(F z ,c)=max(2F z ·d 1-c -2F z ·d c +ρ,0)

[0172] Among them, F z represents the features extracted from sample z, c represents the true and false categories, c∈{0,1}, and ρ is the marginal interval of the loss function, which is used to ensure that the distance between the true sample and the fraudulent sample is greater than a certain threshold. In this embodiment, ρ=1.

[0173] S8: Calculate the loss function of the combination of global cross entropy loss, neighbor supervision contrast loss, and virtual ternary loss. The specific calculation formula is:

[0174] L=L cls +ω1×L supin +ω2×L dmyt

[0175] Wherein, ω1 and ψ2 are weights of corresponding losses. In this embodiment, both ψ1 and ψ2 are set to 1.

[0176] S9: Iteratively update the network parameters of the model, train the model and save the model;

[0177] In this example, the optimizer is set to SGD, with an initial learning rate of 0.005, a first-order smoothing parameter of 0.9, and a weight decay parameter of 0.0005. The learning rate scheduler torch.optim.lr_scheduler.StepLR in PyTorch is used to adjust the learning rate, adjusting it every 40 epochs, multiplying the current learning rate by 0.5 each time. The model's network parameters are iteratively updated using a loss function that combines global cross-entropy loss, virtual ternary loss, and nearest neighbor supervised contrast loss. The training period is 100, and the batch size is 90. After training, the model with the lowest loss on the validation set is saved.

[0178] S10: Model application: Load the model saved in the model training step as the test model, select 7 frames from each video folder of the test set at equal intervals to crop the face area, and convert the real face image without makeup I real (hereinafter referred to as I r ), fraudulent face image I spoofing (hereinafter referred to as I s ) and makeup face image I makeup (hereinafter referred to as I m ) is fed into the trained model to obtain the binary prediction probability and output the final prediction result. It is expected that the model can r ,I m} is determined to be a real face, and I s Determined to be a fraudulent face;

[0179] In this embodiment, a backbone network model trained with multiple database training sets is loaded as a test model for testing; the performance indicators of the face fraud detection algorithm in this embodiment are the half total error rate (HTER) and the area under the receiver operating characteristic curve (ROC) (Area Under Curve, AUC). The ROC curve is a curve drawn based on a series of different thresholds with the true positive rate as the vertical axis and the false positive rate as the horizontal axis. The formula for calculating the half error rate (HTER) is as follows:

[0180]

[0181] Among them, FAR (False Acceptance Rate) is the false acceptance rate, that is, the ratio of the algorithm to identify fraudulent attack faces as real faces; FRR (False Rejection Rate) is the false rejection rate, that is, the ratio of the algorithm to identify real faces as fraudulent attack faces.

[0182] In order to prove the feasibility of the present invention and to test the generalization performance of the method, the method of the present invention is compared with the SAFAS method which also considers the alignment and separation of feature spaces, and Oulu-NPU (hereinafter referred to as O), CASIA (hereinafter referred to as C), Replay attack (hereinafter referred to as I) and MSU-MFSD (hereinafter referred to as M) are used to train and cross-library test the makeup face database generated from the O, C, I, and M databases by the makeup migration method. The specific method is to train on three databases and test on another database. For example, OCI to M means that the model trained on the training sets of the three databases O, C, and I is tested on the test set of the M database and the makeup face image set generated by makeup migration on the test set of the M database; the same is true for OCM to I, ICM to O, and OMI to C. This embodiment will be in the database containing {I r ,I s}, the model trained on the training set containing {I r ,I s ,I m}, and expect the model to be able to r ,I m} is determined to be a real face, and I s It is determined to be a fraudulent face. The cross-database test results are shown in Table 2:

[0183] Table 2 Cross-database test results

[0184]

[0185] As shown in Table 2, the experimental results demonstrate that, compared to the SAFAS method (which also considers feature space alignment and separability), the present invention reduces the HTER by approximately 2% and improves the AUC by approximately 2% in the OCI to M cross-database experiment; reduces the HTER by approximately 7% and improves the AUC by approximately 7% in the OCM to I cross-database experiment; reduces the HTER by approximately 4% and improves the AUC by approximately 4% in the OCI to M cross-database experiment; and reduces the HTER by approximately 3% and improves the AUC by approximately 3% in the ICM to O cross-database experiment. This demonstrates that the present invention can effectively improve the generalization performance of face fraud detection models and is also effective for faces with makeup.

[0186] This embodiment uses only a small number of makeup-appearing face images to augment the input image, enabling the model to learn the characteristics of makeup-appearing faces. Furthermore, batch channel normalization uses adaptive parameters to dynamically adjust the normalization method to better adapt to different datasets and models. Finally, through nearest neighbor supervision comparison and calculation of a virtual ternary loss, the distance between similar samples is shortened and that between dissimilar samples is increased. Experimental results demonstrate that this invention improves the performance of face fraud detection systems.

[0187] Example 2

[0188] like Figure 6 As shown, this embodiment provides a fraud detection system for makeup faces, which is used to implement the fraud detection method for makeup faces in the above embodiment. The system includes: data set division, video decoding module, face region extraction module, face marking module, makeup augmentation module, feature extraction module, different database hyperplane alignment module, buffer area construction module, nearest neighbor supervision comparison module, virtual triple construction module, virtual triple loss calculation module, total loss function construction module, model training module, and prediction module;

[0189] In this embodiment, data set partitioning is used to partition the data set;

[0190] In this embodiment, the video decoding module is used to decode the video of each data set into a frame sequence;

[0191] In this embodiment, the face region extraction module is used to crop the face regions of each data set frame sequence and the selected makeup reference image set;

[0192] In this embodiment, the face marking module is used to detect facial landmarks on the cropped face area;

[0193] In this embodiment, the makeup augmentation module is used to perform makeup augmentation by selecting some real face images without makeup from the training set with cropped face areas, and using the makeup reference face images with cropped face areas as references to generate a makeup augmented image set.

[0194] In this embodiment, the feature extraction module is used for feature extraction. Feature extraction regions are selected, and the improved ResNet18 network with batch channel normalization is used as the backbone network. Feature extraction regions of different databases are input into the backbone network in batches for feature extraction to obtain features of different databases.

[0195] In this embodiment, the different database hyperplane alignment module is used to construct the global optimal true and false classification hyperplane. Based on the features extracted from different databases, the binary classifiers are trained separately to obtain the optimal true and false classification hyperplanes for different databases. The global cross entropy loss is calculated. The optimal true and false classification hyperplanes for different databases are optimized by the projected gradient method and converged to the global true and false classification hyperplane, and the binary classification true and false prediction labels are output.

[0196] In this embodiment, the buffer area construction module is used to establish a buffer area for storing sample feature representations;

[0197] In this embodiment, the nearest neighbor supervision comparison module is used to calculate the nearest neighbor supervision comparison loss between the anchor point and the new sample. For the new sample features extracted by the backbone network, the top-K samples in the cache that are most similar to it are calculated. From these samples, samples of the same type and in the same database as the current sample are selected as anchor points to maximize the similarity between the current sample and the anchor points.

[0198] In this embodiment, the virtual triplet construction module is used to construct a virtual triplet based on the two constructed fixed orthogonal vectors and the features extracted by the backbone network;

[0199] In this embodiment, the virtual ternary loss calculation module is used to calculate the virtual ternary loss of the virtual triplet;

[0200] In this embodiment, the total loss function construction module is used to obtain the total loss function by weighted summation of the global cross entropy loss, the nearest neighbor supervision contrast loss, and the virtual ternary loss;

[0201] In this embodiment, the model training module is used to obtain a prediction model based on the total loss function training;

[0202] In this embodiment, the prediction module is used to input the face image test set into the trained prediction model, obtain the binary classification prediction probability, and output the final prediction result.

[0203] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A fraud detection method for a face wearing makeup, characterized in that: The steps include: Divide the data set and decode the video of each data set into a frame sequence; Crop out the face regions of each dataset frame sequence and the selected makeup reference image set, and perform facial landmark detection; Select some real face images without makeup from the training set with cropped face areas, and use the makeup reference face images after cropping the face areas as references to perform makeup augmentation to generate a makeup augmented image set; Select the feature extraction area, use the ResNet18 network improved by the batch channel normalization module as the backbone network, input the feature extraction areas of different databases into the backbone network in batches for feature extraction, and extract the features of different databases; Based on the features extracted from different databases, the binary classifiers are trained separately to obtain the global optimal true and false classification hyperplanes for different databases. The global cross entropy loss is calculated. The optimal true and false classification hyperplanes for different databases are optimized by the projected gradient method and converged to the global true and false classification hyperplane. The binary classification true and false prediction labels are output. Establish a cache area to store sample feature representations. For new sample features extracted by the backbone network, calculate the top-K samples in the cache area that are most similar to the current sample. Select samples from the same database and of the same type as the current sample as anchor points, and calculate the nearest neighbor supervision contrast loss between the anchor points and the new sample. Construct two fixed orthogonal vectors, form a virtual triplet with the features extracted by the backbone network, and calculate the virtual triplet loss of the virtual triplet; The total loss function is obtained by weighted summation of global cross entropy loss, neighbor supervision contrast loss and virtual ternary loss, and the prediction model is trained based on the total loss function; Input the face image test set into the trained prediction model, obtain the binary classification prediction probability, and output the final prediction result.

2. The fraud detection method for a face wearing makeup according to claim 1, characterized in that: Perform makeup augmentation to generate a makeup augmented image set, specifically including: Perform TPS transformation on the makeup reference face image, using the facial landmarks of the real face image without makeup as the reference points of the TPS transformation, and the facial landmarks of the makeup reference face image as the control points of the TPS transformation. The control points are mapped to the positions of the reference points through affine transformation, and other pixels are corrected based on TPS interpolation; Substitute the coordinates of the control points after affine transformation into the interpolation function of TPS to obtain the mapping of other pixels of the makeup reference face image to the natural face image; The makeup reference face image after TPS transformation and the real face image without makeup are Poisson fused to generate the makeup augmented image.

3. The fraud detection method for a face wearing makeup according to claim 2, characterized in that: The step of performing Poisson fusion on the makeup reference face image after TPS transformation and the real face image without makeup to generate the makeup augmented image specifically includes: Generate an eye region mask and a lip region mask, and perform a bitwise OR operation on the eye region mask and the lip region mask to obtain an eye and lip region mask; Determine whether the face image is a real face based on the true and false labels, and determine whether the face image is a face wearing glasses based on the glasses label; Randomly generate a probability value and set a probability value threshold, and perform lip makeup augmentation, lip and eye makeup augmentation, and eye makeup augmentation based on the probability value threshold and the face image type; Based on different makeup augmentations, the corresponding region mask is selected and Poisson fusion is performed on the makeup reference face image and the real face image without makeup. The specific calculation formula of the objective function is: Among them, I mr_TPS (x, y) is the pixel value of the makeup reference face image, I r (x,y) is the pixel value of the plain face image, represents the gradient operation on the image, and offset(x,y) is the offset of the makeup reference face image relative to the real face image without makeup.

4. The fraud detection method for a face wearing makeup according to claim 1, characterized in that: The ResNet18 network improved by the batch channel normalization module is used as the backbone network. The feature extraction regions of different databases are input into the backbone network in batches for feature extraction. Specifically, the following steps are performed: Improve the ResNet18 network by replacing the batch normalization layer in ResNet18 with the batch channel normalization module. The batch channel normalization module normalizes the input data along the channel dimension and batch dimension respectively, and calculates the mean μ1 and variance of the layer input along the (N, H, W) axis. Calculate the mean μ2 and variance along the (C, H, W) axis Specifically expressed as: Among them, q r For input, are respectively μ1, and μ2, Normalized, ∈ is a constant, N represents the batch, C represents the channel, H represents the feature height, and W represents the feature width; The two normalized outputs are adaptively weighted as the final output. The specific calculation formula is: Among them, Y represents the extracted features, τ, γ and is a learnable parameter, τ is used to adaptively balance the normalized output along the (N, H, W) and (C, H, W) axes, and γ is used to scale the normalized values. Used for translation.

5. The fraud detection method for a face wearing makeup according to claim 1, characterized in that: The calculation of the global cross entropy loss is specifically expressed as: Among them, L cls represents the global cross entropy loss function, L cls_e represents the cross entropy loss function, e represents the database, E represents the number of databases, and p z Represents a predicted probability value in a batch, y z Indicates the value of the true or false label, p z (y z ) indicates the prediction is label y z probability.

6. The method for detecting fraudulent faces wearing makeup according to claim 1, wherein: The projected gradient method is used to optimize the optimal true and false classification hyperplanes for different databases and converge to a global true and false classification hyperplane. Specifically, the following steps are involved: Calculate the gradient of the global cross entropy loss for different database hyperplanes. For each database e, select the hyperplane β that is consistent with the current update. e The farthest hyperplane Based on the interpolation method, the currently updated hyperplane β e and the farthest hyperplane Perform interpolation update and use the gradient descent method to update the feature space parameters, which can be expressed as: Among them, θ is the target point, is the loss function, ξ is the learning rate, P Ω () is the Euclidean projection, Ω is the feasible set, which is used to describe the value range of the hyperplane β, and η is the alignment parameter used to adjust the current updated hyperplane β e Perform interpolation updates.

7. The method for detecting fraudulent faces wearing makeup according to claim 1, wherein: Establish a cache area to store sample feature representations, specifically including: At the beginning of training, an empty buffer is initialized; during the training process, the network Φ is updated using the EMA method. EMA Extract features from samples and transform the network Φ EMA The extracted sample features are stored in the cache, Φ EMA The network architecture is the same as the backbone network, and the EMA method is used to update the parameters. The specific calculation formula is as follows: Among them, Φ EMA_old is the network whose parameters are updated in EMA mode after the last update, Φ grad is the backbone network, which uses the gradient descent method to update parameters, Φ EMA For the network updated by EMA, is the momentum coefficient; The buffer is a queue data structure. New features enter the buffer from the end of the queue. If the buffer is full, the head feature leaves the buffer.

8. The method for detecting fraudulent faces wearing makeup according to claim 1, wherein: For the new sample features extracted by the backbone network, calculate the top-K most similar samples in the cache, select samples from the same database and the same type as the current sample as anchor points, and calculate the nearest neighbor supervision contrast loss between the anchor points and the new sample, specifically including: Calculate the dot product between the new sample feature representation and the sample feature representation in the cache area to obtain the similarity matrix; Based on the KNN algorithm, the first K samples that are most similar to the new sample are selected from the cache, and all elements in the similarity matrix except the elements corresponding to these K samples are set to 0; After selecting K samples, the similarity matrix is ​​normalized. According to the true and false category labels and database labels, N samples of the same type as the new sample and in the same database are selected as anchor points. The nearest neighbor supervision contrast loss is calculated. The specific calculation formula is: Among them, g u is the weight of the anchor point in the similarity matrix, and u is the index value of the anchor point.

9. The method for detecting fraudulent faces wearing makeup according to claim 1, wherein: Construct two fixed orthogonal vectors to form a virtual triplet with the features extracted by the backbone network, and calculate the virtual triplet loss of the virtual triplet, which is specifically expressed as: Construct two fixed orthogonal vectors d c and d 1-c , d c and d 1-c The length is consistent with the length of the feature vector extracted from a single sample; The specific calculation formula of the virtual ternary loss is: L dmyt (F z ,c)=max(2F z ·d 1-c -2F z ·d c +p,0) Among them, F z Represents the features extracted from a single sample, c represents the true and false categories, and ρ is the marginal interval of the loss function.

10. A fraud detection system for faces wearing makeup, characterized in that: include: Dataset division, video decoding module, face region extraction module, face marking module, makeup augmentation module, feature extraction module, hyperplane alignment module for different databases, cache construction module, nearest neighbor supervision comparison module, virtual triplet construction module, virtual triplet loss calculation module, total loss function construction module, model training module, prediction module; The data set partitioning is used to partition the data set; The video decoding module is used to decode the video of each data set into a frame sequence; The face region extraction module is used to crop the face regions of each data set frame sequence and the selected makeup reference image set; The face marking module is used to detect face landmarks on the cropped face area; The makeup augmentation module is used to perform makeup augmentation, selecting some real face images without makeup from the training set with cropped face areas, and using the makeup reference face images after cropping the face areas as references to generate a makeup augmented image set through makeup augmentation; The feature extraction module is used for feature extraction, selecting a feature extraction area, using an improved ResNet18 network with batch channel normalization as a backbone network, and inputting the feature extraction areas of different databases into the backbone network in batches for feature extraction to obtain features of different databases; The different database hyperplane alignment module is used to construct a global optimal true and false classification hyperplane, train binary classifiers based on the features extracted from different databases, obtain the optimal true and false classification hyperplanes for different databases, calculate the global cross entropy loss, optimize the optimal true and false classification hyperplanes for different databases through the projected gradient method and converge to the global true and false classification hyperplane, and output the binary classification true and false prediction labels; The buffer area construction module is used to establish a buffer area for storing sample feature representations; The nearest neighbor supervision comparison module is used to calculate the nearest neighbor supervision comparison loss between the anchor point and the new sample. For the new sample features extracted by the backbone network, the top-K samples in the cache area that are most similar to it are calculated, and samples of the same database and type as the current sample are selected as anchor points to maximize the similarity between the current sample and the anchor point. The virtual triplet construction module is used to construct a virtual triplet based on the two constructed fixed orthogonal vectors and the features extracted by the backbone network; The virtual ternary loss calculation module is used to calculate the virtual ternary loss of the virtual triplet; The total loss function construction module is used to obtain the total loss function according to the weighted sum of the global cross entropy loss, the nearest neighbor supervision contrast loss and the virtual ternary loss; The model training module is used to obtain a prediction model based on total loss function training; The prediction module is used to input the face image test set into the trained prediction model, obtain the binary classification prediction probability, and output the final prediction result.

Images