An AR-MLP-based multi-modal face anti-fraud detection method

Abstract

The application discloses a multi-modal face anti-fraud detection method based on AR-MLP, which comprises the following steps: performing face image processing on input face RGB images, face depth images and face infrared images to realize face multi-modal feature fusion, stacking and coupling an input AR-MLP network, performing transposition and shaping, realizing feature extraction of different face area patches and forming an input tensor; sending the feature tensor into an ACMix integrated with convolution and attention; after mixing the MLP of each block and the global MLP across blocks, the mixture is sent into a classifier for feature classification to obtain a final classification result of real face or fake face, and a face anti-fraud detection model is obtained, which has the technical features of being capable of stacking and fusing the features after inputting multi-modal images into the network.

Description

Technical Field

[0001] This invention relates to a multimodal face anti-fraud detection method, and more specifically, to a multimodal face anti-fraud detection method based on AR-MLP, belonging to the petrochemical field. Background Technology

[0002] The face detection system consists of two parts. First, it performs liveness detection—detecting whether the face in front of the camera is a real face or an attacker's face. Once it is detected as a real face, it performs face recognition—that is, it verifies the identity of the person in front of the camera to confirm whether it is the person in question.

[0003] Facial recognition technology is becoming increasingly mature and has been widely and successfully applied to various identity authentication tasks. Various face detection systems have been deployed on identity verification and recognition machines and face detection systems in various devices. This has led to a surge in various facial attacks, such as the most common printed photo attacks, video attacks, and 3D mask attacks including rigid plastic masks, flexible silicone masks, and paper masks, as well as attacks involving glasses obscuring the face, fake head models, and makeup attacks. These attacks pose a significant threat to the security of facial recognition systems.

[0004] Initially, face fraud detection typically employed machine learning methods that manually extracted features. These methods generally included color and texture features such as LBP, MBSIF-TOP, MLPQ-TOP, CoALBP, illumination intensity features such as LDN and its variant LDN-TOP, as well as features like DoG and SIFT. Motion-based methods, such as LBP-TOP, were also used, and some methods incorporated LPQ features to handle spatially blurred images. These methods generally used traditional classifiers such as SVM and ProCRC. However, these feature extraction algorithms and classification methods have several drawbacks, such as limited attack types they can handle and low accuracy in cross-domain testing.

[0005] Therefore, deep learning-based face fraud detection algorithms are gaining increasing popularity. Li et al. (Li JC, Hassani A, Walton S, et al. ConvMLP: Hierarchical Convolutional MLPs for Vision [EB / OL].arXiv:2109.04454v2 [cs.CV], 2021.) found that schemes composed of multilayer perceptrons (MLPs) have low flexibility and are computationally cumbersome in fully connected layers. Therefore, they proposed a scalable co-design scheme combining CNNs and MLPs, adding convolutional layers, downsampling, and MLPs in the early stages of CNNs to compensate for the lack of spatial interaction and the problem of shared weights in channel-based MLPs. Although CNNs have a strong ability to extract local features, due to the limitation of the receptive field, they have always faced the problem of not being able to acquire long-range dependent features. Wang et al. (Wang WH, Wen F, Zheng H, et al. Conv-MLP: A Convolution and MLP Mixed Model for Multimodal Face Anti-Spoofing[J]. IEEE Transactions on InformationForensics and Security, 2022, 17: 2284-2297.) designed an architecture called Conv-MLP, which combines local CNNs and global MLPs, to address the inherent inductive bias of CNNs. It first performs convolution on each block to extract local features, and then performs cross-block mixing on different blocks to obtain the long-distance global correlation between different blocks. This combination can not only make full use of the ability of CNNs to capture local features, but also solve the problem of limited receptive field that has always existed in CNNs.

[0006] Dosovitskiy et al. (Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image isworth 16×16 words: Transformers for image recognition at scale[C] / / International Conference on Learning Representations,2021:1–22.) designed the Vision Transformer (ViT) framework to address the characteristics of the image vision domain. George et al. (George A, Marcel S. On the effectiveness of vision transformers for zero-shot face anti-spoofing[C] / / IEEE International Joint Conference on Biometrics,2021:1–8.) were the first to apply ViT to the face anti-spoofing detection task. After incorporating zero-shot learning, they fine-tuned the training parameters of the ViT model and achieved good test performance on two authoritative face anti-spoofing detection datasets. However, ViT has the problem of requiring a huge amount of pre-training data. Unlike previous tasks with a large number of attention mechanisms, especially in the specific inductive bias problem of CNNs, ViT only has the MLP layer which is local, translational and isovariant. When there is enough training data, it can effectively reduce the inductive bias problem in images. This also means that it has a disadvantage: it requires pre-training on a large dataset, which is very time-consuming.

[0007] With the continuous advancement of sensor manufacturing technology and the ongoing development of multispectral camera technology, the manufacturing cost of multispectral cameras has been greatly reduced, enabling multimodal images, including depth maps and infrared images, to be more widely used in various target detection and recognition tasks. George et al. (George A, Marcel S. Cross Modal Focal Loss for RGBD Face Anti-Spoofing[C] / / IEEE Conference on Computer Vision and Pattern Recognition,2021:7878-7887.) proposed a dual-stream network architecture that follows late fusion. This architecture combines the two channels of the RGB image and the depth map into a third branch, connects them with a fully connected layer to form a classification head, and designs a CMFL loss function to learn discriminative features from each channel and the joint representation, achieving good detection results.

[0008] In summary, scholars both at home and abroad have conducted extensive research on deep learning-based face fraud detection methods, which have largely solved the feature limitations caused by traditional algorithms. However, current face fraud detection schemes still suffer from the problem of inefficient fusion of multiple features. Furthermore, the generalization ability of the models needs to be enhanced. Summary of the Invention

[0009] To address the aforementioned problems in the prior art, this invention provides a multimodal face anti-fraud detection method based on AR-MLP, which features deep feature fusion, capable of stacking features after inputting multimodal images into a network.

[0010] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0011] A multimodal face fraud detection method based on AR-MLP, the detection method includes:

[0012] Step 1) Perform face image processing on the input face RGB image, face depth image and face infrared image to achieve face multimodal feature fusion, stack and couple the input to AR-MLP network, perform transpose and shape, realize the extraction of features of different face region patches and form input tensor (feature tensor);

[0013] Step 2) Feed the feature tensor into ACMix, which integrates convolution and attention;

[0014] Step 3) After mixing the MLP of each block (patch MLP) and the global MLP across blocks (cross-patch global MLP), the mixture is fed into the classifier for feature classification to obtain the final classification result of real face or fraudulent face, thus obtaining the face anti-fraud detection model.

[0015] Preferably, the face image processing includes: taking a 112×112 image I0∈R at each image input. C×H 0 ×W 0, with overlapping cuts into 9 image patches of 48×48, i.e., three modalities of face images I0∈R C×112×112 The image is divided into N=9 blocks. Based on its channel number C (for multimodal RGB, depth, and infrared, the channel number is the sum of 3, 1, and 1, which is 5; C varies accordingly for different modal inputs), the 9 divided images I... k ∈R 5×48×48 k = 0, 1, 2, 3, 4, 5, 6, 7, 8, are stacked into N = 9 overlapping blocks: I' k ∈R9×5×48×48, k=0,1,2,3,4,5,6,7,8, such asFigure 2 The 9×5×48×48 image above the AR-MLP part is the relevant parameter of the input conv1 image.

[0016] Preferably, in AR-MLP, after three rounds of convolution and MLP mixing, the above operations are not repeated in the last convolutional layer and the MLP rounds. Instead, ACmix calculation of the convolution and self-attention mixing module is performed to reduce the tedious MLP calculations repeated on the lowest feature map. Furthermore, the convolution operation of the last layer is combined with the self-attention operation to jointly calculate the final output vector.

[0017] Preferably, before face image processing, the input face RGB image, face depth image and face infrared image are cropped to a resolution of 112×112, and then processed in the face image processing. Specifically: 1) For positive and negative sample balance, the cropped 112×112 resolution images in the face RGB image, face depth image and face infrared image are cropped into 9 overlapping image blocks, and these 9 overlapping image blocks are input into the AR-MLP network to extract features;

[0018] For imbalanced positive and negative samples, the samples are first randomly flipped and rotated, then cropped to supplement the data samples and perform data augmentation. The images are then cropped into nine overlapping 48×48 image patches. These nine overlapping 48×48 image patches are all input into the AR-MLP network to extract features.

[0019] 2) For an image patch with a resolution of 48×48, a total of four rounds of operations were set up, including three rounds of convolution and MLP hybrid downsampling, and the last round of convolution and self-attention hybrid block operation. After each downsampling of the patch, the length and width of the image patch were halved. During the training of the AR-MLP model, the adaptive moment estimation optimizer algorithm was used for two rounds of training, with 50 batches per round and a batch size of 128. The learning rate lr was set to 1e-3. Finally, the initial value of the weight decay was adjusted to 1e-5, and the decay rate was set to 0.005 to prevent overfitting.

[0020] Preferably, the model parameters of the face liveness detection model are exported and deployed to an embedded device. The network structure and convolution operations between the network layers are executed using Python language to perform the forward inference process and obtain the face anti-fraud detection results.

[0021] Preferably, the specific process for obtaining facial anti-fraud detection results includes:

[0022] Save the model's weight parameters as a pth format file;

[0023] The saved weight parameter file is read using Python and imported into the embedded terminal. Python is then used to construct the network structure of the face anti-fraud detection model and perform convolution operations between network layers. The forward inference process is then executed to obtain the face anti-fraud detection results.

[0024] Preferably, the AR-MLP network architecture is implemented on PyTorch 1.8 and runs on an NVIDIA GeForce RTX 3090 with fast computing power.

[0025] Beneficial Effects: The network is trained using RGB, depth, and infrared images from a multimodal face dataset. A mid-term deep feature fusion approach is employed, stacking and coupling data from different modalities to promote mutual learning of complementary texture, geometric, and albedo features among the three modalities. Local feature extraction and cross-slice global MLP are implemented to extract long-range dependent global information. Furthermore, the final 3×3 convolutional block is innovatively replaced with an ACmix module integrating convolution and self-attention, significantly simplifying the previously independent combination of these two modules, saving computation, and combining their advantages to acquire both global and local information. This ingenious design allows the network to not only possess the good local information acquisition capabilities of CNNs but also the advantages of MLPs—low computational cost but strong spatial feature perception. With the appropriate fusion of self-attention, the network gains the ability to acquire global features. This approach is highly competitive compared to existing face fraud detection schemes. Attached Figure Description

[0026] Figure 1 A flowchart of a multimodal face anti-fraud detection method based on AR-MLP provided in an embodiment of the present invention.

[0027] Figure 2 This is a basic framework diagram of the multimodal face anti-fraud detection method based on AR-MLP of the present invention.

[0028] Figure 3 This is an example image of a multimodal face used in this invention.

[0029] Figure 4 This diagram illustrates the block-to-block interaction process of a specific channel in the cross-chip global MLP from Conv2_x to Conv4_x designed for this invention.

[0030] Figure 5 This is a diagram illustrating the operation process of the ACmix module, which integrates convolution and self-attention in this invention.

[0031] Figure 6 This is one of the schematic diagrams illustrating the face anti-fraud detection effect of the AR-MLP method provided in the embodiments of the present invention.

[0032] Figure 7 This is the second schematic diagram illustrating the face anti-fraud detection effect of the AR-MLP method provided in this embodiment of the invention.

[0033] Figure 8 The third illustration shows the face anti-fraud detection effect of the AR-MLP method provided in the embodiments of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0035] To address the issue of relatively simple features in unimodal face images, this invention performs deep feature fusion by stacking features after inputting multimodal images (face RGB image, face depth image, and face infrared image) into an AR-MLP network. After transposition and shaping, the resulting features are input into the AR-MLP network for feature classification, thereby determining whether the input face is a real face or a fake face. Furthermore, to address the problem of small receptive fields in CNNs making it difficult to establish global long-range dependencies, this invention provides an AR-MLP face liveness detection algorithm based on a hybrid convolutional block of self-attention and MLP. After multiple rounds of convolution and MLP mixing, the above operations are not repeated in the last convolutional layer and the MLP rounds. Instead, ACmix calculation of the convolution and self-attention hybrid module is performed, significantly reducing the tedious MLP calculations repeated on the lowest feature map. Furthermore, the convolution operation in the last layer is combined with the self-attention operation to jointly calculate the final output vector.

[0036] Specific technical solution:

[0037] Face preprocessing: Extract faces from images or videos (this invention includes: face RGB image, face depth image and face infrared image), use DLib to align and locate 68 key points of the face, and then extract face regions (multimodal data);

[0038] This invention processes multimodal facial data, transposing and reshaping it before inputting it into an AR-MLP network for feature classification to determine whether the input face is real or fraudulent. The AR-MLP network is designed based on ResNet18. In the second to fourth convolutional layers of ResNet18, this invention performs local feature extraction and cross-slice global MLP to extract long-range dependent global information. Furthermore, the final 3×3 convolutional block is innovatively replaced with an ACmix module that integrates convolution and self-attention (the ACmix module integrates the two based on the similarity of convolution and self-attention operations). This integration greatly simplifies the previously unrelated combination operations, saves computation, and combines the advantages of both to obtain both global and local information. This ingenious design allows the network to not only possess the good local information acquisition capabilities of CNNs but also the advantages of MLPs—low computational cost but strong spatial feature perception—and, with the appropriate fusion of self-attention, enables the network to acquire global features.

[0039] The main algorithm scheme of this invention is as follows:

[0040] Based on Conv-MLP, a network architecture called AR-MLP with low computational complexity and excellent performance was designed, mainly including:

[0041] (1) Referring to the Conv-MLP network based on ResNet18, its network architecture is decomposed, such as... Figure 1 As shown, in the convolutional layers from conv1 to conv4, local convolutional blocks are combined with a global multilayer perceptron (MLP) to extract local features and long-range global correlation features of the face image, and the parameters of the MLP are fine-tuned to output a five-dimensional vector.

[0042] (2) The idea of ​​integrating convolution and self-attention mechanism can be summarized as follows: replace the convolutional layer of conv5 of Conv-MLP with ACmix hybrid module. Unlike the conventional three-dimensional vector input of ACmix, the five-dimensional vector obtained from the first four layers is subjected to the hybrid integration operation of convolution and self-attention mechanism. Compared with the repeated MLP of conv5 of Conv-MLP, the convolutional basicblock replaced by ACmix makes the network computational cost less than that of the simple basicblock, and can combine the local feature acquisition ability of convolution and the global feature acquisition ability of self-attention mechanism by increasing the receptive field.

[0043] (3) Facial multimodal data fusion is performed within the network, i.e. at the signal level, and a large number of comparative and ablation experiments are conducted on four authoritative multimodal face fraud detection datasets (WMCA, HO-WMCA, CASIA-SURF and CASIA-SURF CEFA). The experimental results strongly demonstrate that the AR-MLP designed in this invention is highly competitive compared with existing face fraud detection schemes.

[0044] like Figure 1-8 The illustration shows a specific embodiment of a multimodal face anti-fraud detection method based on AR-MLP. This embodiment is a multimodal face anti-fraud detection method based on AR-MLP, mainly used to solve the problems of single-modal face image features and the difficulty in establishing global long-distance dependencies due to the small receptive field of CNNs. The detection method includes:

[0045] Step 1) First, crop the input face RGB image (RGB), face depth image (depth), and face infrared image (Infrared) to a resolution of 112×112. Then, proceed to the face image processing stage, specifically: 1) For balanced positive and negative samples, crop the 112×112 resolution images from the face RGB image, face depth image, and face infrared image into 9 overlapping 48×48 image blocks. Input all 9 overlapping image blocks into the AR-MLP network to extract features. For unbalanced positive and negative samples, first randomly flip and rotate, crop to supplement data samples, and perform data augmentation. Then, crop and cut into 9 overlapping 48×48 image blocks. Input all 9 overlapping 48×48 image blocks into the AR-MLP network to extract features.

[0046] Step 2) Stack the input to the AR-MLP network, perform transpose and reshape. For an image patch with a resolution of 48×48, four rounds of operations are set in the AR-MLP network, including three rounds of convolution and MLP mixed downsampling, and the last round of convolution and self-attention mixed block operation (performing ACmix calculation of convolution and self-attention mixed module to reduce the tedious MLP calculation on the lowest feature map, using the convolution operation of the last layer to merge it with the self-attention operation to jointly calculate the final output vector). After each downward sampling of the patch, the length and width of the image patch are halved. When training the AR-MLP model, the adaptive moment estimation optimizer algorithm is used for two rounds of training, with 50 batches per round, batch size set to 128, learning rate lr set to 1e-3, and finally the initial value of weight decay is adjusted to 1e-5, and the decay rate is set to 0.005 to prevent overfitting.

[0047] Step 3) Feed the feature tensors into ACMix, which integrates convolutional kernel attention;

[0048] Step 4) The MLP of each block and the global MLP across blocks are mixed and then fed into the classifier for feature classification to obtain the final classification result of real face or fraudulent face, thus obtaining the face anti-fraud detection model.

[0049] Block convolutions only capture local features and cannot connect global features to capture long-range dependencies. However, adding an MLP-Mixer allows for the mixing of information between blocks during network deepening, continuously acquiring long-range global correlations between blocks. The interaction process between blocks in a specific channel of a cross-slice global MLP is as follows: Figure 4 As shown, for the features of an N×N, N=9 MLPMXer with a resolution of P×P, it can be regarded as an N-dimensional 1x1 convolution.

[0050] Assume the input feature map tensor is x i ∈R 9×C×P×P i = 0, ..., 8, representing a patch x on a specific channel. i '∈R 9×1×P×P Taking i = 0, ..., 8 as an example, extract any pixel x on each facet. i ∈R 1×9×P×P The information of pixels at the same position as i=0,...,8 and all other patches is weighted and used as the output pixel information. The other pixels of the patch are also processed in the same way. In this way, the output feature map of each patch is also a weighted sum of all other patches and has the feature information of other patches. This realizes the process of pixels on the patch "interacting" with other pixels. After passing through the second to fourth layers of cross-patch global MLP, local features and long-range dependency features can be obtained.

[0051] Because convolution and attention have operational similarities, this invention integrates and combines the two, taking into account their respective advantages. The similarities and differences in the operations during the merging process are as follows: Figure 5 As shown.

[0052] Convolution and self-attention operate similarly in the first stage. Both involve performing a 1×1 convolution followed by a linear projection onto the input feature map. Convolution can be viewed as three 1×1 convolutions followed by a linear projection, while self-attention can be seen as a projection matrix W of the query, key, and value. q (l) W k (l) W v (l) Three 1×1 convolutional projections; such as Figure 5 As shown, N×H×W×C is considered as three 1×1 convolutions followed by linear projection, so that... Perform calculations;

[0053] The main difference between the two lies in the second stage (the difference), which includes convolution and self-attention. Convolution involves translation and accumulation operations, specifically through the formula... The final aggregation is H×W×C, with Kc set to 3 for accumulation. Convolution is then used to shift and aggregate the projected feature maps, outputting the feature calculation result F. c ;

[0054] Attention is calculated and aggregated through queries, key-value pairs, and numerical values, specifically through formulas. The calculation ultimately aggregates into H×W×C, with Ka set to 5 and accumulated. Self-attention primarily involves attention weights A(W). q (l) f ij W k (l) f ab The calculation of ) and the aggregation of the weight matrix to output the attention result F. a The final result obtained by the ACmix module is a weighted sum of the two, as shown in equation (1) below.

[0055] F ACmix =F c +F a (1)

[0056] Figure 5 In this context, N represents the quantity, C represents the channel, H represents the height, and w represents the width.

[0057] The specific process of obtaining facial recognition fraud detection results includes:

[0058] Save the model's weight parameters as a pth format file;

[0059] The saved weight parameter file is read using Python and imported into the embedded terminal. Python is then used to construct the network structure of the face anti-fraud detection model and perform convolution operations between network layers. The forward inference process is then executed to obtain the face anti-fraud detection results.

[0060] In a preferred embodiment, the model parameters of the face liveness detection model are exported and deployed to an embedded device. The network structure and convolution operations between the network layers are executed using Python language to perform the forward inference process and obtain the face anti-fraud detection results.

[0061] The AR-MLP network architecture proposed in this invention is implemented on PyTorch 1.8 and runs on an NVIDIA GeForce RTX 3090.

[0062] Table 1 Comparison of experimental results of the method of this invention and conventional methods on the WMCA dataset.

[0063] MA-Net 11.1 2.6 6.8 DeepPixBix 8.2 3.7 6.0 LBP-SVM 8.5 0.6 4.6 ResNet 3.5 1.6 2.6 MLP-Mixer 1.7 2.3 2.0 Conv-MLP 1.62 1.29 1.45 Method of the present invention 2.18 0.95 1.57

[0064] Under the Grandtest protocol, the results of comparing the algorithm of this invention with those of other conventional methods are listed in Table 1. It can be seen that the technical solution of this invention achieves the second lowest error rate of 0.95% in the classification of real faces and the second lowest average error rate of 1.57%, which is relatively competitive with most other methods.

[0065] Table 2 Comparison of experimental results of the method of this invention with other methods on the HQ-WMCA dataset.

[0066]

[0067] In summary, on the HQ-WMCA dataset, the self-attention-based method outperforms the convolution-only method overall, while the method of this invention achieves highly competitive results by balancing the extraction of key local features and global dependencies. It achieves the best results among all methods on invisible makeup attacks: ACER: 17.6%, a 2.2% improvement over the second-lowest ViTFAS (19.8%). On invisible paper mask attacks, it achieves the second-lowest ACER of 0.1%, only 0.1% lower than DCN and DeepPixBix (0%), equivalent to only one misclassified sample out of 1000 real and attack samples. On invisible flexible silicone masks, it also achieves the second-lowest ACER of 1.5%, only 0.1% lower than the best Conv-MLP (1.4%). On invisible glasses attacks, it achieves the second-lowest ACER of 16.7%, only 0.8% lower than the lowest ViTFAS. On invisible human model attacks, it achieves a relatively low ACER of 1.4%. The proposed AR-MLP method achieves a 1.72% improvement in ACER compared to ViTFAS (based on self-attention), and a 0.55% improvement compared to Conv-MLP (a hybrid of convolution and MLP), resulting in a highest ACER of 7.48 ± 6.60. These experimental results demonstrate that AR-MLP can effectively handle and classify various attack methods that the model has not previously encountered.

[0068] Comparison of computational complexity of the method of this invention with other methods

[0069] ViTFAS 16.85G Mac 85.8M 9.20±7.99 MCCNN 10.88G Mac 37.7M 19.00±13.20 DenseNet 7.82G Mac 26.4M 20.91±15.76 MLP-Mixer 3.30G Mac 64.0M 10.11±12.05 ResNet 7.85G Mac 42.5M 24.62±15.33 Conv-MLP 4.02G Mac 17.4M 8.03±8.74 FaceBagNet 3.69G Mac 14.5M 9.20±9.99 CDCN 13 39G Mac 6.9M 22.3±17.7 This method 2.82G Mac 6.48M 7.48±6.60

[0070] The integrated hybrid convolution and self-attention mechanism in this invention's model, which only performs convolution and self-attention in conv5 and matrix transformations across MLPs, has a low computational cost. This invention's method has a significant advantage in computational cost compared to methods like ViTFAS, which are based on pure self-attention mechanisms. The proposed AR-MLP network has a computational cost as low as 2.82 GMac, which is 29.8% lower than Conv-ML and 14.5% lower than methods like MLP-Mixer that simplify networks using MLP concepts. Compared to other networks, the AR-MLP network's minimal computational cost makes model training faster.

[0071] Experiments demonstrate that the proposed AR-MLP has low computational complexity and a strong competitive advantage in terms of computational cost and parameter count. Extensive comparative and ablation experiments on four authoritative multimodal face liveness detection datasets (WMCA, HQ-WMCA, CASIA-SURF, and CASIA-SURF CeFA) effectively demonstrate that the designed AR-MLP is competitive with existing state-of-the-art solutions.

[0072] Finally, it should be noted that the present invention is not limited to the above embodiments, and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A multimodal face anti-fraud detection method based on AR-MLP, characterized in that... The detection method includes: Step 1) Perform face image processing on the input face RGB image, face depth image and face infrared image, stack and couple them into the AR-MLP network, and perform transposition and shaping; Step 2) Feed the feature tensor into ACMix, which integrates convolution and attention; Step 3) The MLP of each block and the global MLP across blocks are mixed and then fed into the classifier for feature classification to obtain the final classification result of real face or fraudulent face, thus obtaining the face anti-fraud detection model. Face image processing includes: taking a 112×112 image as input. The image was processed by overlapping and cutting it into nine 48×48 image blocks, representing three modalities of face images. The image is divided into N=9 blocks, and the resulting 9 images are based on their channel count C. They are stacked into N=9 overlapping blocks: In AR-MLP, after three rounds of convolution and MLP mixing, the above operations are no longer repeated in the last convolutional layer and MLP rounds. Instead, the ACmix calculation of the convolution and self-attention mixing module is performed to reduce the tedious MLP calculations repeated on the lowest feature map. The convolution operation in the last layer is used to combine it with the self-attention operation to jointly calculate the final output vector. AR-MLP includes: a ResNet18-based Conv-MLP network, which combines local convolutional blocks with a global multilayer perceptron (MLP) in convolutional layers conv1 to conv4 to extract local features and long-range global correlation features of face images, and fine-tunes the parameters of the MLP to output a five-dimensional vector; replacing the conv5 convolutional layer of Conv-MLP with the ACmix hybrid module, which performs a hybrid operation of convolution and self-attention mechanism on the five-dimensional vector obtained from the first four layers, replacing the convolutional basicblock with ACmix; and fusing face multimodal data at the signal level within the network.

2. The multimodal face anti-fraud detection method based on AR-MLP according to claim 1, characterized in that: Before face image processing, the input face RGB image, face depth image and face infrared image are cropped to a resolution of 112×112, and then processed in the face image processing. Specifically: 1) For positive and negative sample balance, the cropped 112×112 resolution images in the face RGB image, face depth image and face infrared image are cropped into 9 overlapping image blocks. These 9 overlapping image blocks are all input into the AR-MLP network to extract features. For imbalanced positive and negative samples, the samples are first randomly flipped and rotated, then cropped to supplement the data samples and perform data augmentation. The images are then cropped into nine overlapping 48×48 image patches. These nine overlapping 48×48 image patches are all input into the AR-MLP network to extract features. 2) For an image patch with a resolution of 48×48, four rounds of computation were performed, including three rounds of convolution and MLP-based mixed downsampling, and the final round of convolution and self-attention mixed block computation. After each downsampling of the image patch, the length and width of the patch were halved. The AR-MLP model was trained using the adaptive moment estimation optimizer algorithm for two rounds, with 50 batches per round and a batch size of 128. The learning rate was set to... lr The initial value of the weight decay was set to 1e-3, and the final value was adjusted to 1e-5, with a decay rate of 0.005 set to prevent overfitting.

3. The multimodal face anti-fraud detection method based on AR-MLP according to claim 2, characterized in that: Export the model parameters of the face liveness detection model and deploy it to an embedded device. Use Python to execute the network structure and convolution operations between the network layers, perform the forward inference process, and obtain the face anti-fraud detection results.

4. The multimodal face anti-fraud detection method based on AR-MLP according to claim 3, characterized in that: The specific process of obtaining facial recognition fraud detection results includes: Save the model's weight parameters as a pth format file; The saved weight parameter file is read using Python and imported into the embedded terminal. Python is then used to construct the network structure of the face anti-fraud detection model and perform convolution operations between network layers. The forward inference process is then executed to obtain the face anti-fraud detection results.

5. The multimodal face anti-fraud detection method based on AR-MLP according to claim 1, characterized in that: The AR-MLP network architecture was implemented in PyTorch 1.8 and ran on an NVIDIA GeForce RTX 3090 with fast computing power.

Images