Method and device for training multi-modal face fraud prevention model, and electronic device

Abstract

The present application relates to the technical field of artificial intelligence, and provides a training method and device of a multi-modal face fraud prevention model and electronic equipment. The method comprises the following steps: firstly, obtaining multi-modal images of multiple face samples and inputting the face fraud prevention model, then obtaining initial image features and semantic features of each face sample by using a CLIP network in the face fraud prevention model; and using an attention network and a dual space adaptation network to perform denoising and enhancement operations on the initial image features of each face sample to obtain enhanced image features; then using a classifier to calculate a prediction probability of each face sample being a fake face based on the enhanced image features and the semantic features of each face sample; and finally updating parameters of the attention network and the dual space adaptation network in the face fraud prevention model based on the class labels, the semantic features and the prediction probability of each face sample to obtain the trained face fraud prevention model. Thus, the accuracy and reliability of the model in detecting faces are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal face anti-fraud model training method and device and electronic equipment. BACKGROUND

[0002] The widespread application of face recognition technology has brought many security challenges, especially complex and diverse attack methods. For example, attackers can generate highly realistic fake face images by printing photos, playing videos, making 3D masks, and using deepfake technology and artificial intelligence technology. These attack methods pose a serious threat to traditional image recognition systems.

[0003] To address the above challenges, multi-modal face anti-fraud detection technology has gradually become the focus of academia and industry. This technology can detect real faces and fake faces through multiple modalities of data such as visible light, depth, and infrared data. However, this technology also faces difficulties in practical applications. For example, different collection conditions can cause differences in the feature distribution of different data sets, i.e., domain noise; and different modalities of sensors and imaging principles can also cause differences between different modalities of data, i.e., modal noise. Due to the presence of domain noise and modal noise, face detection becomes more difficult. Moreover, the face anti-fraud model in the prior art usually uses a special network to learn the feature representation of multi-modal data based on training samples. However, multi-modal data in actual application scenarios are diverse and complex, and when the multi-modal data actually processed is significantly different from the training samples, the face anti-fraud model will have difficulty accurately representing the features of the multi-modal data, thereby affecting the reliability of face detection. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a multi-modal face anti-fraud model training method, device and electronic equipment.

[0005] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides a multi-modal face anti-fraud model training method, which comprises:

[0007] Obtaining multi-modal images and class labels of multiple face samples; the class label indicates whether the face sample is a real face or a fake face;

[0008] Using the CLIP network in the face anti-fraud model, based on the multi-modal images of each face sample, obtaining the initial image features of each face sample, and mapping each initial image feature to the same semantic space to obtain the semantic features of each face sample;

[0009] performing a denoising operation on the initial image feature of each face sample by using the attention network in the face anti-fraud model to obtain a denoised image feature of each face sample;

[0010] performing an enhancement operation on the denoised image feature of each face sample based on the semantic feature of each face sample by using the dual-space adaptive network in the face anti-fraud model to obtain an enhanced image feature of each face sample;

[0011] calculating a prediction probability of each face sample being a fake face based on the enhanced image feature and the semantic feature of each face sample by using the classifier in the face anti-fraud model;

[0012] updating the parameters of the attention network and the dual-space adaptive network in the face anti-fraud model based on the class label, the semantic feature and the prediction probability of each face sample to obtain a trained face anti-fraud model.

[0013] In an optional implementation, the CLIP network includes an image encoder, a text encoder and a mapping layer, and the multi-modal image includes a visible light image, an infrared image and a depth image.

[0014] The step of obtaining the initial image feature of each face sample based on the multi-modal image of each face sample by using the CLIP network in the face anti-fraud model and mapping each initial image feature to the same semantic space to obtain the semantic feature of each face sample includes:

[0015] performing an extraction operation on the visible light image, the infrared image and the depth image of each face sample by using the image encoder to obtain the visible light image feature, the infrared image feature and the depth image feature of each face sample and perform a fusion operation to obtain the initial image feature of each face sample;

[0016] performing an extraction operation on the two types of preset class label texts by using the text encoder to obtain the text feature of each type of class label text;

[0017] mapping the initial image feature of each face sample to the semantic space constructed by the two text features by using the mapping layer to obtain the semantic feature of each face sample.

[0018] In an optional implementation, each face sample has a domain label, and the domain label represents the acquisition condition of the multi-modal image of the face sample.

[0019] The step of performing a denoising operation on the initial image feature of each face sample to obtain the denoised image feature of each face sample includes:

[0020] The initial image features of all face samples are divided into multiple batch sample sets; wherein, in each batch sample set, there are initial image features with different domain labels;

[0021] The initial image features in each batch sample set are reorganized to obtain a reorganized sample set of each batch sample set; wherein, the initial image features with the same position in the batch sample set and its reorganized sample set have the same domain label;

[0022] For each batch sample set, the domain noise and the modal noise are estimated based on the batch sample set and its reorganized sample set, and each initial feature image in the batch sample set is denoised to obtain each denoised image feature, thereby obtaining the denoised image features of each face sample.

[0023] In an optional implementation, the step of estimating the domain noise and the modal noise based on the batch sample set and its reorganized sample set, and denoising each initial feature image in the batch sample set to obtain each denoised image feature, comprises:

[0024] The batch sample set and its reorganized sample set are spliced to obtain a spliced matrix; the elements of each row in the spliced matrix include two initial image features with the same position in the batch sample set and its reorganized sample set;

[0025] The spliced matrix is multiplied by the current query weight matrix of the attention network to obtain a query matrix, and the query matrix is split into two matrices with the same number of columns to obtain a first query matrix and a second query matrix;

[0026] The spliced matrix is multiplied by the current key weight matrix of the attention network to obtain a key matrix, and the key matrix is split into two matrices with the same number of columns to obtain a first key matrix and a second key matrix;

[0027] The spliced matrix is multiplied by the current value weight matrix of the attention network to obtain a value matrix;

[0028] The first query matrix is multiplied by the transpose of the first key matrix to obtain a first matrix, and the first matrix is multiplied by a preset scaling factor and then normalized to obtain a sample feature matrix;

[0029] The second query matrix is multiplied by the transpose of the second key matrix to obtain a second matrix, and the second matrix is multiplied by the scaling factor and then normalized to obtain an initial noise matrix; the initial noise matrix represents the domain noise and the modal noise;

[0030] The preset adjustment parameter is multiplied by the initial noise matrix to obtain a target noise matrix, and the sample feature matrix is subtracted from the target noise matrix and multiplied by a value matrix to obtain a denoised feature matrix; elements of each row in the denoised feature matrix represent features of each denoised image.

[0031] In an optional implementation, the dual-space adaptive network comprises a plurality of network layers.

[0032] The step of performing an enhancement operation on the denoised image feature of each face sample based on the semantic feature of the face sample by using the dual-space adaptive network in the face fraud prevention model to obtain an enhanced image feature of each face sample comprises:

[0033] For each face sample, the semantic feature and the denoised image feature of the face sample are input into a first network layer, and an alignment operation is performed by using each network layer in sequence to obtain an aligned image feature output by each network layer;

[0034] For each network layer, the aligned image feature output by a subsequent network layer of the network layer is remapped, and a residual operation is performed on the aligned image feature output by the network layer to obtain a target image feature output by the network layer;

[0035] The target image feature output by the first network layer is taken as the enhanced image feature of the face sample;

[0036] The denoised image feature of each face sample is traversed to obtain the enhanced image feature of each face sample.

[0037] In an optional implementation, the step of updating parameters of the attention network and the dual-space adaptive network in the face fraud prevention model based on the class label, the semantic feature and the prediction probability of each face sample to obtain the trained face fraud prevention model comprises:

[0038] A cross-entropy loss is calculated based on the class label and the semantic feature of each face sample; the cross-entropy loss represents a matching degree of the initial image feature and the semantic feature of all face samples;

[0039] A classification loss is calculated based on the class label and the prediction probability of each face sample; the classification loss represents a consistency degree of the class label and the prediction probability of all face samples;

[0040] Parameters of the attention network and the dual-space adaptive network in the face fraud prevention model are updated based on the cross-entropy loss and the classification loss to obtain the trained face fraud prevention model.

[0041] In a second aspect, the present application provides a face detection method, which comprises:

[0042] obtaining a plurality of modal images of a to-be-processed face;

[0043] obtaining an initial image feature of the to-be-processed face based on the plurality of modal images of the to-be-processed face by using a CLIP network in the trained face anti-fraud model, and mapping the initial image feature to a semantic space to obtain a semantic feature of the to-be-processed face;

[0044] performing a denoising operation on the initial image feature of the to-be-processed face by using an attention network in the trained face anti-fraud model to obtain a denoised image feature of the to-be-processed face;

[0045] performing an enhancement operation on the denoised image feature of the to-be-processed face based on the semantic feature of the to-be-processed face by using a dual-space adaptation network in the trained face anti-fraud model to obtain an enhanced image feature of the to-be-processed face;

[0046] calculating a prediction probability that the to-be-processed face is a fake face based on the enhanced image feature and the semantic feature of the to-be-processed face by using a classifier in the trained face anti-fraud model, and determining whether the to-be-processed face is a fake face according to the prediction probability;

[0047] The trained face anti-fraud model is obtained according to the training method of the multi-modal face anti-fraud model in any one of the preceding embodiments.

[0048] In a third aspect, the present application provides a training device of a multi-modal face anti-fraud model, the device comprising:

[0049] a sample acquisition module for acquiring multi-modal images and category labels of a plurality of face samples; the category labels represent whether the face samples are real faces or fake faces;

[0050] a sample processing module for obtaining an initial image feature of each face sample based on the multi-modal image of each face sample by using a CLIP network in a face anti-fraud model, and mapping each initial image feature to the same semantic space to obtain a semantic feature of each face sample;

[0051] performing a denoising operation on the initial image feature of each face sample by using an attention network in the face anti-fraud model to obtain a denoised image feature of each face sample;

[0052] performing an enhancement operation on the denoised image feature of each face sample based on the semantic feature of each face sample by using a dual-space adaptation network in the face anti-fraud model to obtain an enhanced image feature of each face sample;

[0053] a sample classification module configured to calculate, by using a classifier in the face fraud prevention model, a predicted probability of each face sample being a fake face based on the enhanced image feature and the semantic feature of each face sample;

[0054] a model updating module configured to update parameters of the attention network and the dual spatial adaptation network in the face fraud prevention model based on the class label, the semantic feature and the predicted probability of each face sample, to obtain the trained face fraud prevention model.

[0055] In a fourth aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the training method of the multi-modal face fraud prevention model according to any one of the preceding embodiments and / or the face detection method according to the preceding embodiments when executing the computer program.

[0056] In a fifth aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the training method of the multi-modal face fraud prevention model according to any one of the preceding embodiments and / or the face detection method according to the preceding embodiments.

[0057] The training method, device and electronic device of the multi-modal face fraud prevention model provided by the embodiments of the present application, the method comprising: firstly acquiring multi-modal images of multiple face samples and inputting the multi-modal images to a face fraud prevention model, then using a CLIP network in the face fraud prevention model to obtain initial image features and semantic features of each face sample based on the multi-modal images of each face; and using an attention network and a dual spatial adaptation network to perform denoising operation and enhancement operation on the initial image features of each face sample, to obtain enhanced image features of each face sample; then using a classifier to calculate a predicted probability of each face sample being a fake face based on the enhanced image features and the semantic features of each face sample; and finally updating parameters of the attention network and the dual spatial adaptation network in the face fraud prevention model based on the class label, the semantic features and the predicted probability of each face sample, to obtain the trained face fraud prevention model. The soft alignment of the image features is realized by the CLIP network, the domain noise and the modal noise are filtered out by the attention network, and the representation space of the features is optimized by the dual spatial adaptation network, so that the accuracy and reliability of the model in detecting faces are improved.

[0058] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are referred to. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0060] Figure 1 An example diagram of the presence domain noise and the modal noise of the multi-modal data is shown;

[0061] Figure 2 A schematic diagram of the representation space of the existing face fraud prevention model is shown;

[0062] Figure 3 A flowchart of the training method of the multi-modal face fraud prevention model provided by the embodiments of the present application is shown;

[0063] Figure 4 An example diagram of the processing flow of the CLIP network provided by the embodiments of the present application is shown;

[0064] Figure 5 An example diagram of the processing flow of the attention network provided by the embodiments of the present application is shown;

[0065] Figure 6 An example diagram of the processing flow of the dual-space adaptive network provided by the embodiments of the present application is shown;

[0066] Figure 7 A flowchart of the face detection method provided by the embodiments of the present application is shown;

[0067] Figure 8 An example diagram of the face detection method provided by the embodiments of the present application is shown;

[0068] Figure 9 A schematic diagram of the representation space of the face fraud prevention model after training provided by the embodiments of the present application is shown;

[0069] Figure 10 A functional module diagram of the training device of the multi-modal face fraud prevention model provided by the embodiments of the present application is shown;

[0070] Figure 11 A functional module diagram of the face detection device provided by the embodiments of the present application is shown;

[0071] Figure 12 A block schematic diagram of the electronic device provided by the embodiments of the present application is shown.

[0072] Icon: 100-electronic device; 110-processor; 120-memory; 130-communication module; 300-training device of multi-modal face fraud detection model; 310-sample acquisition module; 330-sample processing module; 350-sample classification module; 370-model updating module; 400-face detection device; 410-acquisition module; 430-detection module. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0074] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0075] It should be noted that the relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0076] The widespread application of face recognition technology has brought many security challenges, especially the complex and diverse attack means. For example, attackers can generate highly realistic fake face images by printing photos, playing videos, making 3D masks, and using deep fake technology and artificial intelligence technology. These attack means pose a serious threat to traditional image recognition systems. In order to cope with the above challenges, multi-modal face fraud detection technology has gradually become the focus of attention in the academic and industrial communities. This technology can detect real faces and fake faces through multiple modal data such as visible light, depth and infrared data. However, this technology also encounters difficulties in practical application.

[0077] For example, different acquisition conditions can lead to differences in the feature distribution of different datasets, i.e., the presence of domain noise; furthermore, different modal sensors and imaging principles can cause differences in data between different modalities, i.e., the presence of modal noise. Figure 1 As shown, for the visible light modality, the differences in image data acquired under different acquisition conditions are relatively small, meaning the domain difference is small. However, for the infrared and depth modalities, the differences in image data acquired under different acquisition conditions are larger, meaning the domain difference is larger. Furthermore, due to the different sensors and imaging principles of the visible light, infrared, and depth modalities, differences exist in the image data between different modalities, i.e., modal bias exists. The simultaneous presence of domain noise and modal noise increases the difficulty of face detection.

[0078] Moreover, existing face fraud prevention models typically employ specialized networks to learn feature representations of multimodal data based on training samples. For example... Figure 2 As shown, it is a schematic diagram of the representation space of the existing face fraud prevention model. Because the boundary of this representation space is too smooth, when the difference between the actual multimodal data and the training data is large, the model will be unable to accurately represent the features of the multimodal data, thus affecting the reliability of face detection.

[0079] Therefore, embodiments of the present invention provide a training method for a multimodal face fraud prevention model to solve the above-mentioned problems. Please refer to... Figure 3 This is a flowchart illustrating the training method of the multimodal face anti-fraud model provided in this embodiment of the invention.

[0080] Step S200: Obtain multimodal images and category labels of multiple face samples; the category labels indicate whether the face sample is a real face or a fake face.

[0081] In this embodiment, multiple modal images of multiple face samples can be acquired, and each face sample can be labeled as a real face or a fake face, so as to obtain multimodal images and category labels of multiple face samples, and these multiple face samples include real faces and fake faces.

[0082] Step S202: Using the CLIP network in the face anti-fraud model, based on the multimodal image of each face sample, the initial image features of each face sample are obtained, and each initial image feature is mapped to the same semantic space to obtain the semantic features of each face sample.

[0083] Step S204: Using the attention network in the face anti-fraud model, the initial image features of each face sample are denoised to obtain the denoised image features of each face sample.

[0084] Step S206, using the dual space adaptation network in the face anti-fraud model, the denoised image features of each face sample are enhanced based on the semantic features of each face sample to obtain enhanced image features of each face sample.

[0085] Step S208, using the classifier in the face anti-fraud model, the prediction probability of each face sample being a fake face is calculated based on the enhanced image features and semantic features of each face sample.

[0086] In this embodiment, the face anti-fraud model sequentially includes a CLIP (Contrastive Language-Image Pre-Training) network, an attention network, a dual space adaptation network, and a classifier. Then the multi-modal images of each face sample can be input into the face anti-fraud model, and the CLIP network is used to extract the features of each multi-modal image to obtain the initial image features of each face sample, and each initial image feature is mapped to the same semantic space to realize soft alignment of image features of multiple modalities to obtain the semantic features of each face sample.

[0087] Then the initial image features of each face sample are denoised using the attention network to filter out domain noise and modal noise to obtain the denoised image features of each face sample. Then the dual space adaptation network is used to enhance the denoised image features of each face sample based on the semantic features of each face sample to optimize the representation space of the features to obtain the enhanced image features of each face sample. Finally, the classifier is used to calculate the probability of each face sample being a fake face based on the enhanced image features and semantic features of each face sample, i.e., to obtain the prediction probability of each face sample.

[0088] Step S210, based on the class labels, semantic features, and prediction probabilities of each face sample, the parameters of the attention network and the dual space adaptation network in the face anti-fraud model are updated to obtain the trained face anti-fraud model.

[0089] In this embodiment, based on the obtained class labels, semantic features, and prediction probabilities of each face sample, the face anti-fraud model is trained to update the parameters of the attention network and the dual space adaptation network in the face anti-fraud model, i.e., to obtain the trained face anti-fraud model. It should be noted that during the training of the face anti-fraud model, since the CLIP network is a pre-trained network, the parameters of the CLIP network are frozen, i.e., the parameters of the CLIP network are not updated; and for the classifier, if a pre-trained classifier is used, the parameters of the classifier are frozen, and if an untrained classifier is used, the parameters of the classifier are updated.

[0090] It can be seen that based on the above steps, first, the multi-modal images of multiple face samples are acquired and input into the face anti-fraud model, then the CLIP network in the face anti-fraud model is used to obtain the initial image feature and the semantic feature of each face sample based on the multi-modal images of each face; the attention network and the dual spatial adaptation network are used to perform denoising operation and enhancement operation on the initial image feature of each face sample to obtain the enhanced image feature of each face sample; then the classifier is used to calculate the prediction probability of each face sample being a fake face based on the enhanced image feature and the semantic feature of each face sample; finally, the parameters of the attention network and the dual spatial adaptation network in the face anti-fraud model are updated based on the class label, the semantic feature and the prediction probability of each face sample to obtain the trained face anti-fraud model. The soft alignment of the image features is realized by the CLIP network, the domain noise and the modal noise are filtered by the attention network, and the representation space of the features is optimized by the dual spatial adaptation network, thereby improving the accuracy and reliability of the model in detecting faces.

[0091] Optionally, for step S202, the embodiment of the present application provides a possible implementation manner.

[0092] In step S202-1, the image encoder is used to perform extraction operation on the visible light image, the infrared image and the depth image of each face sample to obtain the visible light image feature, the infrared image feature and the depth image feature of each face sample, and fusion operation is performed to obtain the initial image feature of each face sample.

[0093] In step S202-2, the text encoder is used to perform extraction operation on the two kinds of preset class label texts to obtain the text feature of each kind of class label text.

[0094] In step S202-3, the mapping layer is used to map the initial image feature of each face sample to the semantic space constructed by the two kinds of text features to obtain the semantic feature of each face sample.

[0095] In the embodiment, the CLIP network includes an image encoder, a text encoder and a mapping layer; the image encoder is used to extract the features of the images; the text encoder is used to extract the features of the texts; and the mapping layer is used to realize the soft alignment of the image features of multiple modalities. In addition, the multi-modal images include visible light images, infrared images and depth images; the visible light images can be RGB images, which can provide color and texture information; the infrared images are images formed based on the principle of thermal imaging according to the infrared radiation emitted by objects; and the depth images are used to reflect the three-dimensional structure information of objects such as shape and contour.

[0096] It can be understood that the CLIP network processes each face sample in a similar manner. For brevity, a face sample is taken as an example for description below. Furthermore, the embodiment of the present application also provides a processing flow diagram of the CLIP network, as shown in Figure 4 .

[0097] For example, the two preset category label texts include a first category label text c s and a second category label text c b , and the content of the first category label text c s is "this is a fake face", and the content of the second category label text c b is "this is a real face". First, the visible light image I RGB , the infrared image I ir , and the depth image I depth of the face sample, and the first category label text c s and the second category label text c b are input into the CLIP network.

[0098] Then, the image encoder is used to perform extraction operation on the visible light image I RGB , the infrared image I ir , and the depth image I depth of the face sample to obtain the visible light image feature v RGB , the infrared image feature v ir , and the depth image feature v depth and perform fusion operation to obtain the initial image feature v of the face sample. At the same time, the text encoder is used to perform extraction operation on the first category label text c s and the second category label text c b to obtain the text feature of the first category label text c s , i.e., the first text feature t s , and the text feature of the second category label text c b , i.e., the second text feature t b .

[0099] Next, the mapping layer is used to map the initial image feature v of the face sample to the semantic space constructed by the first text feature t s and the second text feature t b to obtain the semantic feature t of the face sample. For example, the cosine distance of the initial image feature v of the face sample with the first text feature t s and the second text feature t b can be calculated, and the text feature with the smallest cosine distance is taken as the semantic feature t of the face sample. Furthermore, the process of calculating the smallest cosine distance can be expressed by a formula as follows: wherein r represents the minimum cosine distance corresponding to the face sample, T represents the text feature set, t j represents the jth text feature in the text feature set, v represents the initial image feature of the face sample, and || || represents the norm. In a similar manner, the initial image feature and the semantic feature of each face sample can be obtained by processing each face sample.

[0100] It can be understood that the CLIP network of the embodiment of the present application aligns image features of multiple modalities to a generalized representation space, so as to enhance the distinguishing ability of the model for complex representations in a soft alignment manner, thereby improving the adaptability of the model under different data distributions, and enabling the model to cope with various multi-modal data in actual scenarios.

[0101] Optionally, for step S204, the embodiment of the present application provides a possible implementation manner.

[0102] Step S204-1, the initial image features of all face samples are divided into multiple batch sample sets; wherein the initial image features with different domain labels exist in each batch sample set.

[0103] Step S204-2, the initial image features in each batch sample set are reorganized to obtain the reorganized sample set of each batch sample set; wherein the domain labels of the initial image features with the same position in the batch sample set and its reorganized sample set are the same.

[0104] Step S204-3, for each batch sample set, the domain noise and the modal noise are estimated based on the batch sample set and its reorganized sample set, and the denoising operation is performed on each initial image feature in the batch sample set to obtain each denoised image feature, thereby obtaining the denoised image feature of each face sample.

[0105] In the present embodiment, each face sample has a domain label, which represents the acquisition condition of the multi-modal image of the face sample. Then if the domain labels of the face samples are the same, it indicates that the acquisition conditions of the multi-modal images of the face samples are the same; if the domain labels of the face samples are different, it indicates that the acquisition conditions of the multi-modal images of the face samples are different. Wherein, the acquisition condition can include acquisition information such as illumination, resolution, etc.

[0106] Based on the obtained initial image features of all face samples, they can be divided into multiple batch sample sets, and the initial image features with different domain labels exist in each batch sample set, that is, each batch sample set is a multi-domain sample set. Then, the initial image features in each batch sample set are reorganized to obtain the reorganized sample set of each batch sample set.

[0107] Understandably, the reorganization process for each batch of sample sets is similar. For simplicity, we will use a single batch of sample sets as an example below. For instance, batch sample set X... h Initial image features including three face samples, such as v a v b v c , that is, X h ={v a ,v b ,v c}, and the initial image features v a and v c The domain labels are the same, and the initial image features v b With v a and v c The domain tags are all different.

[0108] So for the batch sample set X h The first initial image feature v a It can be done in batch sample set X h Obtain the initial image features, i.e., v, that are the same as their domain labels. c and the initial image features v c As the first element of the reconstructed sample set. For the second initial image feature v b Because of its batch sample set X h If no initial image features with the same domain label exist, then the initial image feature v will be... b As the second element of the reconstructed sample set. For the third initial image feature v c It can be directly used as the third element of the recombined sample set.

[0109] This gives us the batch sample set X. h The recombined sample set, namely X′ h And X′ h ={v c ,v b ,v c Furthermore, the batch sample set X h and the recombined sample set X′ h Initial image features with the same location share the same domain label. Processing each batch of samples in a similar manner yields a reconstructed sample set for each batch. Each reconstructed sample set is a multi-domain sample set.

[0110] It can be understood that the reorganized sample set is constructed by the batch sample set, which helps to capture the noise according to the reorganized sample set. Then for each batch sample set, the domain noise and the modal noise can be estimated based on the batch sample set and the reorganized sample set thereof, and each initial feature image in the batch sample set is denoised to obtain each denoised image feature, so as to obtain the denoised image feature of each face sample.

[0111] Optionally, for the process of estimating the domain noise and the modal noise based on the batch sample set and the reorganized sample set thereof in step S204-3, and denoising each initial feature image in the batch sample set to obtain each denoised image feature, the embodiment of the present application provides a possible implementation manner.

[0112] In step S204-3-1, the batch sample set and the reorganized sample set thereof are spliced to obtain a spliced matrix, and the elements of each row in the spliced matrix include two initial image features with the same position in the batch sample set and the reorganized sample set thereof.

[0113] In step S204-3-2, the spliced matrix is multiplied by the current query weight matrix of the attention network to obtain a query matrix, and the query matrix is split into two matrices with the same number of columns to obtain a first query matrix and a second query matrix.

[0114] In step S204-3-3, the spliced matrix is multiplied by the current key weight matrix of the attention network to obtain a key matrix, and the key matrix is split into two matrices with the same number of columns to obtain a first key matrix and a second key matrix.

[0115] In step S204-3-4, the spliced matrix is multiplied by the current value weight matrix of the attention network to obtain a value matrix.

[0116] In step S204-3-5, the first query matrix is multiplied by the transpose of the first key matrix to obtain a first matrix, and the first matrix is multiplied by a preset scaling factor and then normalized to obtain a sample feature matrix.

[0117] In step S204-3-6, the second query matrix is multiplied by the transpose of the second key matrix to obtain a second matrix, and the second matrix is multiplied by a scaling factor and then normalized to obtain an initial noise matrix; the initial noise matrix represents the domain noise and the modal noise.

[0118] In step S204-3-7, a preset adjustment parameter is multiplied by the initial noise matrix to obtain a target noise matrix, and the sample feature matrix is subtracted from the target noise matrix and then multiplied by the value matrix to obtain a denoised feature matrix; the elements of each row in the denoised feature matrix represent each denoised image feature.

[0119] For the convenience of understanding, the following continues the batch sample set X hand the recomposed sample set X' h An illustration is made. Also, the embodiment of the present application further provides an example diagram of the processing flow of the attention network, as shown in Figure 5 .

[0120] Firstly, the batch sample set X h and the recomposed sample set X' h are spliced to obtain a spliced matrix X, and the elements of each row in the spliced matrix X include two initial image features at the same position in the batch sample set X h and the recomposed sample set X' h . For example, the elements of the first row in the spliced matrix X include the initial image features v a and v c , the elements of the second row include the initial image features v b and v b , and the elements of the third row include the initial image features v c and v c .

[0121] Then, the spliced matrix X is multiplied by the current query weight matrix W q of the attention network to obtain a query matrix, and the query matrix is split into two matrices with the same number of columns, i.e., a first query matrix and a second query matrix. This process can be expressed by the formula: (Q, Q ′ ) = split(XW q ); wherein Q represents the first query matrix, Q ′ represents the second query matrix, and split() represents a split function. Also, the spliced matrix X is multiplied by the current key weight matrix W k of the attention network to obtain a key matrix, and the key matrix is split into two matrices with the same number of columns, i.e., a first key matrix and a second key matrix. This process can be expressed by the formula: (K, K ′ ) = split(XW k ); wherein K represents the first key matrix, K ′ represents the second key matrix, and split() represents a split function. Also, the spliced matrix X is multiplied by the current value weight matrix W v of the attention network to obtain a value matrix. This process can be expressed by the formula: V = XW v ; wherein V represents the value matrix.

[0122] Next, the first query matrix Q is multiplied by the transpose K T of the first key matrix to obtain a first matrix, and the first matrix is multiplied by a preset scaling factor s and then normalized to obtain a sample feature matrix. Also, the second query matrix Q ′ is multiplied by the transpose K ′TThe second matrix is obtained by multiplication, and the initial noise matrix is obtained by multiplying the second matrix by the scaling factor s and then normalizing. The initial noise matrix represents the domain noise and the modal noise. The preset adjustment parameter λ is multiplied by the initial noise matrix to obtain the target noise matrix. The denoised feature matrix is obtained by subtracting the target noise matrix from the sample feature matrix and then multiplying the result by the value matrix V. Alternatively, the denoised feature matrix can also be obtained by standardizing the result of the sample feature matrix minus the target noise matrix and then multiplying the result by the value matrix V. Each row of elements in the denoised feature matrix represents the denoised image feature corresponding to each initial image feature of the batch sample set X s denoise

[0123] It can be understood that the scaling factor s can be set based on the dimension of the initial image feature, that is, where n represents the dimension of the initial image feature. Then the process of calculating the denoised feature matrix can be represented by the formula:

[0124]

[0125] where X denoise denotes the denoised feature matrix, softmax() denotes a normalization function, Q denotes the first query matrix, Q ′ denotes the second query matrix, K denotes the first key matrix, K ′ denotes the second key matrix, V denotes the value matrix, n denotes the dimension of the initial image feature, and λ denotes the adjustment parameter.

[0126] It can be understood that the attention network of the embodiment of the present application captures noise by reorganizing the sample set, and suppresses the influence of noise on the feature by difference operation, so that the model can focus on effective information, improves the generalization ability of the model, and provides guarantee for the stable operation of the model in complex environment.

[0127] Optionally, for step S206, the embodiment of the present application provides a possible implementation manner.

[0128] Step S206-1, for each face sample, input the semantic feature and the denoised image feature of the face sample into the first network layer, and perform alignment operation on each network layer in turn to obtain the alignment image feature output by each network layer.

[0129] Step S206-2, for each network layer, the alignment image feature output by the next network layer of the network layer is remapped, and then residual operation is performed on the alignment image feature output by the network layer to obtain the target image feature output by the network layer.

[0130] Step S206-3, the target image feature output by the first network layer is taken as the enhanced image feature of the face sample.

[0131] Step S206-4: Traverse the denoised image features of each face sample to obtain the enhanced image features of each face sample.

[0132] It is understandable that the dual-spatial adaptation network processes each face sample in a similar way. For the sake of brevity, a single face sample will be used as an example below. Furthermore, this embodiment of the invention also provides an example diagram of the processing flow of the dual-spatial adaptation network, as shown below. Figure 6 As shown.

[0133] For example, suppose the dual-space adaptation network has D network layers. First, the semantic features t of the face sample and the denoised image features v′ are input into the first network layer, and the first network layer performs an alignment operation based on the semantic features t and the denoised image features v′ to obtain the aligned semantic features t1 and aligned image features v′1 output by the first network layer.

[0134] Then, for each network layer other than the first network layer, an alignment operation is performed based on the alignment semantic features and alignment image features output by the previous network layer of that other network layer, to obtain the alignment semantic features and alignment image features output by that other network layer, thus obtaining the alignment image features output by each network layer, i.e., v′1 to v′. D .

[0135] Next, for each network layer, the aligned image features output by the next network layer are remapped, and then a residual operation is performed between this remapped image features and the aligned image features output by the current network layer to obtain the target image features output by that network layer. This process can be expressed by the following formula:

[0136]

[0137] Where, v″ d v′ represents the target image features output by the d-th network layer. d represents the aligned image features output by the d-th network layer; Remap() represents the remapping function; D represents the total number of network layers in the dual-space adaptation network.

[0138] Finally, the target image feature v″1 output from the first network layer is used as the enhanced image feature of the face sample. This process is repeated for each face sample to obtain its enhanced image feature.

[0139] It can be understood that the dual space adaptation network of the embodiment of the present application aligns the image features and the semantic features through each network layer, and feeds back the deep features to the shallow space through the residual connection to improve the expression ability of the model to the features to optimize the representation space, thereby improving the accuracy and reliability of the model in detecting the face.

[0140] Optionally, for step S210, the embodiment of the present application provides a possible implementation manner.

[0141] Step S210-1, based on the class label and the semantic feature of each face sample, a cross-entropy loss is calculated; the cross-entropy loss represents the matching degree of the initial image features and the semantic features of all face samples.

[0142] Step S210-2, based on the class label and the predicted probability of each face sample, a classification loss is calculated; the classification loss represents the consistency degree of the class label and the predicted probability of all face samples.

[0143] Step S210-3, based on the cross-entropy loss and the classification loss, the parameters of the attention network and the dual space adaptation network in the face fraud prevention model are updated to obtain the trained face fraud prevention model.

[0144] In the embodiment, first, a cross-entropy loss can be calculated based on the class label and the semantic feature of each face sample according to a preset cross-entropy loss function, and the cross-entropy loss represents the matching degree of the initial image features and the semantic features of all face samples. And the cross-entropy loss function is as follows:

[0145]

[0146] Wherein, L align represents the cross-entropy loss function, y i represents the class label of the i-th face sample, t i represents the cosine distance between the initial image features and the semantic features of the i-th face sample, and N represents the total number of face samples.

[0147] Then, a classification loss can be calculated based on the class label and the predicted probability of each face sample according to a preset classification loss function, and the classification loss represents the consistency degree of the class label and the predicted probability of all face samples. And the classification loss function is as follows:

[0148]

[0149] Wherein, L cls represents the classification loss function, y i represents the class label of the i-th face sample, p i represents the predicted probability of the i-th face sample, and N represents the total number of face samples.

[0150] Finally, based on the cross-entropy loss L align With classification loss L cls Calculate the total loss L, i.e., L = L align +L cls The training of the face fraud prevention model is guided by the total loss, that is, the parameters of the attention network and the dual-space adaptation network in the face fraud prevention model are updated by the backpropagation algorithm, thus obtaining the trained face fraud prevention model.

[0151] Understandably, the class label for a face sample can use 0 and 1 to represent whether the face sample is a real face or a fake face. Furthermore, the smaller the cross-entropy loss, the better the initial image features and semantic features of all face samples match. The smaller the classification loss, the higher the consistency between the class labels and predicted probabilities of all face samples.

[0152] Optionally, based on the trained face fraud prevention model described above, this embodiment of the invention also provides a face detection method, please refer to [link to relevant documentation]. Figure 7 This is a flowchart illustrating the face detection method provided in this embodiment of the invention.

[0153] Step S212: Obtain the multimodal image of the face to be processed.

[0154] Step S214: Using the CLIP network in the trained face anti-fraud model, based on the multimodal image of the face to be processed, the initial image features of the face to be processed are obtained and mapped to the semantic space to obtain the semantic features of the face to be processed.

[0155] Step S216: Using the attention network in the trained face anti-fraud model, perform denoising operation on the initial image features of the face to be processed to obtain the denoised image features of the face to be processed.

[0156] Step S218: Using the dual-space adaptation network in the trained face anti-fraud model, the denoised image features of the face to be processed are enhanced based on the semantic features of the face to be processed, so as to obtain the enhanced image features of the face to be processed.

[0157] Step S220: Using the classifier in the trained face anti-fraud model, based on the enhanced image features and semantic features of the face to be processed, calculate the predicted probability that the face to be processed is a fake face, and determine whether the face to be processed is a fake face based on the predicted probability.

[0158] For ease of understanding, an example diagram is provided in this embodiment of the invention, such as... Figure 8 As shown. The following will combine... Figure 8 Steps S212 to S220 above will be explained.

[0159] First, multimodal images of the face to be processed, such as visible light image I1, infrared image I2, and depth image I3, are acquired and input into the trained face fraud prevention model. Then, using the CLIP network in the trained face fraud prevention model, the initial image features v of the face to be processed are obtained based on the visible light image I1, infrared image I2, and depth image I3. target and initial image features v target Mapping to the semantic space yields the semantic features t of the face to be processed. target .

[0160] Next, using the attention network in the trained face fraud prevention model, the initial image features v of the face to be processed are... target Perform denoising to obtain the denoised image features v′ of the face to be processed. target Understandably, the initial image features v for processing a human face... target During denoising, there is no need to perform a reconstruction operation; instead, the initial image features v are directly applied. target The concatenation matrix is ​​obtained by concatenating it with itself. Then, using the dual-space adaptation network in the trained face fraud prevention model, based on the semantic features t of the face to be processed... target Denoising image features v′ target Perform enhancement operations to obtain the enhanced image features v″ of the face to be processed. target .

[0161] Finally, using the classifier in the trained face fraud prevention model, the enhanced image features v″ of the face to be processed are analyzed. target and semantic features t target The algorithm calculates the predicted probability that the face to be processed is a fake face, and determines whether the face to be processed is a fake face based on the predicted probability. For example, if the predicted probability reaches a preset threshold such as 0.5, the face to be processed is determined to be a fake face; if the predicted probability does not reach the preset threshold such as 0.5, the face to be processed is determined to be a real face.

[0162] To facilitate understanding of the technical effects of the present invention, embodiments of the present invention also provide a schematic diagram of the representation space of the trained face anti-fraud model, such as... Figure 9 As shown, this representation space has more complex boundaries and can be adapted to training data and multimodal data in actual processing. Therefore, the trained face anti-fraud model provided in this embodiment of the invention can accurately represent the features of multimodal data, thereby improving the accuracy and reliability of face detection.

[0163] To implement the corresponding steps of the training method for the multimodal face fraud prevention model described above, an implementation of a training device for the multimodal face fraud prevention model is given below. Please refer to [link / reference].Figure 10 FIG. 1 is a functional module diagram of a multi-modal face fraud prevention model training device according to an embodiment of the present application. It should be noted that the multi-modal face fraud prevention model training device 300 provided in this embodiment has the same basic principles and technical effects as the above-mentioned embodiments. For brevity, the description of this embodiment is not mentioned, and the corresponding content can be referred to in the above-mentioned embodiments. The multi-modal face fraud prevention model training device 300 comprises:

[0164] The sample acquisition module 310 is configured to acquire multi-modal images and class labels of multiple face samples. The class label indicates whether the face sample is a real face or a fake face.

[0165] The sample processing module 330 is configured to obtain an initial image feature of each face sample based on the multi-modal image of each face sample by using a CLIP network in the face fraud prevention model, and map each initial image feature to a same semantic space to obtain a semantic feature of each face sample. The sample processing module 330 is further configured to perform a denoising operation on the initial image feature of each face sample by using an attention network in the face fraud prevention model to obtain a denoised image feature of each face sample. The sample processing module 330 is further configured to perform an enhancement operation on the denoised image feature of each face sample based on the semantic feature of each face sample by using a dual-space adaptive network in the face fraud prevention model to obtain an enhanced image feature of each face sample.

[0166] The sample classification module 350 is configured to calculate a prediction probability of each face sample being a fake face based on the enhanced image feature and the semantic feature of each face sample by using a classifier in the face fraud prevention model.

[0167] The model updating module 370 is configured to update parameters of the attention network and the dual-space adaptive network in the face fraud prevention model based on the class label, the semantic feature and the prediction probability of each face sample to obtain the face fraud prevention model after training.

[0168] Optionally, the sample processing module 330 is further configured to perform an extraction operation on the visible light image, the infrared image and the depth image of each face sample by using an image encoder to obtain a visible light image feature, an infrared image feature and a depth image feature of each face sample, and perform a fusion operation to obtain the initial image feature of each face sample. The sample processing module 330 is further configured to perform an extraction operation on the two types of preset class label texts by using a text encoder to obtain a text feature of each class label text. The sample processing module 330 is further configured to map the initial image feature of each face sample to a semantic space constructed by the two text features by using a mapping layer to obtain the semantic feature of each face sample.

[0169] Optionally, the sample processing module 330 is further configured to: divide the initial image features of all face samples into a plurality of batch sample sets; wherein the initial image features in each batch sample set have different domain labels; reorganize the initial image features in each batch sample set to obtain a reorganized sample set of each batch sample set; wherein the initial image features in the batch sample set and its reorganized sample set at the same position have the same domain label; estimate the domain noise and the modal noise based on each batch sample set and its reorganized sample set, and perform a denoising operation on each initial image feature in the batch sample set to obtain a denoised image feature, thereby obtaining the denoised image features of each face sample.

[0170] Optionally, the sample processing module 330 is further configured to: splice the batch sample set and its reorganized sample set to obtain a spliced matrix; the elements in each row of the spliced matrix include two initial image features in the batch sample set and its reorganized sample set at the same position; multiply the spliced matrix and the current query weight matrix of the attention network to obtain a query matrix, and split the query matrix into two matrices with the same number of columns to obtain a first query matrix and a second query matrix; multiply the spliced matrix and the current key weight matrix of the attention network to obtain a key matrix, and split the key matrix into two matrices with the same number of columns to obtain a first key matrix and a second key matrix; multiply the spliced matrix and the current value weight matrix of the attention network to obtain a value matrix; multiply the first query matrix and the transpose of the first key matrix to obtain a first matrix, multiply the first matrix by a preset scaling factor, and then normalize to obtain a sample feature matrix; multiply the second query matrix and the transpose of the second key matrix to obtain a second matrix, multiply the second matrix by the scaling factor, and then normalize to obtain an initial noise matrix; the initial noise matrix represents the domain noise and the modal noise; multiply a preset adjustment parameter and the initial noise matrix to obtain a target noise matrix, and multiply the sample feature matrix minus the target noise matrix by the value matrix to obtain a denoised feature matrix; the elements in each row of the denoised feature matrix represent each denoised image feature.

[0171] Optionally, the sample processing module 330 is further configured to: for each face sample, input the semantic feature and the denoised image feature of the face sample into a first network layer, and sequentially perform alignment operations on each network layer to obtain the aligned image features output by each network layer; for each network layer, perform residual operation on the aligned image features output by the next network layer of the network layer after remapping the aligned image features, to obtain the target image features output by the network layer; take the target image features output by the first network layer as the enhanced image features of the face sample; traverse the denoised image features of each face sample to obtain the enhanced image features of each face sample.

[0172] Optionally, the model updating module 370 is further configured to: calculate a cross-entropy loss based on the class label and the semantic feature of each face sample; the cross-entropy loss represents the matching degree of the initial image feature and the semantic feature of all face samples; calculate a classification loss based on the class label and the predicted probability of each face sample; the classification loss represents the consistency degree of the class label and the predicted probability of all face samples; and update the parameters of the attention network and the dual-space adaptive network in the face fraud prevention model based on the cross-entropy loss and the classification loss to obtain the trained face fraud prevention model.

[0173] To perform the corresponding steps of the face detection method described above, an implementation of a face detection device is given below. Please refer to Figure 11 , which is a functional module diagram of the face detection device provided by the embodiment of the present application. It should be noted that the face detection device 400 provided by the present embodiment has the same basic principles and technical effects as the above-mentioned embodiments, and for brief description, the contents not mentioned in the present embodiment can be referred to the corresponding contents in the above-mentioned embodiments. The face detection device 400 comprises:

[0174] The acquisition module 410 is configured to acquire a multi-modal image of a face to be processed.

[0175] The detection module 430 is configured to: obtain an initial image feature of the face to be processed and map it to a semantic space based on the multi-modal image of the face to be processed by using the CLIP network in the trained face fraud prevention model to obtain a semantic feature of the face to be processed; perform a denoising operation on the initial image feature of the face to be processed by using the attention network in the trained face fraud prevention model to obtain a denoised image feature of the face to be processed; perform an enhancement operation on the denoised image feature of the face to be processed based on the semantic feature of the face to be processed by using the dual-space adaptive network in the trained face fraud prevention model to obtain an enhanced image feature of the face to be processed; and calculate a predicted probability that the face to be processed is a fake face based on the enhanced image feature and the semantic feature of the face to be processed by using the classifier in the trained face fraud prevention model, and determine whether the face to be processed is a fake face according to the predicted probability.

[0176] Please refer to Figure 12 , which is a block schematic diagram of the electronic device provided by the embodiment of the present application. The electronic device 100 comprises a processor 110, a memory 120 and a communication module 130, and each element is electrically connected to each other directly or indirectly to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0177] The processor 110 is used to read / write data or programs stored in the memory 120 and perform corresponding functions. It can be a general-purpose processor, including CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be a DSP digital signal processor, ASIC application-specific integrated circuit, FPGA off-the-shelf programmable gate array or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0178] The memory 120 is used to store programs or data, and it can be RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0179] The communication module 130 is used for signaling or data communication with other devices.

[0180] Understandable, Figure 12 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown. Figure 12 The components shown can be implemented using hardware, software, or a combination thereof.

[0181] The electronic device provided in this embodiment of the invention has a memory that stores a computer program. When the processor executes the computer program, it implements the training method for the multimodal face anti-fraud model and / or the face detection method disclosed in this embodiment of the invention.

[0182] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the training method for the multimodal face anti-fraud model and / or the face detection method disclosed in this invention.

[0183] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0184] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0185] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0186] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a multi-modal face anti-fraud model, characterized in that, The method comprises: obtaining multi-modal images and category labels of multiple face samples; the category label indicates that the face sample is a real face or a fake face; using a CLIP network in a face anti-fraud model, obtaining an initial image feature of each face sample based on the multi-modal image of each face sample, and mapping each initial image feature to the same semantic space to obtain a semantic feature of each face sample; using an attention network in the face anti-fraud model, performing a denoising operation on the initial image feature of each face sample to obtain a denoised image feature of each face sample; using a dual-space adaptive network in the face anti-fraud model, performing an enhancement operation on the denoised image feature of each face sample based on the semantic feature of each face sample to obtain an enhanced image feature of each face sample; the dual-space adaptive network comprises multiple network layers; the step of using the dual-space adaptive network in the face anti-fraud model to perform an enhancement operation on the denoised image feature of each face sample based on the semantic feature of each face sample to obtain an enhanced image feature of each face sample comprises: for each face sample, input the semantic feature and the denoised image feature of the face sample into a first network layer, and sequentially use each network layer to perform an alignment operation to obtain an aligned image feature output by each network layer; for each network layer, perform a residual operation on the aligned image feature output by the next network layer of the network layer after remapping the aligned image feature to obtain a target image feature output by the network layer; the target image feature output by the first network layer is taken as the enhanced image feature of the face sample; traverse the denoised image feature of each face sample to obtain the enhanced image feature of each face sample; using a classifier in the face anti-fraud model, calculating a prediction probability of each face sample being a fake face based on the enhanced image feature and the semantic feature of each face sample; based on the category label, the semantic feature and the prediction probability of each face sample, updating parameters of the attention network and the dual-space adaptive network in the face anti-fraud model to obtain a trained face anti-fraud model.

2. The method of claim 1, wherein, The CLIP network comprises an image encoder, a text encoder and a mapping layer, and the multi-modal image comprises a visible light image, an infrared image and a depth image; the step of using the CLIP network in the face anti-fraud model to obtain an initial image feature of each face sample based on the multi-modal image of each face sample and map each initial image feature to the same semantic space to obtain a semantic feature of each face sample comprises: using the image encoder, performing an extraction operation on the visible light image, the infrared image and the depth image of each face sample to obtain a visible light image feature, an infrared image feature and a depth image feature of each face sample and performing a fusion operation to obtain an initial image feature of each face sample; using the text encoder, performing an extraction operation on the two types of preset category label texts to obtain a text feature of each category label text; The initial image features of each face sample are mapped to a semantic space constructed by two text features by using the mapping layer, to obtain semantic features of each face sample.

3. The method of claim 1, wherein, Each face sample has a domain label, which represents the acquisition condition of the multi-modal image of the face sample. The step of denoising the initial image features of each face sample to obtain the denoised image features of each face sample comprises: The initial image features of all face samples are divided into multiple batch sample sets; wherein the initial image features with different domain labels exist in each batch sample set; The initial image features in each batch sample set are reorganized to obtain a reorganized sample set of each batch sample set; wherein the initial image features with the same domain label in the same position in the batch sample set and its reorganized sample set are the same; For each batch sample set, the domain noise and modal noise are estimated based on the batch sample set and its reorganized sample set, and each initial feature image in the batch sample set is denoised to obtain each denoised image feature, thereby obtaining the denoised image features of each face sample.

4. The method of claim 3, wherein, The step of estimating the domain noise and modal noise based on the batch sample set and its reorganized sample set, and denoising each initial feature image in the batch sample set to obtain each denoised image feature comprises: The batch sample set and its reorganized sample set are spliced to obtain a splicing matrix; the elements of each row in the splicing matrix include two initial image features in the same position in the batch sample set and its reorganized sample set; The splicing matrix is multiplied by the current query weight matrix of the attention network to obtain a query matrix, and the query matrix is split into two matrices with the same number of columns to obtain a first query matrix and a second query matrix; The splicing matrix is multiplied by the current key weight matrix of the attention network to obtain a key matrix, and the key matrix is split into two matrices with the same number of columns to obtain a first key matrix and a second key matrix; The splicing matrix is multiplied by the current value weight matrix of the attention network to obtain a value matrix; The first query matrix is multiplied by the transpose of the first key matrix to obtain a first matrix, and the first matrix is multiplied by a preset scaling factor and then normalized to obtain a sample feature matrix; The second query matrix is multiplied by the transpose of the second key matrix to obtain a second matrix, and the second matrix is multiplied by the scaling factor and then normalized to obtain an initial noise matrix; the initial noise matrix represents the domain noise and the modal noise; A preset adjustment parameter is multiplied by the initial noise matrix to obtain a target noise matrix, and the sample feature matrix is subtracted from the target noise matrix and multiplied by the value matrix to obtain a denoised feature matrix; each row of elements in the denoised feature matrix represents each denoised image feature.

5. The method of claim 1, wherein, The step of updating the parameters of the attention network and the dual-space adaptive network in the face fraud prevention model based on the class label, the semantic feature and the prediction probability of each face sample to obtain the trained face fraud prevention model comprises: Calculate a cross-entropy loss based on the class label and the semantic feature of each face sample; the cross-entropy loss represents the matching degree of the initial image feature and the semantic feature of all face samples; Calculate a classification loss based on the class label and the prediction probability of each face sample; the classification loss represents the consistency degree of the class label and the prediction probability of all face samples; Update the parameters of the attention network and the dual-space adaptive network in the face anti-fraud model based on the cross-entropy loss and the classification loss, to obtain the trained face anti-fraud model.

6. A face detection method, characterized by, The method comprises: Obtaining a multi-modal image of a face to be processed; Using the CLIP network in the trained face anti-fraud model, obtaining the initial image feature of the face to be processed based on the multi-modal image of the face to be processed, and mapping the initial image feature to a semantic space to obtain the semantic feature of the face to be processed; Using the attention network in the trained face anti-fraud model, performing a denoising operation on the initial image feature of the face to be processed to obtain a denoised image feature of the face to be processed; Using the dual-space adaptive network in the trained face anti-fraud model, performing an enhancement operation on the denoised image feature of the face to be processed based on the semantic feature of the face to be processed to obtain an enhanced image feature of the face to be processed; the dual-space adaptive network comprises a plurality of network layers; The step of using the dual-space adaptive network in the trained face anti-fraud model to perform an enhancement operation on the denoised image feature of the face to be processed based on the semantic feature of the face to be processed to obtain an enhanced image feature of the face to be processed comprises: inputting the semantic feature and the denoised image feature of the face to be processed into a first network layer, and sequentially using each network layer to perform an alignment operation to obtain an aligned image feature output by each network layer; for each network layer, performing a residual operation on the aligned image feature output by the next network layer after the network layer after remapping the aligned image feature to obtain a target image feature output by the network layer; taking the target image feature output by the first network layer as the enhanced image feature of the face to be processed; and traversing the denoised image feature of the face to be processed to obtain the enhanced image feature of the face to be processed; Using the classifier in the trained face anti-fraud model, calculating a prediction probability that the face to be processed is a fake face based on the enhanced image feature and the semantic feature of the face to be processed, and determining whether the face to be processed is a fake face according to the prediction probability; The trained face anti-fraud model is obtained according to the training method of the multi-modal face anti-fraud model in any one of claims 1 to 5.

7. An apparatus for training a multi-modal face anti-fraud model, comprising: The device comprises: A sample acquisition module for acquiring multi-modal images and class labels of a plurality of face samples; the class label indicates whether the face sample is a real face or a fake face; A sample processing module for using the CLIP network in the face anti-fraud model to obtain an initial image feature of each face sample based on the multi-modal image of each face sample, and mapping each initial image feature to the same semantic space to obtain a semantic feature of each face sample; The attention network in the face anti-fraud model is used to perform a denoising operation on the initial image feature of each face sample to obtain a denoised image feature of each face sample; The dual-space adaptive network in the face anti-fraud model is used to perform an enhancement operation on the denoised image feature of each face sample based on the semantic feature of the face sample to obtain an enhanced image feature of the face sample; the dual-space adaptive network includes multiple network layers, and the sample processing module is specifically configured to: For each face sample, the semantic feature and the denoised image feature of the face sample are input into a first network layer, and an alignment operation is performed on each network layer in sequence to obtain an aligned image feature output by each network layer; For each network layer, the aligned image feature output by a subsequent network layer of the network layer is remapped and subjected to a residual operation with the aligned image feature output by the network layer to obtain a target image feature output by the network layer; The target image feature output by the first network layer is taken as the enhanced image feature of the face sample; The denoised image features of each face sample are traversed to obtain the enhanced image feature of each face sample; The sample classification module is configured to use a classifier in the face anti-fraud model to calculate a predicted probability that each face sample is a fake face based on the enhanced image feature and the semantic feature of the face sample; The model updating module is configured to update parameters of the attention network and the dual-space adaptive network in the face anti-fraud model based on the class label, the semantic feature, and the predicted probability of each face sample to obtain a trained face anti-fraud model.

8. An electronic device, comprising: The storage medium stores a computer program, and the processor executes the computer program to implement the training method of the multi-modal face anti-fraud model according to any one of claims 1 to 5 and / or the face detection method according to claim 6.

9. A storage medium, characterized by The storage medium stores a computer program, and the processor executes the computer program to implement the training method of the multi-modal face anti-fraud model according to any one of claims 1 to 5 and / or the face detection method according to claim 6.

Images