Face anti-fraud model training method and device based on image-text comparison assisted learning

Abstract

The invention discloses a face anti-fraud model training method and device based on image-text comparison assisted learning, and belongs to the technical field of artificial intelligence. The method comprises the following steps: identifying the content of each sample image in an image set to obtain a description text corresponding to each sample image; generating text data corresponding to each sample image in combination with the description text and the label; and inputting each sample image in the image set into an image encoder in the multi-modal pre-training model, and inputting text data corresponding to each image into the text encoder in the multi-modal pre-training model, so as to train the multi-modal pre-training model and obtain a face anti-fraud model. According to the method, the image content description text and the face type label are integrated, the features are mixed, the central difference convolution adapter module is introduced to capture the fine features of the face, the features are subjected to semantic alignment through image-text comparison learning and matching, and the generalization ability of the model in different application scenes is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a face anti-fraud model training method and device based on image-text comparison assisted learning. Background Art

[0002] With the rapid development of information technology, face recognition technology has brought unprecedented convenience to our daily lives and is widely used in many important fields such as finance, transportation, government affairs, and retail, greatly improving our daily lives. Despite its remarkable success, face recognition systems may still be subject to impersonation attacks, such as photo printing, video replay, and 3D masks.

[0003] A lot of research has been conducted in academia and industry, and a variety of face anti-spoofing (FAS) algorithms have been proposed. The FAS algorithm is a key technology in face recognition systems and is used to distinguish real faces from forged faces (such as photos, videos, masks, etc.).

[0004] Previous studies have shown that face anti-spoofing algorithms can achieve satisfactory performance in scenarios where the training set and test set are distributed in the same way. However, the real and fake face spoofing information in different domains is affected by lighting, attack devices, cameras, environmental conditions, etc., which results in the inability of neural networks to extract essential spoofing information well, making the cross-domain generalization ability of current face anti-spoofing tasks poor. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a face anti-spoofing model training method and device based on image-text comparison assisted learning to improve the cross-domain generalization ability of face anti-spoofing tasks.

[0006] In a first aspect, the present application provides a face anti-spoofing model training method based on image-text comparison assisted learning, comprising:

[0007] Recognize the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type;

[0008] Combining the description text and the label to generate text data corresponding to each sample image;

[0009] Input each sample image in the image set into the image encoder in the multimodal pre-training model, and input the text data corresponding to each image into the text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain a face anti-fraud model.

[0010] The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application obtains the description text corresponding to each sample image by identifying the content of each sample image in the image set; wherein each sample image in the image set corresponds to a label indicating the type of face; the description text and the label are combined to generate text data corresponding to each sample image; each sample image in the image set is input into the image encoder in the multimodal pre-training model, and the text data corresponding to each image is input into the text encoder in the multimodal pre-training model to train the multimodal pre-training model to obtain a face anti-fraud model. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them into the image encoder and text encoder of the multimodal pre-training model together with the image, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face type in the image, and improve the generalization ability of the model in different application scenarios.

[0011] According to one embodiment of the present application, the image encoder includes a feature mixing module;

[0012] The feature mixing module is used to mix the feature statistical information of each sample image to generate image features with mixed styles.

[0013] According to one embodiment of the present application, according to the formula

[0014]

[0015]

[0016]

[0017] Mixing the feature statistics of each sample image;

[0018] Where MixSytle(x) represents the image features obtained after mixing, μ(x) represents the channel mean of the sample image features, σ(x) represents the channel standard deviation of each sample image feature, and γ mix and β mix represents the affine parameter corresponding to the style feature of the target sample image, λ is the weight sampled from the Beta distribution, x is a batch of image feature sequences, is a random image feature sequence generated by x.

[0019] In this embodiment, by introducing the feature mixing module, the model can not only learn the features of a single image, but also simulate the transition style features between multiple samples by mixing the feature statistical information of different images, thereby capturing richer texture and style information, greatly enriching the image features, and this mixing operation only involves operations at the feature level, not mixing at the image level, so the operation is simple and does not increase the computational complexity of the neural network.

[0020] According to one embodiment of the present application, the image encoder includes a center differential convolution adapter; the center differential convolution adapter is inserted into a Transformer block in the image encoder in a parallel manner;

[0021] The center difference convolution adapter is used to perform center difference convolution processing on each sample image.

[0022] According to one embodiment of the present application, by formula

[0023]

[0024] Perform central difference convolution processing on each sample image;

[0025] in, It is the same as the multi-head attention layer in the Transformer block Feature extractor of parallel centered differential convolution adapter modules, It is the same as the multilayer perceptron in the Transformer block Feature extractor of the parallel centered-difference convolutional adapter module. x is the input sequence of the Transformer block and y is the output sequence corresponding to x.

[0026] In this embodiment, a central difference convolution adapter is introduced in the image encoder and inserted into the Transformer block in parallel to perform central difference convolution on the sample image. By aggregating intensity and gradient information to capture inherent detail information of the face image, deceptive features such as deceptive artifacts can be effectively identified, thereby improving the model's ability to recognize image details.

[0027] According to an embodiment of the present application, the step of inputting each sample image in the image set into an image encoder in a multimodal pre-trained model, and inputting text data corresponding to each image into a text encoder in the multimodal pre-trained model to train the multimodal pre-trained model to obtain a face anti-spoofing model includes:

[0028] Acquire image features output by the image encoder and text features output by the text encoder;

[0029] Calculating a cross entropy loss based on the cosine similarity of the image features and the text features to obtain a category loss representing the difference between a predicted value output by the multimodal pre-training model and a true value corresponding to the label;

[0030] Semantically aligning each sample image and text data according to the image features and text features to obtain a matching loss representing a matching degree between each sample image and text data;

[0031] Calculate the target loss function according to the category loss and the matching loss;

[0032] Back propagation is performed in the multimodal pre-trained model according to the target loss function to update the parameters of the multimodal pre-trained model to obtain a face anti-spoofing model.

[0033] In this embodiment, the features of the image and the corresponding description text are extracted respectively by the image encoder and the text encoder, and then the similarity between these features is calculated using the cosine similarity to obtain the cross entropy loss, which reflects the difference between the model prediction and the true label. The training process also introduces the matching loss to measure the degree of semantic alignment between the image and text data, which further enhances the consistency of the features and improves the recognition accuracy of the model for different face types.

[0034] According to an embodiment of the present application, semantically aligning each sample image and text data according to the image features and text features to obtain a matching loss indicating a matching degree between each sample image and text data includes:

[0035] Selecting image features of negative samples for text features, and concatenating the selected image features with image features corresponding to the text features to obtain an image feature list; wherein the negative samples are sample images whose labels are fake faces;

[0036] Selecting text features of negative samples for image features, concatenating the selected text features with the text features corresponding to the image features to obtain a text feature list;

[0037] splicing the image feature list and the text feature list to obtain a spliced ​​feature list;

[0038] The concatenated feature list is projected into a two-dimensional vector and cross entropy is calculated with the label to obtain a matching loss representing the matching degree between each sample image and text data.

[0039] In this embodiment, through image-text comparison learning and matching, negative samples are used to splice image features and text features, and finally a spliced ​​feature list containing rich multimodal information is formed to perform deep semantic alignment on each sample image and text data. The matching loss is obtained by projecting the spliced ​​feature list into a two-dimensional vector and performing cross entropy calculation with the label to measure the model's prediction accuracy of the degree of matching between sample images and text data, effectively narrowing the difference between modalities, and compensating for the difference between source and target domains, significantly improving the cross-domain generalization capability.

[0040] In a second aspect, the present application provides a face anti-fraud method based on image-text comparison assisted learning, the method comprising:

[0041] Obtain a face image to be recognized;

[0042] Identify the content of the face image to be identified, and obtain a description text corresponding to the face image to be identified;

[0043] The facial image to be identified and the description text corresponding to the facial image to be identified are input into a pre-trained facial anti-fraud model to obtain a recognition result output by the facial anti-fraud model; wherein the facial anti-fraud model is trained using the method described in the first aspect above.

[0044] The face anti-fraud method based on image-text comparison assisted learning provided in the embodiment of the present application is performed by obtaining a face image to be identified; identifying the content of the face image to be identified to obtain a description text corresponding to the face image to be identified; inputting the face image to be identified and the description text corresponding to the face image to be identified into a pre-trained face anti-fraud model to obtain a recognition result output by the face anti-fraud model; wherein the face anti-fraud model is trained using the method described in the first aspect above. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them together with the image into the image encoder and text encoder of the multimodal pre-trained model, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face types in the image, and improve the generalization ability of the model in different application scenarios.

[0045] In a third aspect, the present application provides a face anti-fraud model training device based on image-text comparison assisted learning, the device comprising:

[0046] A first recognition module is used to recognize the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type;

[0047] A generating module, used for generating text data corresponding to each sample image by combining the description text and the label;

[0048] A training module is used to input each sample image in the image set into an image encoder in a multimodal pre-training model, and to input text data corresponding to each image into a text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model to obtain a face anti-spoofing model.

[0049] The face anti-fraud model training device based on image-text comparison assisted learning provided in the embodiment of the present application obtains the description text corresponding to each sample image by identifying the content of each sample image in the image set; wherein each sample image in the image set corresponds to a label indicating the type of face; the description text and the label are combined to generate text data corresponding to each sample image; each sample image in the image set is input into the image encoder in the multimodal pre-training model, and the text data corresponding to each image is input into the text encoder in the multimodal pre-training model to train the multimodal pre-training model to obtain a face anti-fraud model. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them into the image encoder and text encoder of the multimodal pre-training model together with the image, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face type in the image, and improve the generalization ability of the model in different application scenarios.

[0050] In a fourth aspect, the present application provides a face anti-fraud device based on image-text comparison assisted learning, the device comprising:

[0051] An acquisition module, used to acquire a face image to be recognized;

[0052] A second recognition module is used to recognize the content of the face image to be recognized and obtain a description text corresponding to the face image to be recognized;

[0053] An input module is used to input the facial image to be identified and the description text corresponding to the facial image to be identified into a pre-trained facial anti-fraud model to obtain the recognition result output by the facial anti-fraud model; wherein the facial anti-fraud model is trained using the method described in the first aspect above.

[0054] The face anti-fraud device based on image-text comparison assisted learning provided in the embodiment of the present application obtains a face image to be identified; identifies the content of the face image to be identified to obtain a description text corresponding to the face image to be identified; inputs the face image to be identified and the description text corresponding to the face image to be identified into a pre-trained face anti-fraud model to obtain a recognition result output by the face anti-fraud model; wherein the face anti-fraud model is trained using the method described in the first aspect above. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them together with the image into the image encoder and text encoder of the multimodal pre-trained model, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face types in the image, and improve the generalization ability of the model in different application scenarios.

[0055] In a fifth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first or second aspect above when executing the computer program.

[0056] In a sixth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first or second aspect above is implemented.

[0057] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first or second aspect above.

[0058] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0059] The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application obtains the description text corresponding to each sample image by identifying the content of each sample image in the image set; wherein each sample image in the image set corresponds to a label indicating the type of face; the description text and the label are combined to generate text data corresponding to each sample image; each sample image in the image set is input into the image encoder in the multimodal pre-training model, and the text data corresponding to each image is input into the text encoder in the multimodal pre-training model to train the multimodal pre-training model to obtain a face anti-fraud model. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them into the image encoder and text encoder of the multimodal pre-training model together with the image, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face type in the image, and improve the generalization ability of the model in different application scenarios.

[0060] Furthermore, by introducing the feature mixing module, the model can not only learn the features of a single image, but also simulate the transition style features between multiple samples by mixing the feature statistical information of different images, capturing richer texture and style information, greatly enriching the image features, and this mixing operation only involves operations at the feature level, not mixing at the image level. The operation is simple and does not increase the computational complexity of the neural network.

[0061] Furthermore, by introducing a central difference convolution adapter in the image encoder and inserting it into the Transformer block in parallel, the sample image is processed by central difference convolution. By aggregating intensity and gradient information to capture the inherent detail information of the face image, deceptive features such as deceptive artifacts can be effectively identified, thereby improving the model's ability to recognize image details.

[0062] Furthermore, in this embodiment, the features of the image and the corresponding description text are extracted respectively by the image encoder and the text encoder, and then the similarity between these features is calculated using cosine similarity to obtain the cross entropy loss, which reflects the difference between the model prediction and the true label. The training process also introduces matching loss to measure the degree of semantic alignment between the image and text data, further enhancing the consistency of the features and improving the model's recognition accuracy for different face types.

[0063] Furthermore, in this embodiment, through image-text comparison learning and matching, negative samples are used to splice image features and text features, and finally a spliced ​​feature list containing rich multimodal information is formed to perform deep semantic alignment of each sample image and text data. By projecting the spliced ​​feature list into a two-dimensional vector and performing cross entropy calculation with the label, the matching loss is obtained to measure the model's prediction accuracy of the degree of matching between sample images and text data, effectively narrowing the difference between modalities, and compensating for the difference between source and target domains, significantly improving the cross-domain generalization capability.

[0064] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0066] Figure 1 It is a flowchart of a face anti-fraud model training method based on image-text comparison assisted learning provided in an embodiment of the present application;

[0067] Figure 2 It is a schematic diagram of the architecture of the multimodal pre-training model provided in an embodiment of the present application;

[0068] Figure 3 is a schematic diagram of a multimodal pre-training model training process provided in an embodiment of the present application;

[0069] Figure 4 It is a schematic diagram of the processing process of the central differential convolution adapter provided in an embodiment of the present application;

[0070] Figure 5 It is a flowchart of a face anti-fraud method based on image-text comparison assisted learning provided in an embodiment of the present application;

[0071] Figure 6 It is a schematic diagram of the reasoning process of the face anti-fraud model provided in the embodiment of the present application;

[0072] Figure 7 It is a structural schematic diagram of a face anti-fraud model training device based on image-text comparison assisted learning provided in an embodiment of the present application;

[0073] Figure 8 It is a structural schematic diagram of a face anti-fraud device based on image-text comparison assisted learning provided in an embodiment of the present application;

[0074] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0076] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0077] The face anti-fraud model training method, face anti-fraud method and device provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0078] In the related art, the FAS algorithm may include a domain-adaptive FAS algorithm and a domain-generalized FAS algorithm. Due to the difference in data distribution between the source domain and the target domain, the features extracted after model mapping usually retain domain information. In order to obtain more adaptive features, the feature extractor needs to weaken domain-related information and enhance the utilization of living information to achieve migration on the target domain data. Based on this, a domain-generalized FAS algorithm based on domain adaptation came into being. Such methods usually map images from different domains to a feature space and align features in this space. Although the domain-adaptive FAS algorithm can improve the performance of face anti-fraud tasks in target domain data, a large amount of unlabeled target domain data is not always available.

[0079] It should be noted that the source domain refers to the data set used for model training, and the target domain is the data environment faced when the model is deployed. The target domain may differ from the source domain in data distribution.

[0080] In view of the problems of the domain-adaptive FAS algorithm, researchers began to study the FAS algorithm based on domain generalization, which no longer relies on target domain data and can learn a model with good generalization in unknown domains using only source domain data. The core of this type of method is to learn cross-domain invariant representations and strengthen living body information. Although these methods are better than previous methods in cross-domain performance, there are two problems: 1. Various auxiliary information is required, such as image depth information; 2. The model generalization ability is insufficient.

[0081] Therefore, re-examining the reasons for the poor generalization of face anti-fraud tasks can be attributed to the following: the true and false face deception information in different domains is affected by lighting, attack equipment, cameras, environmental conditions, etc., which causes the neural network to be unable to extract the essential deception information well, making the current face anti-fraud tasks have poor cross-domain generalization capabilities.

[0082] In order to solve at least one of the above technical problems, an embodiment of the present application proposes a face anti-fraud model training method and device based on image-text comparison assisted learning.

[0083] Among them, the face anti-fraud model training method based on image-text comparison assisted learning can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.

[0084] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0085] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0086] The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application may be an electronic device or a functional module or functional entity in the electronic device that can implement the method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0087] like Figure 1 As shown, the face anti-fraud model training method includes: step 110, step 120 and step 130.

[0088] Step 110: Identify the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type.

[0089] In the embodiment of the present application, the image set includes multiple sample images, which are images containing human faces, such as images of real faces and images of fake faces, and the labels are used to indicate the type of human faces. The types of human faces may include real faces and fake faces. A fake face indicates that the human face in the sample image is an image generated by a photo, head model, video, etc. of the person, while a real face indicates that the sample image is an image generated by a real human face. The labels may be constructed by text prompts, such as constructing a text prompt as "a photo of a <class>face", where <class>The labels are Real and Spoof, representing real face and spoof face respectively. The labels can also represent the face type in a digital way, for example, label "0" represents a real face, and label "1" represents a spoof face. Of course, labels can also be represented by other numbers, or by other means such as letters, symbols, etc., which are not limited in the embodiments of the present application.

[0090] In the embodiments of the present application, sample images can be collected in any manner. For example, sample images can be obtained through data sets such as OULU-NPU (O), CASIA-MFSD (C), CASIA-SURF (S) or other data sets. Among them, these data sets are widely used in the field of face recognition and liveness detection. These data sets provide a large number of real faces (sample images of real faces) and attack samples (sample images of fake faces).

[0091] After acquiring the image set, the content of each sample image can be identified to obtain the description text corresponding to each sample image. The description text can be text describing the content of the sample image, such as a short sentence or paragraph. In one example, for a sample image, the corresponding description text can be: a man wearing black-rimmed glasses and a white shirt.

[0092] Specifically, the content of each sample image can be identified in any way to obtain the description text corresponding to each sample image. For example, image recognition technology, such as Convolutional Neural Networks (CNN), can be used to analyze and understand the image content to obtain the description text corresponding to each sample image. Multimodal Large Language Model (MLLM) such as BLIP-2, GPT-4V, DALL-E, etc. can also be used to identify the content of each sample image to obtain the description text corresponding to each sample image.

[0093] Step 120: Generate text data corresponding to each sample image by combining the description text and the label.

[0094] In an embodiment of the present application, after obtaining the description text corresponding to each sample image in the image set, the description text can be combined with the corresponding label to generate text data corresponding to each sample image. The combination process can be achieved by simple string concatenation, for example, combining the description text and the label into a text file in a specific format. In one example, if a sample image is a real face and the description text is "a smiling young man", the generated text data can be "real face: a smiling young man".

[0095] In another example, during the training phase, batch processing is used, and the recommended batch size is 32 sample images per batch. To ensure consistency in data processing, it is recommended that the batch size be set to an even number, such as 24 or 32. In each batch, the source domain images consist of 16 real face images and 16 fake face images. The description text and labels of these images are combined to obtain text data. Among them, the text data corresponding to the 16 real face images is "This is a real face," and the 16 fake faces are accompanied by the text "This is a spoof face," where the content is the description text.

[0096] Step 130: input each sample image in the image set into the image encoder in the multimodal pre-training model, and input the text data corresponding to each image into the text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain a face anti-fraud model.

[0097] In an embodiment of the present application, the multimodal pre-training model is a deep learning model that can process and understand various types of data (such as text, images, audio, etc.). The multimodal pre-training model combines information of different modalities in a unified framework to enhance the ability to understand and generate complex data. In an embodiment of the present application, the multimodal pre-training model used can be a CLIP (Contrastive Language-Image Pretraining) model, DALL-E, Flamingo, PaLM-E, or other models, which is not limited to this embodiment of the present application.

[0098] In the embodiments of the present application, Figure 2 and Figure 3 As shown, the multimodal pre-trained model can include an image encoder and a text encoder. Among them, the image encoder is responsible for converting the input sample image into an image feature vector, and can use architectures such as convolutional neural networks or vision transformers (ViT) to extract important features in the image. The text encoder is used to convert the input text data into a text feature vector, and can use word embeddings such as Word2Vec, BERT (Bidirectional Encoder Representation from Transformers), and use the Transformer architecture to extract features in the text. Through the image encoder and text encoder, the model can convert data of different modalities into a unified feature representation, which is convenient for subsequent fusion and processing.

[0099] During the training process, the sample image can be input into the image encoder to obtain the image features output by the image encoder, and the text data can be input into the text encoder to obtain the text features output by the text encoder. The image features and text features are fused in the model to form a unified multimodal feature representation so that the model can use visual and language information at the same time. Use cosine similarity or other similarity metrics to evaluate the similarity between image features and text features, calculate the loss function, and use the back propagation algorithm to update the parameters of the model according to the value of the loss function so that the model can gradually converge to the optimal solution. Through training, the features of related images and texts can be brought closer, and irrelevant features can be pushed away. After completing the training, a face anti-fraud model is obtained.

[0100] The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application obtains the description text corresponding to each sample image by identifying the content of each sample image in the image set; wherein each sample image in the image set corresponds to a label indicating the type of face; the description text and the label are combined to generate text data corresponding to each sample image; each sample image in the image set is input into the image encoder in the multimodal pre-training model, and the text data corresponding to each image is input into the text encoder in the multimodal pre-training model to train the multimodal pre-training model to obtain a face anti-fraud model. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them into the image encoder and text encoder of the multimodal pre-training model together with the image, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face type in the image, and improve the generalization ability of the model in different application scenarios.

[0101] In some embodiments, the image encoder includes a feature mixing module;

[0102] The feature mixing module is used to mix the feature statistical information of each sample image to generate image features with mixed styles.

[0103] In this embodiment, if Figure 3 As shown, the image encoder may include a feature mixing module, which can mix the feature statistical information of each sample image to generate an image feature with a mixed style, that is, Figure 2 The image feature mixed vector in . The feature statistical information may include statistical information such as the mean and variance of the feature.

[0104] In this embodiment, the feature mixing module can mix samples from different domains of the same category or multiple samples from the same domain to reduce the problem that the category labels of the generated image features with mixed styles are difficult to define. The same category refers to sample images with the same labels, that is, real faces are one category and fake faces are another category.

[0105] In some embodiments, according to the formula

[0106]

[0107]

[0108]

[0109] Mixing the feature statistics of each sample image;

[0110] Where MixSytle(x) represents the image feature obtained after mixing, that is, the image feature mixing vector, μ(x) represents the channel mean of the sample image feature, σ(x) represents the channel standard deviation of each sample image feature, and γ mix and β mix represents the affine parameter corresponding to the style feature of the target sample image, λ is the weight sampled from the Beta distribution, x is a batch of image feature sequences, is a random image feature sequence generated by x. In one example, λ∈Beta(α,α), α can be 0.1.

[0111] In this embodiment, by introducing the feature mixing module, the model can not only learn the features of a single image, but also simulate the transition style features between multiple samples by mixing the feature statistical information of different images, thereby capturing richer texture and style information, greatly enriching the image features, and this mixing operation only involves operations at the feature level, not mixing at the image level, so the operation is simple and does not increase the computational complexity of the neural network.

[0112] In some embodiments, the image encoder includes a center differential convolution adapter; the center differential convolution adapter is inserted in parallel in a Transformer block in the image encoder;

[0113] The center difference convolution adapter is used to perform center difference convolution on each sample image.

[0114] In this embodiment, if Figure 3 As shown, the image encoder can be a ViT architecture. The image encoder can include N feed-forward Transformer blocks, and the i-th Transformer block can be formally represented as a feature extractor Each Transformer block contains a Multi-Head Attention (MHA) layer and Multi-Layer Perceptron (MLP) And each layer is accompanied by a normalization layer and a nonlinear activation layer. By omitting the connection, normalization, and activation layers, the reasoning process of each Transformer block can be formally expressed as:

[0115]

[0116] Here, x is the input sequence of the Transformer block and y is the output sequence corresponding to x.

[0117] like Figure 4 As shown, in this embodiment, the central difference convolution (CDC) adapter is inserted into the Transformer block in parallel, so that the central difference convolution adapter (CDC-Adapter) and the Transformer block share the same data input. Through this parallel connection method, the input sequence is cleverly passed to the central difference convolution adapter for a series of processing related to the face anti-spoofing task.

[0118] The structure of the central difference convolution adapter is as follows Figure 4 As shown, for the Transformer block integrated with the center difference convolution adapter, the reasoning process of the i-th Transformer block can be converted from the above formula to:

[0119]

[0120] in, is the same as the multi-head attention layer W in the Transformer block i MHA Feature extractor of parallel centered differential convolution adapter modules, is the multilayer perceptron W in the Transformer block i MLP Feature extractor of the parallel centered-difference convolutional adapter module. x is the input sequence of the Transformer block and y is the output sequence corresponding to x.

[0121] Inserted adapter module and First, it is randomly initialized and then updated by back-propagation. As can be seen from the above formula, the central difference convolution adapter shares the same data input with the Transformer block, indicating that the central difference convolution adapter transforms the input sequence features from the original space to a new space related to face anti-spoofing. The processing process mainly involves the extraction and processing of sequence features. The specific process is as follows: In the ViT architecture, the input sequence x of the Transformer block has no spatial structure. In order to extract local information, the structure of the sequence x is reconstructed into a two-dimensional structure. Given an input sequence x∈R N*C , where N represents the number of sequences and C represents the sequence feature channel, which is first transformed into x∈x∈R H*W*C , where H*W=N. Then, change the order of dimensions to obtain x∈R C*H*W Thus, the sequence is represented in the style of a 2D image, enabling learning using 2D convolutional techniques.

[0122] exist Figure 4 In the central difference convolution adapter shown, it is important to note that the shapes of the input and output sequence features must be consistent. Therefore, after the central difference convolution, the sequence features need to be converted from two dimensions to one dimension, that is, flattened. In order to keep the shapes of the input and output features consistent, a nonlinear activation function needs to be applied before the dimension increase, and then the dimension increase output is performed.

[0123] In this embodiment, a central difference convolution adapter is introduced in the image encoder and inserted into the Transformer block in parallel to perform central difference convolution on the sample image. By aggregating intensity and gradient information to capture inherent detail information of the face image, deceptive features such as deceptive artifacts can be effectively identified, thereby improving the model's ability to recognize image details.

[0124] In some embodiments, each sample image in the image set is input into an image encoder in a multimodal pre-training model, and text data corresponding to each image is input into a text encoder in the multimodal pre-training model to train the multimodal pre-training model to obtain a face anti-spoofing model, including:

[0125] Obtain image features output by the image encoder and text features output by the text encoder;

[0126] The cross entropy loss is calculated based on the cosine similarity of image features and text features to obtain the category loss representing the difference between the predicted value output by the multimodal pre-training model and the true value corresponding to the label;

[0127] Semantically align each sample image and text data according to the image features and text features, and obtain a matching loss representing the matching degree of each sample image and text data;

[0128] Calculate the target loss function based on the category loss and matching loss;

[0129] Back propagation is performed in the multimodal pre-trained model according to the target loss function to update the parameters of the multimodal pre-trained model to obtain a face anti-spoofing model.

[0130] In this embodiment, taking the CLIP model as an example of a multimodal pre-training model, the sample image is passed through an image encoder to obtain image features, and the text data is passed through a text encoder to obtain text features, and the cosine similarity of the image features and the text features is calculated. Specifically, for each sample image, two cosine similarities can be calculated, and the two cosine similarities include the cosine similarity between the image features of the sample image and the text features of the positive sample, and the cosine similarity between the image features of the sample image and the text features of the negative sample. Then, the cross entropy loss is calculated using these two cosine similarity values. Since the face anti-fraud task is a binary classification problem, the sigmoid function can be used to convert the similarity value into a probability, and then the cross entropy loss is calculated by probability to obtain the category loss of the difference between the predicted value and the true value corresponding to the label. Among them, the positive sample represents a sample image with a label of a real face, and the negative sample represents a sample image with a label of a fake face.

[0131] In this embodiment, semantic alignment refers to the process of making data from different modalities (such as images and text) consistent at the semantic level in multimodal learning. Through semantic alignment, features from different sources can be adjusted and matched so that they can correspond and relate to each other when representing the same content. Specifically, image features and text features can be integrated together for semantic alignment through splicing, weighting, or more complex methods such as attention mechanisms.

[0132] During the semantic alignment process, you can also calculate the matching loss that represents the degree of matching between each sample image and text data, and then calculate the target loss function in combination with the category loss:

[0133] L total =L cls +δL itm

[0134] Among them, L total represents the target loss function, L cls represents the category loss, L itm represents the matching loss, and δ represents the matching loss coefficient.

[0135] In this embodiment, the features of the image and the corresponding description text are extracted respectively by the image encoder and the text encoder, and then the similarity between these features is calculated using the cosine similarity to obtain the cross entropy loss, which reflects the difference between the model prediction and the true label. The training process also introduces the matching loss to measure the degree of semantic alignment between the image and text data, which further enhances the consistency of the features and improves the recognition accuracy of the model for different face types.

[0136] In some embodiments, semantic alignment is performed on each sample image and text data according to image features and text features to obtain a matching loss indicating a matching degree between each sample image and text data, including:

[0137] Select image features of negative samples for text features, and concatenate the selected image features with image features corresponding to the text features to obtain an image feature list; wherein the negative samples are sample images whose labels are fake faces;

[0138] Selecting text features of negative samples for image features, concatenating the selected text features with the text features corresponding to the image features to obtain a text feature list;

[0139] Splicing the image feature list and the text feature list to obtain a spliced ​​feature list;

[0140] The concatenated feature list is projected into a two-dimensional vector and the cross entropy is calculated with the label to obtain the matching loss representing the degree of matching between each sample image and text data.

[0141] In this embodiment, the text feature f output by the text encoder can be t The first dimension of is averaged to obtain the text feature vector f t ∈R B*512 , where B represents the number of sample images in the batch and 512 represents the dimension length of the feature. The image feature f output by the image encoder i Perform normalization so that f i ∈R B*512 , the similarity between image features and text features is calculated by the following formula, and two B-order square matrices sim i2t and sim t2i .

[0142] sim i2t =norm(f i T )norm(f t )

[0143] sim t2i =norm(f t T )norm(f i )

[0144] Among them, square matrix sim i2t and sim t2i The elements on the diagonal represent the maximum similarity score between the corresponding features, through the square matrix sim i2t and sim t2i Construct a negative sample pair, that is, clear all the diagonal data to zero, and then randomly select a non-zero data in each row of the matrix as a negative sample, and select B data in total. The meaning of this process is: for each text feature, select an image feature of a negative sample, select B image features according to the above method, and the selected image features are represented as f i_neg , for each image feature, select a negative sample text feature, select B text features according to the above method, and the selected text features are represented as f t_neg .

[0145] The selected image feature f i_neg Image features corresponding to text features f i After splicing, the image feature list list1 is obtained as [f i ||f i_neg ||f i ],list1∈R 3B*512 . The selected text feature f t_neg The text feature f corresponding to the image feature t After concatenation, the text feature list list2 is obtained as [f t ||f t ||f t _neg],list2∈R 3B*512 .

[0146] Then concatenate list1 and list2 into concatenated feature list list3, list3∈R 3B*1024 . Project list3 to a two-dimensional vector, denoted by f proj , and generate a label Label∈{0,1}, the first B labels are 1, and the next 2B labels are 0. proj Do cross entropy with Label, and record the matching loss as L itm By omitting connections, normalization, and feature shape transformation, the entire reasoning process can be formally expressed as:

[0147] f t =TextEncoder(text), f t ∈R B*512

[0148] f i =ImageEncoder(image), f i ∈R B*512

[0149] sim i2t =(f i )(f t T ), sim i2t ∈[B, B]

[0150] sim t2i =(f t )(f i T ), sim t2i ∈[B, B]

[0151] list3=[list1||list2],list3∈R 3B*1024

[0152] f proj =FC(list3)

[0153]

[0154] Among them, [·||·] represents the concatenated features, H(·) is the cross entropy loss, FC(·) is a fully connected layer, and Label is the label of the two-dimensional vector.

[0155] In this embodiment, through image-text comparison learning and matching, negative samples are used to splice image features and text features, and finally a spliced ​​feature list containing rich multimodal information is formed to perform deep semantic alignment on each sample image and text data. The matching loss is obtained by projecting the spliced ​​feature list into a two-dimensional vector and performing cross entropy calculation with the label to measure the model's prediction accuracy of the degree of matching between sample images and text data, effectively narrowing the difference between modalities, and compensating for the difference between source and target domains, significantly improving the cross-domain generalization capability.

[0156] The present application also provides a face anti-fraud method based on image-text comparison assisted learning, such as Figure 5 As shown, the face anti-fraud method includes: step 510, step 520 and step 530.

[0157] Step 510: Obtain a face image to be recognized;

[0158] Step 520: Identify the content of the face image to be identified, and obtain a description text corresponding to the face image to be identified;

[0159] Step 530: input the facial image to be identified and the description text corresponding to the facial image to be identified into a pre-trained facial anti-fraud model to obtain the recognition result output by the facial anti-fraud model; wherein, the facial anti-fraud model is trained using the facial anti-fraud model training method based on image-text comparison assisted learning as mentioned above.

[0160] In the embodiment of the present application, the reasoning process of the face anti-fraud model is as follows: Figure 6 shown.

[0161] The face anti-fraud method based on image-text comparison assisted learning provided in the embodiment of the present application is achieved by obtaining a face image to be identified; identifying the content of the face image to be identified to obtain a description text corresponding to the face image to be identified; inputting the face image to be identified and the description text corresponding to the face image to be identified into a pre-trained face anti-fraud model to obtain a recognition result output by the face anti-fraud model; wherein the face anti-fraud model is trained using the method of the first aspect as described above. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them together with the image into the image encoder and text encoder of the multimodal pre-trained model, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face types in the image, and improve the generalization ability of the model in different application scenarios.

[0162] The following introduces the experimental evaluation process of the face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application.

[0163] Datasets and evaluation protocols: The evaluation process of this experiment is based on three test protocols and eight internationally commonly used face anti-spoofing datasets to verify the proposed algorithm. Protocol 1 uses the OULU-NPU (O), CASIA-MFSD (C), Idiap Replay-Attack (I), and MSU-MFSD (M) datasets; Protocol 2 uses the WMCA (W), CASIA-CeFA (C), and CASIA-SURF (S) datasets. Both Protocol 1 and Protocol 2 use the leave-one-out method for cross-domain testing. For example, among the four datasets O, C, I, and M, three are randomly selected as source domains, and the remaining one is used as the target domain to conduct domain generalization experiments. For example, in the MCI->O evaluation protocol, C, I, and M are source domains, and O is the target domain. Protocol 3 is a cross-domain test of a single source domain and a single target domain, using the M, C, I, and O datasets. Each source domain has three combinations, with a total of 12 test scenarios. It should be noted that the evaluation process of this experiment selects the CelebA-Spoof dataset as supplementary training data.

[0164] Experimental evaluation method and experimental details: The evaluation process of this experiment uses the half total error rate (HTER), the area under the receiver operating characteristic curve (AUC), and the true positive rate (TPR) when the false positive rate (FPR) is fixed at 1% as the main performance evaluation indicators. The experiment is based on the PyTorch 2.1 framework, and the training and testing are completed on the NVIDIA Tesla T4 GPU (Graphics Processing Unit). In this experimental evaluation process, YOLO5Face is first used for face detection, and then the detected face image is cropped to a size of 224×224 pixels to adapt to the requirements of the model input. In the training stage, the multimodal large model BLIP2 is used to automatically generate description texts for the source domain images. These description texts are combined with the corresponding image labels to form the composite text language input of the model. During the training process, the Adam optimizer is selected, and the initial learning rate is set to 1e-6 and the weight decay is set to 0.05. The batch size is 32 and the epoch is 240.

[0165] Effectiveness of each component: To evaluate the contribution of each component in the algorithm framework, such as the feature mixing module (denoted as "mix"), the center difference convolution adapter module (denoted as "cdc"), and the semantic alignment (denoted as "itc / itm"), an ablation experiment was performed on Protocol 1. Each component was gradually added to the baseline (denoted as "baseline") to evaluate its effect.

[0166] Table 1 Ablation test results of each component based on protocol 1 (unit: %)

[0167]

[0168] Table 2 Ablation experimental results of different text prompts based on Protocol 1 (unit: %)

[0169]

[0170] Table 3 Ablation experimental results of different text prompts based on Protocol 2 (unit: %)

[0171]

[0172] Table 4 Experimental results of different image-text contrast learning loss coefficients based on protocol 1 (unit: %)

[0173]

[0174] As shown in Table 1, after adding the cdc module, the average HTER of the four scenarios dropped from 3.06% to 2.30%, significantly improving the generalization ability. Subsequently, the itc / itm module was added to calculate the matching loss, and the average HTER of the four scenarios was further reduced to 1.43%. Especially in the OCI->M experiment, TPR@FPR=1% reached 100%, and HTER dropped from 5.27% to 1.19%, indicating that the itc / itm module can effectively narrow the modality gap and enhance generalization. After adding the mix module, the overall performance was further improved, and the average HTER dropped from 1.43% to 1.28%. Compared with the experiment without the mix module, the HTER index (the same AUC conclusion) dropped by 0.48%, 0%, 0.79% and 0.28% in the M, C, I and O scenarios, respectively. Especially in the COM->I experiment, TPR@FPR=1% reached 100%, and HTER dropped from 1.50% to 0.71%, confirming that the mix module did improve the generalization performance. Ablation experiments also show that when only mix and itc / itm are included, the average HTER increases from 1.28% to 2.08%, indicating that the cdc module contributes significantly to the performance improvement.

[0175] The impact of different textual cues: When evaluating the textual data effect of label + description text, this experimental evaluation process conducted experiments on Protocol 1 and Protocol 2 respectively to examine its generalization performance under different textual data conditions.

[0176] Table 5 Examples of real face and fake face labels

[0177]

[0178] Table 5 shows examples of real and fake face labels. Tables 2 and 3 show that using text data of labels + description text, the average HTER is reduced by 0.65% in Protocol 1 and 0.29% in Protocol 2 compared to using only labels as text data. In particular, in Protocol 1, the HTER, AUC, and TPR@FPR=1% indicators of the four experiments are better than or equal to using only labels as text data, which shows that incorporating the description text of the image into the text has significant advantages.

[0179] Comparison of learning loss coefficients: In the four experiments of Protocol 1, the loss coefficients were set to 0.3, 0.5, 0.8, and 1.0 respectively to test the changes in generalization performance. Table 4 shows that when the loss coefficient is set to 1.0, the model achieves the best performance. Observing the settings of different loss coefficients, the advantages of model performance are more subtle, and the effects of settings of 0.3 and 0.8 are not as good as 0.5. Based on the experimental results, it is recommended to set the loss coefficient to 1.0.

[0180] The face anti-fraud model training method based on image-text comparison assisted learning provided in the embodiment of the present application can be executed by a face anti-fraud model training device based on image-text comparison assisted learning. In the embodiment of the present application, the face anti-fraud model training method based on image-text comparison assisted learning is executed by a face anti-fraud model training device based on image-text comparison assisted learning as an example to illustrate the face anti-fraud model training device based on image-text comparison assisted learning provided in the embodiment of the present application.

[0181] The embodiment of the present application also provides a face anti-fraud model training device based on image-text comparison assisted learning.

[0182] like Figure 7 As shown, the face anti-fraud model training device based on image-text comparison assisted learning includes:

[0183] The first recognition module 710 is used to recognize the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type;

[0184] A generating module 720, for generating text data corresponding to each sample image by combining the description text and the label;

[0185] The training module 730 is used to input each sample image in the image set into the image encoder in the multimodal pre-training model, and input the text data corresponding to each image into the text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain a face anti-fraud model.

[0186] The face anti-fraud model training device based on image-text comparison assisted learning provided in the embodiment of the present application obtains the description text corresponding to each sample image by identifying the content of each sample image in the image set; wherein each sample image in the image set corresponds to a label indicating the type of face; generates text data corresponding to each sample image by combining the description text and the label; inputs each sample image in the image set into the image encoder in the multimodal pre-training model, and inputs the text data corresponding to each image into the text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain the face anti-fraud model. The embodiment of the present application generates rich text data by integrating the image content description text and the face type label, and inputs it into the image encoder and text encoder of the multimodal pre-training model together with the image, so as to enable the model to learn the cross-modal joint feature representation, better capture the subtle clues related to the face type in the image, and improve the generalization ability of the model in different application scenarios.

[0187] The face anti-fraud method provided in the embodiment of the present application can be executed by a face anti-fraud device based on image-text comparison assisted learning. In the embodiment of the present application, the face anti-fraud device based on image-text comparison assisted learning executes the face anti-fraud method based on image-text comparison assisted learning as an example to illustrate the face anti-fraud device based on image-text comparison assisted learning provided in the embodiment of the present application.

[0188] like Figure 8 As shown, the face anti-fraud device based on image-text comparison assisted learning includes:

[0189] An acquisition module 810 is used to acquire a face image to be recognized;

[0190] The second recognition module 820 is used to recognize the content of the face image to be recognized and obtain the description text corresponding to the face image to be recognized;

[0191] The input module 830 is used to input the facial image to be identified and the description text corresponding to the facial image to be identified into a pre-trained facial anti-fraud model to obtain the recognition result output by the facial anti-fraud model; wherein the facial anti-fraud model is trained using the method of the first aspect mentioned above.

[0192] The face anti-fraud device based on image-text comparison assisted learning provided in the embodiment of the present application obtains a face image to be identified; identifies the content of the face image to be identified to obtain a description text corresponding to the face image to be identified; inputs the face image to be identified and the description text corresponding to the face image to be identified into a pre-trained face anti-fraud model to obtain a recognition result output by the face anti-fraud model; wherein the face anti-fraud model is trained using the method of the first aspect as described above. The embodiment of the present application generates rich text data by integrating image content description text and face type labels, and inputs them together with the image into the image encoder and text encoder of the multimodal pre-trained model, so that the model can learn cross-modal joint feature representation, better capture subtle clues related to face types in the image, and improve the generalization ability of the model in different application scenarios.

[0193] The face anti-fraud model training device based on image-text comparison assisted learning and the face anti-fraud device based on image-text comparison assisted learning in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0194] The face anti-fraud model training device based on image-text comparison assisted learning and the face anti-fraud device based on image-text comparison assisted learning in the embodiments of the present application can be a device with an operating system. The operating system can be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0195] In some embodiments, Fig. 9 As shown, an embodiment of the present application also provides an electronic device 900, including a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the program is executed by the processor 901, the various processes of the above-mentioned face anti-fraud model training method based on image-text comparison assisted learning or the face anti-fraud method embodiment based on image-text comparison assisted learning are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0196] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0197] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned face anti-fraud model training method based on image and text comparison assisted learning or the face anti-fraud method embodiment based on image and text comparison assisted learning, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0198] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0199] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned face anti-fraud model training method based on image-text comparison assisted learning or the face anti-fraud method based on image-text comparison assisted learning.

[0200] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0201] An embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned face anti-fraud model training method based on image-text comparison assisted learning or the face anti-fraud method embodiment based on image-text comparison assisted learning, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0202] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0203] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0204] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0205] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.< / class> < / class>

Claims

1. A face anti-fraud model training method based on image-text comparison assisted learning, characterized in that: include: Identify the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type; Combining the description text and the label to generate text data corresponding to each sample image; Input each sample image in the image set into the image encoder in the multimodal pre-training model, and input the text data corresponding to each image into the text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain a face anti-fraud model.

2. The method according to claim 1, characterized in that: The image encoder includes a feature mixing module; The feature mixing module is used to mix the feature statistical information of each sample image to generate image features with mixed styles.

3. The method according to claim 1, characterized in that According to the formula Mixing the feature statistics of each sample image; Where MixSytle(x) represents the image features obtained after mixing, μ(x) represents the channel mean of the sample image features, σ(x) represents the channel standard deviation of each sample image feature, and γ mix and β mix represents the affine parameter corresponding to the style feature of the target sample image, λ is the weight sampled from the Beta distribution, x is a batch of image feature sequences, is a random image feature sequence generated by x.

4. The method according to claim 1, characterized in that: The image encoder includes a center differential convolution adapter; the center differential convolution adapter is inserted in parallel into a Transformer block in the image encoder; The center difference convolution adapter is used to perform center difference convolution processing on each sample image.

5. The method according to claim 4, characterized in that By formula Perform central difference convolution processing on each sample image; in, is the same as the multi-head attention layer W in the Transformer block i MHA Feature extractor of parallel centered differential convolution adapter modules, is the multilayer perceptron W in the Transformer block i MLP Feature extractor of the parallel centered-difference convolutional adapter module. x is the input sequence of the Transformer block and y is the output sequence corresponding to x.

6. The method according to claim 1, characterized in that The step of inputting each sample image in the image set into an image encoder in a multimodal pre-training model, and inputting text data corresponding to each image into a text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model and obtain a face anti-spoofing model, comprises: Acquire image features output by the image encoder and text features output by the text encoder; Calculating a cross entropy loss based on the cosine similarity of the image features and the text features to obtain a category loss representing the difference between a predicted value output by the multimodal pre-training model and a true value corresponding to the label; Semantically aligning each sample image and text data according to the image features and text features to obtain a matching loss representing a matching degree between each sample image and text data; Calculate the target loss function according to the category loss and the matching loss; Back propagation is performed in the multimodal pre-trained model according to the target loss function to update the parameters of the multimodal pre-trained model to obtain a face anti-spoofing model.

7. The method according to claim 6, characterized in that The semantic alignment of each sample image and text data according to the image features and text features to obtain a matching loss indicating the degree of matching between each sample image and text data includes: Selecting image features of negative samples for text features, and concatenating the selected image features with image features corresponding to the text features to obtain an image feature list; wherein the negative samples are sample images whose labels are fake faces; Selecting text features of negative samples for image features, concatenating the selected text features with the text features corresponding to the image features to obtain a text feature list; splicing the image feature list and the text feature list to obtain a spliced ​​feature list; The concatenated feature list is projected into a two-dimensional vector and cross entropy is calculated with the label to obtain a matching loss representing the matching degree between each sample image and text data.

8. A face anti-fraud method based on image-text comparison assisted learning, characterized in that: include: Obtain a face image to be recognized; Identify the content of the face image to be identified, and obtain a description text corresponding to the face image to be identified; The facial image to be identified and the description text corresponding to the facial image to be identified are input into a pre-trained facial anti-fraud model to obtain a recognition result output by the facial anti-fraud model; wherein the facial anti-fraud model is trained using the method described in any one of claims 1 to 7.

9. A face anti-fraud model training device based on image-text comparison assisted learning, characterized in that: include: A first recognition module is used to recognize the content of each sample image in the image set to obtain a description text corresponding to each sample image; wherein each sample image in the image set corresponds to a label indicating a face type; A generating module, used for generating text data corresponding to each sample image by combining the description text and the label; A training module is used to input each sample image in the image set into an image encoder in a multimodal pre-training model, and to input text data corresponding to each image into a text encoder in the multimodal pre-training model, so as to train the multimodal pre-training model to obtain a face anti-spoofing model.

10. A face anti-fraud device based on image-text comparison assisted learning, characterized in that: include: An acquisition module, used to acquire a face image to be recognized; A second recognition module is used to recognize the content of the face image to be recognized and obtain a description text corresponding to the face image to be recognized; An input module is used to input the facial image to be identified and the description text corresponding to the facial image to be identified into a pre-trained facial anti-fraud model to obtain the recognition result output by the facial anti-fraud model; wherein the facial anti-fraud model is trained using the method described in any one of claims 1-7.

Images