Face in-vivo detection method, device and equipment based on multi-modal large language model, and storage medium

Abstract

The invention discloses a face living body detection method, device and equipment based on a multi-modal large language model and a storage medium, and relates to the technical field of image detection, and the method comprises the steps: inputting a to-be-recognized face image, a target auxiliary modal image and a user instruction into a trained multi-modal large language model, the multi-modal large language model comprises a visual token mask, and the multi-task output result comprises at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result and an attack positioning result. According to the invention, the safety and reliability of human face in-vivo detection can be improved.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular, to a face liveness detection method, device, equipment, and storage medium based on a multimodal large language model. Background Technique

[0002] Face Anti-Spoofing (FAS) plays a crucial role in facial recognition systems and can effectively prevent various presentation attacks (such as printing, replay, and 3D masks, etc.). However, traditional face liveness detection methods still face many challenges, including limited generalization ability, poor interpretability, and the lack of fine-grained localization of attack regions.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a face liveness detection method, device, equipment, and storage medium based on a multimodal large language model, aiming to solve the technical problems that traditional face liveness detection methods still face insufficient model generalization ability, poor model interpretability, and the lack of fine-grained localization of attack regions.

[0005] To achieve the above object, this application proposes a face liveness detection method based on a multimodal large language model. The face liveness detection method based on a multimodal large language model includes:

[0006] Input the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multimodal large language model to obtain a multi-task output result for the user instruction. Among them, the multimodal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason inference result, and an attack localization result.

[0007] In one embodiment, the multimodal large language model further includes a mapper and a target large language model; among them, before the step of inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multimodal large language model to obtain a multi-task output result for the user instruction, it further includes:

[0008] Construct a face image pre-training set and a face image supervised fine-tuning data set;

[0009] After pre-training the mapper in the multi-modal large language model based on the face image pre-training set, the mapper and the target large language model in the multi-modal large language model are supervised and fine-tuned based on the face image supervised fine-tuning data set to obtain a trained multi-modal large language model.

[0010] In one embodiment, the steps of constructing the face image pre-training set and the face image supervised fine-tuning data set include:

[0011] Construct a joint data set based on face images of different categories, and label the prosthesis regions of the face images in the joint data set;

[0012] Construct the instruction information of each face image in the joint data set;

[0013] Based on the instruction information, divide the joint data set into an initial pre-training data set and an initial supervised fine-tuning data set;

[0014] Based on the text-image similarity threshold of each face image in the initial pre-training data set, filter the initial pre-training data set to obtain the face image pre-training set;

[0015] Based on the large language model, specific vocabulary, and manual review results, filter the initial supervised fine-tuning data set to obtain the face image supervised fine-tuning data set.

[0016] In one embodiment, the multi-modal large language model further includes a visual encoder and a text tokenizer; wherein, the step of pre-training the mapper in the multi-modal large language model based on the face image pre-training set includes:

[0017] Determine a first visible light image and a first text prompt from the face image pre-training set, and determine a first auxiliary modal image according to the first visible light image;

[0018] Input the first visible light image, the first label data, and the first auxiliary modal image into the multi-modal large language model, extract first visual features from the first visible light image and the first auxiliary modal image through the visual encoder, and process the first text prompt through the text tokenizer to obtain first text tokens;

[0019] Map the first visual features to a feature space through the mapper to obtain first mapped features;

[0020] Process the first text tokens and the first mapped features through the target large language model to obtain a first cross-entropy loss, and pre-train the mapper according to the first cross-entropy loss.

[0021] In one embodiment, the multimodal large language model further includes a visual encoder and a text tokenizer; wherein, the step of performing supervised fine-tuning on the mapper and the target large language model in the multimodal large language model based on the face image supervised fine-tuning dataset includes:

[0022] Determine a second visible light image and a second text prompt from the face image supervised fine-tuning dataset, and determine a second auxiliary modality image according to the second visible light image;

[0023] Input the second visible light image, the second label data, and the second auxiliary modality image into the multimodal large language model, extract second visual features from the second visible light image and the second auxiliary modality image through the visual encoder, and process the second text prompt through the text tokenizer to obtain second text tokens;

[0024] Perform masking processing on the second visual features according to the second label data through the visual token masker to obtain target visual features, and map the target visual features to the feature space through the mapper to obtain second mapped features;

[0025] Process the second text tokens and the second mapped features through the target large language model to obtain a second cross-entropy loss, and perform supervised fine-tuning on the mapper according to the second cross-entropy loss.

[0026] In one embodiment, the step of performing masking processing on the second visual features according to the second text prompt through the visual token masker to obtain target visual features includes:

[0027] Determine the similarity between each visual image block in the second visual features and the second text prompt through the visual token masker;

[0028] After sorting all the visual image blocks of the second visual features according to the similarity through the visual token masker, obtain a sorting result;

[0029] Screen out important visual image blocks and remaining visual image blocks of the second visual features according to the sorting result through the visual token masker;

[0030] Randomly determine visual image blocks to be masked from the remaining visual image blocks through the visual token masker, and perform masking processing on the visual image blocks to be masked in the second visual features through the visual token masker to obtain target visual features.

[0031] In one embodiment, before the step of inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multimodal large language model to obtain the multitask output result for the user instruction, the method further includes:

[0032] Obtain the face image to be recognized;

[0033] Perform grayscale processing on the face image to be recognized to obtain a grayscale image;

[0034] Perform orientation gradient processing on the face image to be recognized to obtain a histogram of oriented gradients;

[0035] Perform local binarization processing on the face image to be recognized to obtain a local binary pattern image;

[0036] Based on the grayscale image, the histogram of oriented gradients, and the local binary pattern image, obtain the target auxiliary modality image.

[0037] In addition, to achieve the above object, the present application further provides a face liveness detection device based on a multimodal large language model, where the face liveness detection device based on a multimodal large language model includes:

[0038] An input module, configured to input the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multimodal large language model to obtain the multitask output result for the user instruction, where the multimodal large language model includes a visual token masker, and the multitask output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason inference result, and an attack localization result.

[0039] In addition, to achieve the above object, the present application further provides a face liveness detection device based on a multimodal large language model, where the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the face liveness detection method based on a multimodal large language model as described above.

[0040] In addition, to achieve the above object, the present application further provides a storage medium, where the storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the face liveness detection method based on a multimodal large language model as described above.

[0041] In addition, to achieve the above object, the present application further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the face liveness detection method based on a multimodal large language model as described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] The face liveness detection method, device, equipment and storage medium based on the multi-modal large language model proposed in this application input the face image to be recognized, the target auxiliary modal image and the user instruction into the trained multi-modal large language model to obtain the multi-task output result for the user instruction. Among them, the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result and an attack localization result. This solves the technical problems that traditional face liveness detection methods still face insufficient model generalization ability, poor model interpretability and lack of coarse and fine-grained localization of attack areas. Compared with the prior art, this application inputs the face image to be recognized and the target auxiliary image modal image into the multi-modal large language model for fusion, thereby enhancing the model's representation ability. Then, the visual token masker in the multi-modal large language model randomly masks the fused image features to reduce interference information and further improve the model's generalization ability. In addition, this application also extends the traditional face liveness detection task to four sub-tasks: coarse-grained classification, fine-grained classification, reasoning and attack localization, which can more comprehensively cope with various attack means, thereby improving the security and reliability of face liveness detection. Description of the Drawings

[0044] The drawings here are incorporated into the specification and constitute a part of this specification, showing the embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart provided for Embodiment 1 of the face liveness detection method based on the multi-modal large language model of this application;

[0047] Figure 2 It is an input-output schematic diagram of the multi-modal large language model provided for Embodiment 1 of the face liveness detection method based on the multi-modal large language model of this application;

[0048] Figure 3 It is a schematic flowchart provided for Embodiment 2 of the face liveness detection method based on the multi-modal large language model of this application;

[0049] Figure 4Schematic diagram of data structure provided by Embodiment 2 of the face liveness detection method based on a multimodal large language model in this application;

[0050] Figure 5 Framework diagram of the multimodal large language model provided by Embodiment 2 of the face liveness detection method based on a multimodal large language model in this application;

[0051] Figure 6 Schematic diagram of visual feature masking provided by Embodiment 2 of the face liveness detection method based on a multimodal large language model in this application;

[0052] Figure 7 Schematic diagram of the module structure of the face liveness detection device based on a multimodal large language model in an embodiment of this application;

[0053] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the face liveness detection method based on a multimodal large language model in an embodiment of this application.

[0054] The implementation, functional features, and advantages of this application will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0056] To better understand the technical solutions of this application, the following will be described in detail in combination with the accompanying drawings of the specification and specific implementation manners.

[0057] The main solution of the embodiment of this application is: input the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multimodal large language model to obtain the multitask output result for the user instruction, where the multimodal large language model includes a visual token masker, and the multitask output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result, and an attack localization result.

[0058] In this embodiment, for the convenience of description, the face liveness detection device based on a multimodal large language model is used as the execution subject for elaboration below.

[0059] Since traditional face liveness detection methods still face many challenges, including limited generalization ability, poor interpretability, and lack of coarse-grained and fine-grained localization of attack regions.

[0060] The present application provides a solution. The face image to be recognized and the target auxiliary image modality image are input into a multi-modal large language model for fusion, thereby enhancing the model's representation ability. Then, the fused image features are randomly masked by the visual token masker in the multi-modal large language model to reduce interference information and further improve the model's generalization ability. In addition, the present application also extends the traditional face liveness detection task to four subtasks: coarse-grained classification, fine-grained classification, reasoning, and attack localization, which can more comprehensively handle various attack means, thereby improving the security and reliability of face liveness detection.

[0061] As can be seen from the above embodiments, the face liveness detection method, device, equipment, and storage medium based on a multi-modal large language model proposed in the present application input the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multi-modal large language model to obtain a multi-task output result for the user instruction. Among them, the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result, and an attack localization result. This solves the technical problems faced by traditional face liveness detection methods, such as insufficient model generalization ability, poor model interpretability, and lack of coarse and fine-grained localization of attack areas. Compared with the prior art, the present application inputs the face image to be recognized and the target auxiliary image modality image into the multi-modal large language model for fusion, thereby enhancing the model's representation ability. Then, the fused image features are randomly masked by the visual token masker in the multi-modal large language model to reduce interference information and further improve the model's generalization ability. In addition, the present application also extends the traditional face liveness detection task to four subtasks: coarse-grained classification, fine-grained classification, reasoning, and attack localization, which can more comprehensively handle various attack means, thereby improving the security and reliability of face liveness detection.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions, a face liveness detection device based on a multi-modal large language model, etc. Hereinafter, taking face liveness detection as an example, this embodiment and the following embodiments will be described.

[0063] Based on this, the embodiment of the present application provides a face liveness detection method based on a multi-modal large language model, referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the face liveness detection method based on a multi-modal large language model of the present application.

[0064] In this embodiment, the face liveness detection method based on a multi-modal large language model includes step S10:

[0065] Step S10, input the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multi-modal large language model to obtain a multi-task output result for the user instruction. Among them, the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason inference result, and an attack localization result.

[0066] It can be understood that different user instructions output different results, and the multi-task output result is driven by different user instructions. For example, the user instruction corresponding to the coarse-grained task result is: "Is the face in the image real or spoof?"; the user instruction corresponding to the fine-grained task result is: "What attack type of the face in the image?"; the user instruction corresponding to the reason inference result is: "Why the face in the image real or spoof?"; the user instruction corresponding to the attack localization result is: "Locate the spoof area of the face in the image".

[0067] It should be noted that as Figure 2 shown, the coarse-grained classification result mainly answers true / false and briefly explains the reason; the fine-grained classification result mainly answers the specific attack type and briefly explains the reason; the reason inference result mainly answers true / false, conducts detailed analysis and reasoning based on visual clues, and finally gives a conclusion; the attack localization result mainly outputs the bbox coordinates to locate and visualize the attack area.

[0068] In a feasible implementation manner, before the step of inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multi-modal large language model to obtain a multi-task output result for the user instruction, it further includes: obtaining the face image to be recognized; performing grayscale processing on the face image to be recognized to obtain a grayscale image; performing orientation gradient processing on the face image to be recognized to obtain a histogram of oriented gradients; performing local binary pattern processing on the face image to be recognized to obtain a local binary pattern image; and obtaining the target auxiliary modality image based on the grayscale image, the histogram of oriented gradients, and the local binary pattern image.

[0069] It should be noted that the face image to be recognized is a visible light image; the target auxiliary modal image can be determined through the face image to be recognized. Specifically, the face image to be recognized can be respectively subjected to Gray (grayscale processing), HOG (orientation gradient processing), and LBP (local binarization processing), and then the results of these three processes are used as the three channels of the image for fusion to obtain the final target auxiliary modal image.

[0070] It should be noted that the specific calculation method for grayscale processing (Gray) of the face image to be recognized is as follows:

[0071] Gray(I) = 0.299·R + 0.587·G + 0.114·B

[0072] In the formula, Gray represents the grayscale image, R represents the red intensity value of the pixel in the face image to be recognized, G represents the green intensity value of the pixel in the face image to be recognized, and B represents the blue intensity value of the pixel in the face image to be recognized.

[0073] It should be noted that the specific calculation method for local binarization processing (LBP) of the face image to be recognized is as follows:

[0074]

[0075] In the formula, g c represents the pixel value of the central pixel of the face image to be recognized, g i is the pixel value of the surrounding neighborhood pixels of the central pixel, and s(x) is a sign function. When x ≥ 0, s(x) = 1; otherwise, s(x) = 0.

[0076] In this embodiment, by inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into a trained multi-modal large language model, a multi-task output result for the user instruction is obtained. Among them, the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result, and an attack localization result. This solves the technical problems faced by traditional face liveness detection methods, such as insufficient model generalization ability, poor model interpretability, and lack of coarse-grained and fine-grained localization of attack areas. Compared with the prior art, in this application, the face image to be recognized and the target auxiliary image modality image are input into the multi-modal large language model for fusion, thereby enhancing the model's representation ability. Then, the visual token masker in the multi-modal large language model performs random masking on the fused image features to reduce interference information and further improve the model's generalization ability. In addition, this application also expands the traditional face liveness detection task into four subtasks: coarse-grained classification, fine-grained classification, reasoning, and attack localization, which can more comprehensively handle various attack means, thereby improving the security and reliability of face liveness detection.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 , before step S10, the face liveness detection method based on the multi-modal large language model further includes steps S01 to S02:

[0078] Step S01, constructing a face image pre-training set and a face image supervised fine-tuning data set;

[0079] It should be noted that the pictures in the WMCA data set, the SiW-Mv2 data set, and the PADISI data set can be summarized and then re-organized and divided into a joint data set of 12 different categories, and the face image pre-training set and the face image supervised fine-tuning data set can be constructed according to the joint data set.

[0080] In a feasible implementation manner, the step of constructing the face image pre-training set and the face image supervised fine-tuning data set includes: constructing a joint data set based on face images of different categories, and annotating the prosthesis areas of the face images in the joint data set; constructing instruction information for each face image in the joint data set; based on the instruction information, dividing the joint data set into an initial pre-training data set and an initial supervised fine-tuning data set; filtering the initial pre-training data set based on the text-image similarity threshold of each face image in the initial pre-training data set to obtain a face image pre-training set; filtering the initial supervised fine-tuning data set based on the large language model, specific vocabulary, and manual review results to obtain a face image supervised fine-tuning data set.

[0081] In a specific implementation, as Figure 4 shown, the pictures in the WMCA dataset, SiW-Mv2 dataset, and PADISI dataset can be aggregated and then reorganized and divided into a combined dataset with 12 different categories. Then, bounding box annotations are made on the attack areas of the face prosthesis pictures in the combined dataset. Then, an MLLM (Large Language Model) for generating QA data is deployed, and corresponding instructions are designed for the data construction in different stages and different tasks. Finally, data generation is performed according to different instructions. Specifically, first, the picture context information of the large language model is told, including the label (real / spoof) corresponding to the picture for which data needs to be constructed currently and some descriptions (paper printing, screen replay, wearing a mask, etc.), then its construction requirements (content, format) are told, and finally some examples are provided to it.

[0082] In a specific implementation, as Figure 4 shown, the CLIP model can also be used to filter the generated pre-training data by setting a threshold for the similarity between text and image (similarity > 15%) (the image similarity refers to the similarity between the face images in the pre-training data and the label information / description information), to obtain the FaceShield-pre10K dataset (i.e., the face image pre-training set); the generated supervised fine-tuning data can also be filtered by combining the large language model, specific vocabulary, and manual review methods. For example, specific instructions are designed, and the large language model is used to enhance the filtered supervised fine-tuning data to obtain the FaceShield-sft45K dataset (i.e., the face image supervised fine-tuning dataset).

[0083] Step S02: After pre-training the mapper in the multimodal large language model based on the face image pre-training set, the mapper in the multimodal large language model and the target large language model are supervised and fine-tuned based on the face image supervised fine-tuning dataset to obtain a trained multimodal large language model.

[0084] It should be noted that in the pre-training stage of the multimodal large language model, only the mapper in the multimodal large language model is trained (other modules remain frozen), which can ensure the effective alignment of image and text information and provide a better starting point for subsequent more complex multimodal tasks (that is, it can enable the mapper to have the ability to understand images).

[0085] It should be noted that in the supervised fine-tuning stage of the multimodal large language model, based on the weights obtained in the pre-training stage, the Projector (mapper) and the LoRA module (i.e., the low-rank adaptation module in the target large language model) are continuously trained, and other modules remain frozen.

[0086] In a specific implementation, such as Figure 5 shown, in the pre-training stage: the first visible light image and the first auxiliary modality image are input into the frozen image encoder of the multi-modal large language model to extract the first visual feature. Then, the first visual feature V is input into the Projector of the multi-modal large language model to obtain the first mapped feature. Finally, the text token (the first text tokenization) obtained through the Tokenizer (text tokenizer) is aligned with the visual token (the first mapped feature) obtained through the Projector, and is fed into the multi-modal large language model for processing to update the weights of the Projector; in the supervised fine-tuning stage: the second visible light image and the second auxiliary modality image are input into the image encoder of the multi-modal large language model to extract the second visual feature, and the second text prompt is input into the text tokenizer to obtain the text Token (i.e., the second text tokenization). Then, the second visual feature and the second text prompt are input into the Visual Token Masker (PVTM module) for visual Token screening and masking processing to obtain the target visual feature. Next, the mapper maps the target visual feature to the feature space to obtain the second mapped feature. Finally, after the second text tokenization is aligned with the second mapped feature, it is fed into the target large language model for processing, and the final model weights are obtained after training with batch QA data.

[0087] In a feasible implementation manner, the multi-modal large language model further includes a visual encoder and a text tokenizer; wherein, the step of pre-training the mapper in the multi-modal large language model based on the face image pre-training set includes: determining the first visible light image and the first text prompt from the face image pre-training set, and determining the first auxiliary modality image according to the first visible light image; inputting the first visible light image, the first label data, and the first auxiliary modality image into the multi-modal large language model, extracting the first visual feature from the first visible light image and the first auxiliary modality image through the visual encoder, and processing the first text prompt through the text tokenizer to obtain the first text tokenization; mapping the first visual feature to the feature space through the mapper to obtain the first mapped feature; processing the first text tokenization and the first mapped feature through the target large language model to obtain the first cross-entropy loss, and pre-training the mapper according to the first cross-entropy loss.

[0088] It should be noted that the first auxiliary modality image can be determined through the first visible light image. Specifically, the first visible light image can be respectively processed by Gray, HOG, and LBP, and then the results of these three processes are fused as the three channels of the image to obtain the final first auxiliary modality image; the text prompt can be understood as the label information or description information of the visible light image.

[0089] It should be noted that the first visible light image and the first auxiliary modality image can be input into the frozen image encoder of the multimodal large language model to extract the first visual feature V, which is described by the formula as follows:

[0090] V = Contact[V RGB ,V SAV

[0091] In the formula, V RGB represents the first visible light image, and V sav represents the first auxiliary modality image.

[0092] It should be noted that the first visual feature V is input into the Projector of the multimodal large language model to map the first visual feature V to the feature space, which is described by the formula as follows:

[0093] V align = projection(V RGB ,V SAV )

[0094] In the formula, V align represents the first mapped feature.

[0095] It should be noted that the text token (the first text tokenization) obtained through the Tokenizer is aligned with the visual token (the first mapped feature) obtained through the Projector, and then sent into the multimodal large language model for processing. Then, the alignment effect is judged by the first cross-entropy loss, and the weights in the Projector are continuously updated to minimize the first cross-entropy loss function, gradually optimizing the weight parameters of the Projector, so that the alignment effect between the text and the image features in the shared space is getting better and better, thus realizing the pre-training of the mapper in the multimodal large language model.

[0096] ​In a feasible implementation, the multimodal large language model further includes a visual encoder and a text tokenizer; wherein, the step of supervising and fine-tuning the mapper and the target large language model in the multimodal large language model based on the face image supervised fine-tuning dataset includes: determining a second visible light image and a second text prompt from the face image supervised fine-tuning dataset, and determining a second auxiliary modal image according to the second visible light image; inputting the second visible light image, the second label data, and the second auxiliary modal image into the multimodal large language model, extracting second visual features from the second visible light image and the second auxiliary modal image through the visual encoder, and processing the second text prompt through the text tokenizer to obtain second text tokens; masking the second visual features according to the second label data through the visual token masker to obtain target visual features, and mapping the target visual features to the feature space through the mapper to obtain second mapped features; processing the second text tokens and the second mapped features through the target large language model to obtain a second cross-entropy loss, and supervising and fine-tuning the mapper according to the second cross-entropy loss.

[0097] It should be noted that the second auxiliary modal image can be determined through the second visible light image. Specifically, the second visible light image can be respectively processed by Gray, HOG, and LBP, and then the results of these three processes are fused as the three channels of the image to obtain the final second auxiliary modal image.

[0098] In a specific implementation, the second visible light image and the second auxiliary modal image can be input into the image encoder of the multimodal large language model to extract second visual features, and the second text prompt can be input into the text tokenizer to obtain text Tokens (i.e., second text tokens); then the second visual features and the second text prompt are input into the visual token masker (PVTM module) for visual Token screening and masking processing to obtain target visual features; then the mapper maps the target visual features to the feature space to obtain second mapped features; finally, after the second text tokens and the second mapped features are aligned, they are sent into the target large language model for processing, and then the weights of the mapper and the LoRA module (i.e., the low-rank adaptation module in the target large language model) are continuously updated through the second cross-entropy loss to minimize the second cross-entropy loss function, and finally the weight fine-tuning of the mapper and the LoRA module is achieved.

[0099] In a feasible implementation manner, the step of masking the second visual feature according to the second text prompt by the visual token masker to obtain a target visual feature includes: determining, by the visual token masker, the similarity between each visual image block in the second visual feature and the second text prompt; obtaining, by the visual token masker, a sorting result after sorting all the visual image blocks of the second visual feature according to the similarity; screening, by the visual token masker, important visual image blocks and remaining visual image blocks of the second visual feature according to the sorting result; randomly determining, by the visual token masker, visual image blocks to be masked from the remaining visual image blocks, and masking, by the visual token masker, the visual image blocks to be masked in the second visual feature to obtain a target visual feature.

[0100] It should be noted that, as Figure 6 shown, the visual token masker (PVTM module) can mask some less important visual image blocks in the second visual feature according to the second text prompt, so as to reduce the influence of these unimportant visual image blocks on the model output, while maintaining an acceptable information loss in the final decision-making process.

[0101] It can be understood that the important visual image blocks refer to the visual image blocks whose importance ranks among the top K% in the second visual feature; the remaining visual image blocks refer to all the visual image blocks in the second visual image except the important visual image blocks; the visual image blocks to be masked refer to p% of the visual image blocks in the remaining visual image blocks (that is, the visual image blocks randomly masked with a probability of p% in the remaining visual image blocks).

[0102] In specific implementation, the sorting result can be directly determined according to the similarity between each visual image block in the second visual feature and the second text prompt; it can also be determined according to the similarity between each visual image block in the second visual feature and the second text prompt to obtain the importance measurement result of each visual image block, and then the sorting result is determined according to the importance measurement result of each visual image block.

[0103] In specific implementation, in order to further enhance the alignment between the visual feature and the text prompt and alleviate the overfitting problem, the token selection after the visual encoder is guided by the text prompt. Specifically, the text tokens (text segments) extracted from the text prompt are pooled and the similarity is calculated with all the visual tokens (visual image blocks). Assuming that the visual tokens more relevant to the subsequent task have a higher similarity, and even for the same image, the important visual tokens may be inconsistent in different tasks, the similarity calculation formula is as follows:

[0104]

[0105] Wherein, V i represents the i-th visual image patch in the second visual feature, P represents the second text prompt, and Sim represents the similarity.

[0106] Next, importance measurement is performed on the similarity between all visual tokens and the text prompt to determine the importance measurement results of all visual image patches, and then sorting is performed according to the importance measurement results (i.e., the sorting result). The calculation formula for importance measurement is as follows:

[0107]

[0108] Wherein, S(V i , P) represents the similarity between V i and P, j represents the number of visual image patches in the second visual feature, represents the importance measurement result of V i .

[0109] It can be understood that the visual image patches in the second visual feature whose importance measurement results rank among the top K% are determined as important visual image patches.

[0110] In this embodiment, a face image pre-training set and a face image supervised fine-tuning data set are constructed; after pre-training the mapper in the multi-modal large language model based on the face image pre-training set, the mapper and the target large language model in the multi-modal large language model are supervised and fine-tuned based on the face image supervised fine-tuning data set to obtain a trained multi-modal large language model. By pre-training the mapper in the multi-modal large language model, effective alignment of image and text information is ensured, and by supervising and fine-tuning the mapper and the target large language model in the multi-modal large language model, a more accurate fine-grained localization result of the face image attack area is obtained from the model output.

[0111] It should be noted that the above examples are only for understanding this application and do not constitute a limitation to the face liveness detection method based on the multi-modal large language model of this application. Any simple transformation in more forms based on this technical concept is within the protection scope of this application.

[0112] This application also provides a face liveness detection device based on a multi-modal large language model. Please refer to Figure 7 , the face liveness detection device based on the multi-modal large language model includes:

[0113] An input module 10 is configured to input a face image to be recognized, a target auxiliary modality image, and a user instruction into a trained multi-modal large language model to obtain a multi-task output result for the user instruction. The multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result, and an attack localization result.

[0114] The face liveness detection device based on a multi-modal large language model provided in this application adopts the face liveness detection method based on a multi-modal large language model in the above embodiment, and can solve the technical problems that the traditional face liveness detection method based on a multi-modal large language model still faces insufficient model generalization ability, poor model interpretability, and lack of coarse and fine-grained localization of the attack area. Compared with the prior art, the beneficial effects of the face liveness detection device based on a multi-modal large language model provided in this application are the same as those of the face liveness detection method based on a multi-modal large language model provided in the above embodiment, and other technical features in the face liveness detection device based on a multi-modal large language model are the same as the features disclosed in the method of the above embodiment, which will not be elaborated here.

[0115] This application provides a face liveness detection device based on a multi-modal large language model. The face liveness detection device based on a multi-modal large language model includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the face liveness detection method based on a multi-modal large language model in the first embodiment above.

[0116] Reference is made below Figure 8 FIG. shows a schematic structural diagram of a face liveness detection device based on a multi-modal large language model suitable for implementing the embodiments of this application. The face liveness detection device based on a multi-modal large language model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The face liveness detection device shown in FIG. is only an example and should not impose any limitation on the functions and usage scope of the embodiments of this application.

[0117] As Figure 8As shown, the face liveness detection device based on the multimodal large language model may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 into the RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the face liveness detection device based on the multimodal large language model are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the face liveness detection device based on the multimodal large language model to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a face liveness detection device based on the multimodal large language model with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems can be alternatively implemented or had.

[0118] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the method of the embodiments disclosed in the present application are executed.

[0119] The face liveness detection device based on the multimodal large language model provided by this application adopts the face liveness detection method based on the multimodal large language model in the above-mentioned embodiment, and can solve the technical problems that the traditional face liveness detection method based on the multimodal large language model still faces insufficient model generalization ability, poor model interpretability, and lack of fine-grained positioning of attack areas. Compared with the prior art, the beneficial effects of the face liveness detection device based on the multimodal large language model provided by this application are the same as those of the face liveness detection method based on the multimodal large language model provided by the above-mentioned embodiment, and other technical features in the face liveness detection device based on the multimodal large language model are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0121] As described above, this is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0122] This application provides a computer-readable storage medium with computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the face liveness detection method based on the multimodal large language model in the above-mentioned embodiment.

[0123] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0124] The above computer-readable storage medium can be included in a face liveness detection device based on a multimodal large language model; or it can exist independently without being assembled into a face liveness detection device based on a multimodal large language model.

[0125] The above computer-readable storage medium carries one or more programs. When the one or more programs are executed by a face liveness detection device based on a multimodal large language model, the face liveness detection device based on the multimodal large language model is caused to: input the face image to be recognized, the target auxiliary modal image, and the user instruction into the trained multimodal large language model to obtain a multi-task output result for the user instruction, where the multimodal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason inference result, and an attack localization result.

[0126] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0129] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned face liveness detection method based on a multimodal large language model, which can solve the technical problems that the traditional face liveness detection method based on a multimodal large language model still faces insufficient model generalization ability, poor model interpretability, and lack of fine-grained positioning of attack regions. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the face liveness detection method based on a multimodal large language model provided in the above embodiments, and will not be elaborated here.

[0130] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the above-mentioned face liveness detection method based on a multimodal large language model.

[0131] The computer program product provided by this application can solve the technical problems that the traditional face liveness detection method based on a multimodal large language model still faces insufficient model generalization ability, poor model interpretability, and lack of fine-grained positioning of attack regions. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the face liveness detection method based on a multimodal large language model provided in the above embodiments, and will not be elaborated here.

[0132] The above are only partial embodiments of this application, and thus do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. A face liveness detection method based on a multimodal large language model, characterized in that, The method includes: Inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into a trained multi-modal large language model to obtain a multi-task output result for the user instruction. Among them, the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason reasoning result, and an attack localization result.

2. The method according to claim 1, wherein, The multi-modal large language model further includes a mapper and a target large language model. Before the step of inputting the face image to be recognized, the target auxiliary modality image, and the user instruction into the trained multi-modal large language model to obtain a multi-task output result for the user instruction, it further includes: Constructing a face image pre-training set and a face image supervised fine-tuning data set; After pre-training the mapper in the multi-modal large language model based on the face image pre-training set, performing supervised fine-tuning on the mapper and the target large language model in the multi-modal large language model based on the face image supervised fine-tuning data set to obtain a trained multi-modal large language model.

3. The method according to claim 2, wherein The step of constructing the face image pre-training set and the face image supervised fine-tuning data set includes: Constructing a joint data set based on face images of different categories and annotating the prosthesis regions of the face images in the joint data set; Constructing instruction information for each face image in the joint data set; Based on the instruction information, dividing the joint data set into an initial pre-training data set and an initial supervised fine-tuning data set; Filtering the initial pre-training data set based on the text-image similarity threshold of each face image in the initial pre-training data set to obtain a face image pre-training set; Filtering the initial supervised fine-tuning data set based on the large language model, specific vocabulary, and manual review results to obtain a face image supervised fine-tuning data set.

4. The method according to claim 2, wherein The multi-modal large language model further includes a visual encoder and a text tokenizer. Among them, the step of pre-training the mapper in the multi-modal large language model based on the face image pre-training set includes: Determining a first visible light image and a first text prompt from the face image pre-training set, and determining a first auxiliary modality image according to the first visible light image; Inputting the first visible light image, the first label data, and the first auxiliary modality image into the multi-modal large language model, extracting first visual features from the first visible light image and the first auxiliary modality image through the visual encoder, and processing the first text prompt through the text tokenizer to obtain first text tokens; Mapping the first visual features to a feature space through the mapper to obtain first mapped features; Processing the first text tokens and the first mapped features through the target large language model to obtain a first cross-entropy loss, and pre-training the mapper according to the first cross-entropy loss.

5. The method according to claim 2, wherein The multimodal large language model further includes a visual encoder and a text tokenizer; wherein, the step of supervising and fine-tuning the mapper and the target large language model in the multimodal large language model based on the face image supervised fine-tuning dataset includes: Determine a second visible light image and a second text prompt from the face image supervised fine-tuning dataset, and determine a second auxiliary modal image according to the second visible light image; Input the second visible light image, the second label data, and the second auxiliary modal image into the multimodal large language model, extract second visual features from the second visible light image and the second auxiliary modal image through the visual encoder, and process the second text prompt through the text tokenizer to obtain second text tokens; Perform masking processing on the second visual features according to the second label data through the visual token masker to obtain target visual features, and map the target visual features to the feature space through the mapper to obtain second mapped features; Process the second text tokens and the second mapped features through the target large language model to obtain a second cross-entropy loss, and perform supervised fine-tuning on the mapper according to the second cross-entropy loss.

6. The method according to claim 5, wherein The step of performing masking processing on the second visual features according to the second text prompt through the visual token masker to obtain target visual features includes: Determine the similarity between each visual image patch in the second visual features and the second text prompt through the visual token masker; After sorting all the visual image patches of the second visual features according to the similarity through the visual token masker, obtain a sorting result; Screen out important visual image patches and remaining visual image patches of the second visual features according to the sorting result through the visual token masker; Randomly determine the visual image patches to be masked from the remaining visual image patches through the visual token masker, and perform masking processing on the visual image patches to be masked in the second visual features through the visual token masker to obtain target visual features.

7. The method according to claim 1, wherein Before the step of inputting the face image to be recognized, the target auxiliary modal image, and the user instruction into the trained multimodal large language model to obtain a multitask output result for the user instruction, it further includes: Obtain a face image to be recognized; Perform grayscale processing on the face image to be recognized to obtain a grayscale image; Perform orientation gradient processing on the face image to be recognized to obtain an orientation gradient histogram; Perform local binarization processing on the face image to be recognized to obtain a local binary pattern map; Based on the grayscale image, the orientation gradient histogram, and the local binary pattern map, obtain a target auxiliary modal image.

8. A face liveness detection device based on a multimodal large language model, characterized in that, The device includes: An input module for inputting a face image to be recognized, a target auxiliary modality image, and a user instruction into a trained multi-modal large language model to obtain a multi-task output result for the user instruction, wherein the multi-modal large language model includes a visual token masker, and the multi-task output result includes at least one of a coarse-grained classification result, a fine-grained classification result, a reason inference result, and an attack localization result.

9. A face liveness detection device based on a multimodal large language model, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the face liveness detection method based on a multi-modal large language model according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the face liveness detection method based on a multi-modal large language model according to any one of claims 1 to 7 are implemented.

Images