Model training method, living body detection method and related equipment

Abstract

The invention discloses a model training method, a living body detection method and related equipment. The method comprises the following steps: acquiring a to-be-trained living body detection sample image; according to a preset text description prompt word, performing text description on the living body detection sample image to obtain a first description text; performing feature extraction on the first description text through the text branch to obtain a first text feature; performing feature extraction on the living body detection sample image through the image branches to obtain image features; performing feature extraction on a preset in-vivo detection attack type text through the text branch to obtain a second text feature; obtaining a first loss and a second loss through the first text feature, the second text feature and the image feature; and updating living body identification parameters of the branch according to the first loss and the second loss. According to the method, the model can be trained in combination with the text features, the supervision ability of the language text can be better combined, the model can better process related data in combination with the text, and then the generalization ability of the model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training method, a liveness detection method, and related equipment. Background Technology

[0002] Face liveness detection is a crucial step in the face recognition process, directly related to the security of user authentication. With the widespread deployment of face recognition systems in daily life and industrial applications, various new liveness attack methods are constantly emerging.

[0003] These newly emerging attack data may differ significantly from the attack data in the training set in many aspects, such as facial features, lighting conditions, background environment, and attack methods, resulting in inconsistent data distribution. Therefore, directly applying the model trained on the training set to the test data may lead to a decrease in the model's generalization ability.

[0004] Currently, improving model generalization ability often involves integrating vision-language models into liveness detection, leveraging their pre-training performance to enhance generalization. However, this approach, which only uses coarse-grained or single-element cues to fine-tune the liveness task, results in unsatisfactory generalization capabilities. Summary of the Invention

[0005] This application provides a model training method, a liveness detection method, and related equipment. The related equipment may include a model training device, electronic equipment, a computer-readable storage medium, and a computer program product. By combining text content, the ability of language supervision can be fully explored, thereby improving the generalization ability of the model.

[0006] This application provides a model training method, including:

[0007] Obtain the liveness detection sample images to be trained;

[0008] Based on the preset text description prompts for non-liveness attack types, the liveness detection sample image is described using an image description model to obtain the first description text;

[0009] The first text features are obtained by extracting features from the first descriptive text through the text branch of the cross-modal detection model;

[0010] Feature extraction is performed on the liveness detection sample images using the image branch of the cross-modal detection model to obtain image features;

[0011] The text branch is used to extract features from the preset liveness detection attack type text to obtain the second text feature. The liveness detection attack type text is generated according to the attack type of liveness attack.

[0012] A first loss is obtained based on the first text features and the image features, and a second loss is obtained based on the second text features and the image features;

[0013] Based on the first loss and the second loss, update the liveness detection parameters of the text branch and the image branch until the first loss and the second loss are re-acquired and satisfy the preset loss value.

[0014] Optionally, in some embodiments, the text description prompts include environmental prompts and face prompts. The step of performing text description on the liveness detection sample image using an image description model based on preset non-liveness attack type text description prompts to obtain the first description text includes:

[0015] Based on the environmental cue words, the liveness detection sample image is described using an image description model to obtain the environmental description text.

[0016] Based on the facial prompts, the liveness detection sample image is described using an image description model to obtain facial description text;

[0017] Based on the environmental description text and the face description text, a first description text is obtained.

[0018] Optionally, in some embodiments, obtaining the first loss based on the first text features and the image features includes:

[0019] Obtain the text feature vector of the first text feature and the image feature vector of the image feature;

[0020] The first loss is obtained by performing a dot product on the text feature vector and the image feature vector.

[0021] Optionally, in some embodiments, the step of extracting features from the preset liveness detection attack type text through the text branch to obtain second text features includes:

[0022] Based on preset text expansion prompts, the text expansion model is used to expand the text of liveness detection attack types, resulting in multiple attack type expanded texts;

[0023] By extracting features from multiple attack type extended texts through the text branches, a second text feature is obtained.

[0024] Optionally, in some embodiments, the step of extracting features from multiple attack type extended texts through the text branches to obtain second text features includes:

[0025] By extracting features from multiple attack type extended texts through the text branches, a third text feature for each attack type extended text is obtained;

[0026] The feature similarity is calculated by combining the third text feature and the image feature.

[0027] Multiple attack type extended texts that match the feature similarity with a preset similarity are obtained and used as target extended texts;

[0028] The second text features are obtained by extracting features from multiple target extended texts through the text branches.

[0029] Optionally, in some embodiments, the step of extracting features from multiple target extended texts through the text branches to obtain second text features includes:

[0030] Obtain the target weight coefficient for each of the target extended texts;

[0031] Based on the target weight coefficient, feature extraction is performed on multiple target extended texts through the text branch to obtain the second text features.

[0032] Optionally, in some embodiments, obtaining the target weight coefficient for each of the target expanded texts includes:

[0033] Obtain the initial weight coefficient for each of the target expanded texts;

[0034] Based on the initial weight coefficients, feature extraction is performed on multiple target extended texts through the text branches to obtain the fourth text feature;

[0035] The fourth loss is calculated by combining the fourth text features and the image features;

[0036] Based on the fourth loss and the preset loss, the initial weight coefficients are updated to obtain the target weight coefficients for each target extended text.

[0037] Optionally, in some embodiments, updating the initial weight coefficients based on the fourth loss and the preset loss includes:

[0038] When the fourth loss does not match the preset loss, the initial weight coefficient is adjusted until the fourth loss matches the preset loss, at which point the update of the initial weight coefficient stops.

[0039] Accordingly, embodiments of this application also provide a liveness detection method, including:

[0040] Acquire the image to be detected;

[0041] The image features of the image to be detected are obtained by extracting features from the image branch of the cross-modal detection model. The cross-modal detection model is trained using the model training method described in any one of claims 1 to 8.

[0042] Obtain different attack texts that attack the image to be detected;

[0043] By extracting features from different attack texts through the text branch of the cross-modal detection model, the text features corresponding to different attack texts are obtained.

[0044] Obtain the similarity between the image features and each of the text features;

[0045] Based on the similarity, the liveness detection result of the image to be detected is obtained.

[0046] Accordingly, embodiments of this application provide a model training apparatus, including:

[0047] The image acquisition module is used to acquire images of liveness detection samples to be trained.

[0048] The first text acquisition module is used to perform text description on the liveness detection sample image based on preset text description prompts for non-liveness attack types using an image description model, and obtain the first description text.

[0049] The first feature acquisition module is used to extract features from the first descriptive text through the text branch of the cross-modal detection model to obtain the first text features;

[0050] The image feature acquisition module is used to extract features from the liveness detection sample image through the image branch of the cross-modal detection model to obtain image features;

[0051] The second feature acquisition module is used to extract features from the preset liveness detection attack type text through the text branch to obtain the second text feature, wherein the liveness detection attack type text is generated according to the attack type of liveness attack.

[0052] The loss acquisition module is used to obtain a first loss based on the first text features and the image features, and to obtain a second loss based on the second text features and the image features;

[0053] The parameter update module is used to update the liveness detection parameters of the text branch and the image branch according to the first loss and the second loss until the re-obtained first loss and second loss meet the preset loss values.

[0054] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the model training method provided in this application.

[0055] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the model training method provided in this application.

[0056] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the model training method provided in embodiments of this application.

[0057] This application provides a model training method, a liveness detection method, and related equipment, which can acquire liveness detection sample images to be trained; based on preset non-liveness attack type text description prompts, perform text description on the liveness detection sample images using an image description model to obtain a first description text; extract features from the first description text using the text branch of a cross-modal detection model to obtain a first text feature; extract features from the liveness detection sample images using the image branch of the cross-modal detection model to obtain image features; extract features from the preset liveness detection attack type text using the text branch to obtain a second text feature; calculate a first loss using the first text feature and the image feature; calculate a second loss using the second text feature and the image feature; and update the liveness recognition parameters of the text branch and the image branch based on the first loss and the second loss. This application extracts first text features from the first descriptive text of non-liveness attack types through the text branch of a cross-modal detection model, and extracts image features from the liveness detection sample image through the image branch. Then, a first loss is calculated using the first text features and the image features. Next, a second text feature is extracted from the text of the liveness detection attack type through the text branch, and a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Thus, in addition to training the model using image features, it can also train the model using the first text features that are unrelated to liveness detection. This allows the model to better decouple non-liveness attack content in the image. The smaller the first loss value, the stronger the decoupling ability, and the more accurately the model can exclude non-liveness attack factors in the image. Since the model can learn discriminative features unrelated to liveness detection, it can enhance the model's cross-domain capability, and thus better combine the supervision capability of language text, enabling the model to better process relevant data in conjunction with text and improve the model's generalization ability. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of a scenario for the model training method provided in an embodiment of this application;

[0060] Figure 2 This is a flowchart of the model training method provided in the embodiments of this application;

[0061] Figure 3 This is another flowchart of the model training method provided in the embodiments of this application;

[0062] Figure 4 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0063] Figure 5 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0064] Figure 6 This is a flowchart of the liveness detection method provided in the embodiments of this application;

[0065] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] This application provides a model training method and related equipment. The related equipment may include a model training device, an electronic device, a computer-readable storage medium, and a computer program product. Specifically, the model training device may be integrated into an electronic device, which may be a terminal or a server, etc.

[0068] The image recognition method provided in this application relates to computer vision (CV) technology in the field of artificial intelligence (AI). The application of the cross-modal detection model involves adaptive computation. This application extracts features from a first descriptive text unrelated to liveness detection using the text branch of the cross-modal detection model to obtain first text features, and extracts features from a liveness detection sample image using the image branch to obtain image features. Then, a first loss is calculated using the first text features and the image features. Next, features are extracted from liveness detection attack type text using the text branch to obtain second text features. Then, a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Therefore, in addition to training the model using image features, it can also train the model using text features, better combining the supervision capabilities of language text, enabling the model to better process relevant data in conjunction with text, thereby improving the model's generalization ability.

[0069] Adaptive computing refers to automatically adjusting the computational load and accuracy of a model based on different input data, aiming to improve computational efficiency while maintaining model accuracy. Adaptive computing can flexibly adjust the computational load and accuracy of a model on different input data, thereby better balancing computational efficiency and accuracy.

[0070] It is understood that the model training method in this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0071] like Figure 1 As shown, the example illustrates a model training method where a terminal and a server jointly perform the training. The model training system provided in this embodiment includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, etc., wherein the model training device can be integrated into the terminal.

[0072] Terminal 10 can be used to: acquire liveness detection sample images to be trained; describe the liveness detection sample images using an image description model based on preset non-liveness attack type text description prompts to obtain a first description text; extract features from the first description text using the text branch of a cross-modal detection model to obtain a first text feature; extract features from the liveness detection sample images using the image branch of a cross-modal detection model to obtain image features; extract features from the preset liveness detection attack type text using the text branch to obtain a second text feature; calculate the first text feature and image feature to obtain a first loss; calculate the second text feature and image feature to obtain a second loss; and update the liveness recognition parameters of the text branch and image branch based on the first loss and the second loss.

[0073] Terminal 10 may include mobile phones, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, tablet computers, laptops, or personal computers (PCs), etc. A client may also be configured on terminal 10, which may be an application client or a browser client, etc.

[0074] The server 11 can be used to: receive liveness detection sample images sent by the terminal 10 to be trained; perform text description on the liveness detection sample images using an image description model based on preset non-liveness attack type text description prompts to obtain a first description text; extract features from the first description text using the text branch of a cross-modal detection model to obtain a first text feature; extract features from the liveness detection sample images using the image branch of a cross-modal detection model to obtain image features; extract features from the preset liveness detection attack type text using the text branch to obtain a second text feature; calculate the first text feature and image feature to obtain a first loss; calculate the second text feature and image feature to obtain a second loss; and update the liveness recognition parameters of the text branch and image branch based on the first loss and the second loss.

[0075] The server 11 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In the model training method or apparatus disclosed in this application, multiple servers can form a blockchain, and the server is a node on the blockchain.

[0076] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0077] This embodiment will be described from the perspective of a model training device, which can be integrated into an electronic device, such as a server or a terminal.

[0078] This embodiment can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0079] like Figure 2 As shown, the specific process of this model training method can be as follows:

[0080] 110. Obtain the liveness detection sample images to be trained.

[0081] Before training the model, the first step is to acquire liveness detection sample images for training. Since the model training aims to improve liveness detection, these sample images are all facial images. This can be done using a camera; the captured liveness detection sample images can be real facial images or facial images obtained by capturing images containing facial features. Of course, uploaded images can also be used, but this embodiment does not limit this.

[0082] In some embodiments, after obtaining a facial image, it can be preprocessed to improve image recognition. For example, it can use MTCNN (Multi-task Cascaded Convolutional Networks) to detect faces in the image. MTCNN is a face detection algorithm that can accurately identify faces in images and detect key facial features (such as the positions of the eyes, nose, and mouth). After detecting a face, MTCNN aligns the face, meaning it adjusts the angle and position of the face to ensure it is facing forward and centered in the image.

[0083] After identifying and defining the face region, the system will expand the image by a certain factor, which can be determined according to specific needs, such as 0.9 times. This operation allows the image to contain more background content, thus facilitating the acquisition of more environmental information in subsequent processes.

[0084] Finally, the image is cropped. The system crops the image based on the enlarged area, removing unwanted parts and retaining only the portion containing the face and its surrounding background. This cropping ensures the image includes both the user's facial details and a suitable background environment.

[0085] 120. Based on the preset text description prompts for non-liveness attack types, the image description model is used to perform text descriptions on the liveness detection sample images to obtain the first description text.

[0086] To enable cross-modal detection models to better identify information unrelated to liveness detection, improve their cross-domain detection capabilities, and enhance their generalization ability, a first description text can be generated for liveness detection sample images using a pre-defined text description prompt for non-liveness attack types via an image description model. Here, non-liveness attack types refer to types other than liveness attacks such as physical attacks, printing attacks, and 3D attacks, including information in the image such as the environment (e.g., lighting) and facial attributes (e.g., irrelevant information, facial expressions, etc.).

[0087] In other words, the image description model generates a text description of the liveness detection sample image based on the text description prompts and according to the logic of the text description prompts. This results in the first description text, whose content and format conform to the requirements of the text description prompts. The image description model can be a multimodal large model, LLAVA (Large Language and Vision Assistant).

[0088] LLAVA's core is a combination of large-scale pre-trained language models (such as the GPT series) and visual models (such as CLIP, ViT, etc.). This model, trained on a large amount of image-text pairing data, enables it to understand visual information in images and associate it with text.

[0089] LLAVA combines visual and language models, enabling it to comprehensively understand mixed image and text input. Its multimodal fusion layer combines image and language features to generate richer expressions.

[0090] Specifically, further, in some embodiments, the text description prompts include environmental prompts and face prompts. Based on preset non-liveness attack type text description prompts, a text description is performed on the liveness detection sample image using an image description model to obtain a first description text, including:

[0091] Based on environmental cue words, text descriptions are generated for liveness detection sample images using an image description model to obtain environmental description text.

[0092] Based on the facial prompts, the liveness detection sample images are described with text using an image description model to obtain facial description text;

[0093] The first description text is obtained based on the environmental description text and the face description text.

[0094] Since the environmental information and facial semantics of an image are irrelevant to the liveness detection task, an image description model can be used to generate textual descriptions related to the environmental information and facial semantics. The resulting descriptive text is the first descriptive text. Environmental information can include background and lighting, while facial information includes facial features, gender, and expression. Of course, more information can be included in the environmental and facial information during this process, depending on the specific requirements.

[0095] Specifically, in this process, the text description prompts include environment prompts and face prompts. For example, the prompts can be set as "Please describe the environment (background, lighting, intensity, etc.) and the face (age, gender, expression, etc.) of the person." Here, "background, lighting, intensity, etc." are environment prompts, and "age, gender, expression, etc." are face prompts.

[0096] When creating a text description for a liveness detection sample image, the image description model can first generate a text description based on environmental cues, resulting in an environmental description text. For example, it could be described as: "The image depicts a tranquil outdoor scene with a clear blue sky and soft sunlight illuminating the entire picture. The background is a lush park, with trees and shrubs creating a natural and peaceful atmosphere. The light is gentle, with the warm evening light casting soft shadows."

[0097] Then, based on the facial cue words, the image description model can be used to generate text descriptions of the liveness detection sample images, resulting in facial description text. For example, it could be described as: "There is a man in his thirties in the foreground. He has short brown hair and no beard. His expression is calm and content, with a slight smile, seemingly in a pleasant and relaxed mood. His eyes are slightly narrowed, perhaps due to the sunlight. The man is wearing a light blue casual shirt, and his posture is also relaxed, appearing to be standing or leaning slightly forward."

[0098] Of course, obtaining the environment description text and the face description text can be done simultaneously. Finally, the obtained environment description text and face description text are integrated to obtain the first description text.

[0099] The first descriptive text obtained after describing the above-mentioned liveness detection sample image is: The image depicts a tranquil outdoor scene with a clear blue sky and soft sunlight illuminating the entire picture. The background is a lush park, with trees and shrubs creating a natural and peaceful atmosphere. The light is gentle, and the warm evening light casts soft shadows.

[0100] In the foreground is a man in his thirties. He has short brown hair and no beard. His expression is calm and content, with a slight smile, suggesting he is in a pleasant and relaxed mood. His eyes are slightly narrowed, perhaps due to the sunlight. The man is wearing a light blue casual shirt and his posture is also relaxed, seemingly standing or leaning slightly forward.

[0101] The text description prompts can be obtained by inputting prompts through a third-party device. In some embodiments, prompts can also be generated through model learning. The difference between the two is that prompts input by a third party are in language that humans can understand, such as the aforementioned statements with logical meaning, or they can be incomprehensible logical code generated through training.

[0102] The prompts generated through model learning can be continuously optimized during training, making the generated text more compliant with requirements. In this process, no prompts need to be input. Once the image is obtained, the required text can be automatically generated by the model. In other words, the prompts are embedded in the model. As training continues, the prompts are iteratively optimized until the generated text gets closer to and meets the preset requirements.

[0103] 130. Extract features from the first descriptive text using the text branch of the cross-modal detection model to obtain the first text features.

[0104] After obtaining the first descriptive text, it is input into the text branch of the cross-modal detection model (such as a pre-trained Transformer model). The model encodes each word or subword into an embedding vector, which is text encoding. After all word vectors are processed, the model generates a global semantic representation, which is a fixed-dimensional feature vector. The generated feature vector will capture the key semantic information in the descriptive text, such as "tranquil outdoor scene", "middle-aged man", "soft sunlight", etc. The obtained feature vectors are the first text features.

[0105] 140. The image features are obtained by extracting features from the liveness detection sample images through the image branch of the cross-modal detection model.

[0106] After the liveness detection sample image is input into the cross-modal detection model, the image branch of the cross-modal detection model (such as the ResNet model, VGG model, Vision Transformer (ViT) model, CLIP model, and other visual models) will encode the liveness detection sample image to obtain a high-dimensional feature vector, which represents the global or local features of the image.

[0107] 150. Extract features from the preset liveness detection attack type text through text branches to obtain the second text features.

[0108] In liveness detection, attack types include printing attacks (using printed photos or photos from mobile phone screens to impersonate a live person), replay attacks (using pre-recorded videos to deceive the liveness detection system), and 3D head model attacks (using 3D printed masks or silicone masks to mimic the facial features of the target object). The liveness detection attack type text is generated based on the attack type.

[0109] The preset liveness detection attack type text is the text corresponding to the attack type mentioned above, such as "A photo of a print face," which corresponds to a printing attack. Feature extraction is performed on the liveness detection attack type text through a text branch to obtain the second text feature. The method for obtaining the second text feature is the same as that for obtaining the first text feature: acquiring a global semantic representation. The global semantic representation is a fixed-dimensional feature vector, and the generated feature vector will capture the key semantic information describing the text.

[0110] Furthermore, in some embodiments, feature extraction is performed on the text of a preset liveness detection attack type through text branching to obtain second text features, including:

[0111] Based on preset text expansion prompts, the text expansion model is used to expand the text of liveness detection attack types, resulting in multiple attack type expanded texts;

[0112] By extracting features from extended texts of multiple attack types through text branches, a second text feature is obtained.

[0113] Because there are many types of attacks in liveness detection, and the attack levels are different, for example, printing attacks focus on high-level visual semantics, while playback attacks involve low-level texture features, such as moiré patterns, processing all attacks in a single attack type will result in insufficient detail and reduced accuracy of liveness detection.

[0114] Therefore, during liveness detection, when extracting features from liveness detection attack-type text, it is necessary to augment the text by generating fine-grained category-specific hints. This fine-grained category augmentation further subdivides attack categories, providing the model with more forgery information (attack information). This allows the model to learn the subtle differences between different attack types, thereby enhancing its ability to distinguish them.

[0115] In the process of generating fine-grained category prompts, the text for liveness detection attack types is first expanded using a text expansion model based on preset text expansion prompts, resulting in multiple expanded texts for different attack types. For example, the original liveness detection attack type text is: "A photo of a print face." The preset text expansion prompt can be set as: Paraphrase the sentence: "The photo of a {class} face.} with similarsemantics." Here, "class" can be modified according to the prompt. When the original liveness detection attack type text is: "A photo of a print face," then "class" is modified to "print."

[0116] The text expansion model can be set according to the actual situation. For example, it can be an OpenAI language model such as GPT-3 / GPT-4, a BERT (Bidirectional Encoder Representations from Transformers) model, or a Paraphrase Generation Models (Pegasus, ParaNMT-50M) model. The specific model can be set according to actual needs, and this embodiment does not limit it.

[0117] For example, expanding the text of "A photo of a print face" could result in multiple hints that maintain a similar meaning to the original statement but are expressed differently, such as "The image of a face from a particular category," "The photograph of a face within a specific group," or "A picture of a face that belongs to a certain classification." This results in multiple attack-type expanded texts.

[0118] Then, feature extraction is performed on multiple attack type extended texts through text branches to obtain the second text features. The process of obtaining the second text features is the same as the process of obtaining the first text features. It can be that the second text features are composed of the text features extracted from each attack type extended text, or it can be obtained by merging the text features extracted from all attack type extended texts.

[0119] Of course, furthermore, in some embodiments, during the text expansion process,

[0120] Due to the inherent randomness in the prompt generation process, it is inevitable that the generated prompts will not meet the detection requirements. Therefore, in order to filter out prompts that do not meet the requirements, in some embodiments, feature extraction is performed on the extended text of multiple attack types through text branching to obtain second text features, including:

[0121] By extracting features from extended texts of multiple attack types through text branches, the third text features of each extended text of attack type are obtained;

[0122] The feature similarity is calculated by analyzing the third text features and image features.

[0123] Multiple attack type extended texts that match the feature similarity with the preset similarity are obtained and used as target extended texts;

[0124] By extracting features from multiple target extended texts through text branches, the second text features are obtained.

[0125] Specifically, in the process of obtaining the second text feature, the first step is to extract features from multiple attack type extended texts through text branches to obtain the third text feature of each attack type extended text. That is, feature extraction is performed on each attack type extended text obtained after expansion. Of course, this extraction process can be performed on all attack type extended texts at the same time, or it can be performed in batches and stages to obtain the third text feature of each attack type extended text. The extraction method of the third text feature is the same as the extraction method of the first text feature, and will not be repeated here.

[0126] Then, the third text features are compared with the image features to obtain the feature similarity. Specifically, the feature similarity of each third text feature is calculated with the image features of all liveness detection sample images.

[0127] For example, suppose there are 64 third text features, numbered T1, T2, T3, ..., T64, and 64 liveness detection sample images, with their image features named V1, V2, V3, ..., V64. The feature similarity between the first third text feature and the image features is calculated by taking the feature similarities between T1 and V1, T1 and V2, ..., and T1 and V64 respectively, and then averaging their values ​​(or using the median, mode, etc., depending on the specific requirements). This average value represents the feature similarity between the first third text feature and the image features. Alternatively, the feature similarity can be calculated by taking the cosine similarity between the third text feature and the image features, and using the magnitude of the cosine similarity as the feature similarity.

[0128] The feature similarity between the second and third text features and the image features is obtained by calculating the feature similarity between T2 and V1, T2 and V2, ..., T2 and V64 respectively, and then taking their average value. This gives the feature similarity between the second and third text features and the image features. The same principle applies to the rest, and the feature similarity between all the third text features and the image features can be obtained.

[0129] Then, all feature similarities are matched against a preset similarity value. If the feature similarity is less than the preset similarity value, it indicates a mismatch; otherwise, it indicates a match. At this point, the third text features corresponding to feature similarities less than the preset similarity value are filtered out. The attack type extended text corresponding to the remaining third text features is the final remaining text that meets the requirements, i.e., the target extended text. In other words, multiple attack type extended texts that match the preset similarity are obtained and used as the target extended text.

[0130] The screening process can be performed using the following formula: Where p represents the filtered text feature set. This indicates a hint belonging to the j-th category of class. V represents the j-th text feature belonging to the class. i This represents the i-th image feature belonging to the class. Representing text feature vectors and image feature vector V i The similarity between them is defined by N, which represents the total number of sample images belonging to this category in the training dataset, and θ, which represents the similarity threshold, i.e., the preset similarity.

[0131] The remaining attack type extended texts after filtering, also known as target extended texts, can be better adapted to liveness detection sample images, resulting in higher accuracy during liveness detection. Therefore, feature extraction is performed on multiple target extended texts through text branches to obtain second text features.

[0132] Furthermore, in some embodiments, to aggregate prompts of the same type to obtain the final features and to make the final feature values ​​more accurate, different weight values ​​can be set for different target extended texts. Therefore, feature extraction is performed on multiple target extended texts through text branches to obtain second text features, including:

[0133] Obtain the target weight coefficient for each target expanded text;

[0134] Based on the target weight coefficient, feature extraction is performed on multiple target extended texts through text branches to obtain the second text features.

[0135] In this process, different weight coefficients can be set for each attack type extended text in advance. The allocation ratio of the weight coefficients can be set according to the actual situation. After the attack type extended text is filtered, the target extended text can continue to use the corresponding weight coefficient, which is the target weight coefficient.

[0136] Then, based on the target weight coefficients, features are extracted from multiple target extended texts through text branches to obtain the second text features. For example, suppose there are three target extended texts: Text 1: "I love programming", weight W1 = 2; Text 2: "Programming is fun", weight W2 = 1; Text 3: "I love learning programming", weight W3 = 3. The word vectors for the three texts can be obtained through text branches, assuming they are as follows:

[0137] Text 1:

[0138] "Me": T 我 =[0.2,-0.1,0.3]

[0139] "Love": T 爱 =[-0.2, 0.3, -0.1]

[0140] "Programming": T 编程 =[0.1,0.1,0.2]

[0141] Document vector: T d1 =13([0.2,-0.1,0.3]+[-0.2,0.3,-0.1]+[0.1,0.1,0.2])=[0.033,0.100,0.133]

[0142] Text 2:

[0143] "Programming": T 编程 =[0.1,0.1,0.2]

[0144] "Yes": T 是 =[-0.1,0.2,0.1]

[0145] "Interesting": T 有趣 =[0.2,-0.1,0.3]

[0146] Document vector: T d2 =13([0.1,0.1,0.2]+[-0.1,0.2,0.1]+[0.2,-0.1,0.3])=[0.067,0.033,0.200]

[0147] Text 3:

[0148] "Me": T 我 =[0.2,-0.1,0.3]

[0149] "Love": T 爱 =[-0.2, 0.3, -0.1]

[0150] “Learning”: T 学习 =[0.1,0.2,-0.2]

[0151] "Programming": T 编程 =[0.1,0.1,0.2]

[0152] Document vector: T d3 =14([0.2,-0.1,0.3]+[-0.2,0.3,-0.1]+[0.1,0.2,-0.2]+[0.1,0.1,0.2])=[0.050,0.125,0.050]

[0153] The final feature vector, which is the second text feature, is:

[0154] T avg =(2T) d1 +1T d2 +3T d3 ) / (2+1+3)=[0.283,0.608,0.616] / 6=[0.047,0.101,0.103].

[0155] Furthermore, in some embodiments, directly using the respective weight coefficients during the process of obtaining the second text features is not the optimal choice. This is because unsuitable text is filtered out during this process, resulting in the final aggregated feature vector not matching the image feature vector better. Therefore, it is necessary to adjust the weight coefficients of the target expanded text. Thus, obtaining the target weight coefficients for each target expanded text includes:

[0156] Obtain the initial weight coefficients for each target expanded text;

[0157] Based on the initial weight coefficients, feature extraction is performed on multiple target extended texts through text branches to obtain the fourth text feature;

[0158] The fourth loss is calculated by analyzing the fourth text features and image features;

[0159] Based on the fourth loss and the preset loss, the initial weight coefficients are updated to obtain the target weight coefficients for each target expanded text.

[0160] The initial weight coefficient of each target extended text is the same as the pre-set weight coefficient of the corresponding attack type extended text, and the weight coefficient is not adjusted at this time. Then, based on the initial weight coefficient, feature extraction is performed on multiple target extended texts through text branches to obtain the fourth text feature. The method of obtaining the fourth text feature is the same as the method of obtaining the second text feature by performing feature extraction on multiple target extended texts through text branches based on the target weight coefficient.

[0161] The fourth text feature can be obtained by calculating it using the following formula:

[0162]

[0163] in, The final feature result is represented, which in this embodiment corresponds to the fourth text feature. N represents the number of category hints obtained after filtering, that is, the number of target expanded texts. Wj represents the learnable weights, and σ(w) j This is the result of performing a Softmax operation on Wj.

[0164] Then, the fourth loss is obtained through the fourth text features and image features. The method of obtaining the fourth loss is the same as that of obtaining the first loss. In this embodiment, the method of merging all fourth text features is used as an example. Then, based on the fourth loss and the preset loss, the initial weight coefficients are updated to obtain the target weight coefficients of each target extended text.

[0165] The update process can be to modify the initial weight coefficients of one or more target extended texts in a fixed way, such as increasing or decreasing the initial weight coefficients of fixed target extended texts, so as to update the initial weight coefficients, make the fourth loss match the preset loss, and then obtain the target weight coefficients of the final target extended texts.

[0166] To obtain better weight coefficients, in some embodiments, the initial weight coefficients are further updated based on the fourth loss and the preset loss, including:

[0167] When the fourth loss does not match the preset loss, adjust the initial weight coefficient until the fourth loss matches the preset loss, then stop updating the initial weight coefficient.

[0168] When the fourth loss is obtained, it is matched with the preset loss. If the preset loss is less than or equal to the preset loss, the two are considered a match; otherwise, they are not a match. When they do not match, the initial weight coefficients can be adaptively calculated to match the fourth loss with the preset loss. During the adaptive adjustment process, the initial weight coefficients of all target expanded texts can be adjusted. When the fourth loss matches the preset loss, the update of the initial weight coefficients stops. Among the adjusted initial weight coefficients, some coefficients may remain unchanged, or all weight coefficients may change.

[0169] During the process of updating the initial weight coefficients, the cross-entropy loss function can also be used to narrow the gap between image features and text features, thereby increasing the distance between liveness detection attack type text and liveness detection sample images. The formula is as follows:

[0170]

[0171] Where N is the number of categories or the number of images, and sim() represents the cosine similarity. Representing the visual features of an image, while This represents the corresponding category hint feature, which is the corresponding text feature, y. i It is the i-th true label. The true label is the correct pairing relationship in the image-text pair. A correct pair is 1, and other pairs are 0.

[0172] 160. Based on the first text features and image features, obtain the first loss, and based on the second text features and image features, obtain the second loss.

[0173] Vector calculations are performed on the first text features and image features to obtain a first loss, and vector calculations are performed on the second text features and image features to obtain a second loss. Specifically, the first loss, obtained based on the first text features and image features, includes:

[0174] Obtain the text feature vector of the first text feature and the image feature vector of the image feature;

[0175] The first loss is obtained by multiplying the text feature vector and the image feature vector.

[0176] That is Where V i =Proj(VisEncoder(I i )) indicates that image I will be i Image features obtained by inputting into the image encoder This indicates that the text will be... The text features are input to the text encoder. Both the image encoder and the text encoder belong to the cross-modal detection model, with the image encoder belonging to the image branch and the text encoder belonging to the text branch.

[0177] In addition to training the model using image features, the model can also be trained using first text features unrelated to liveness detection. This allows the model to better decouple non-liveness attack content from the image. The smaller the first loss value, the stronger the decoupling ability, enabling the model to more accurately exclude non-liveness attack content from the image. Since the model can learn discriminative features unrelated to liveness detection, it can enhance the model's cross-domain capability and better integrate with language text supervision. This allows the model to better process relevant data in conjunction with text and improve the model's generalization ability.

[0178] In some embodiments, the decoupling of features in a liveness image can also be performed using adversarial learning. For example, features unrelated to liveness detection in the image can be first obtained, and then a first loss can be calculated using the cross-entropy loss function. Where N is the number of categories for irrelevant features; for example, when classifying gender, N is 2. These are true labels that are irrelevant to the features, such as the gender label being male or female. It is the probability distribution of irrelevant feature categories predicted by the adversarial network, and the probability values ​​are obtained by using the Softmax function.

[0179] Suppose the output vector of the adversarial network is z = [1.5, 2.5, 0.3], representing the scores of the three categories in a three-class classification task (e.g., a person's gender might be categorized into three classes: male, female, and other). These scores are transformed into a probability distribution using the Softmax function, with an exponent of e^(-1 / 2). 1.5 =4.48, e 2.5 =12.18, e 0.3=1.35, the sum of the exponents is 4.48 + 12.18 + 1.35 = 18.01, then the probabilities for each are 4.48 / 18.01 = 0.25, 12.18 / 18.01 = 0.68, and 1.35 / 18.01 = 0.07, respectively. The final result is... This indicates that the adversarial network predicts the highest probability of category 2 (the category with a score of 2.5). In adversarial networks, a smaller adversarial loss indicates a higher accuracy in identifying irrelevant features. In the main network, i.e., the liveness detection network, the goal is to maximize the adversarial loss, which means suppressing the adversarial network's ability to identify irrelevant features, thereby removing irrelevant features and focusing on features relevant to liveness detection. Through this game of adversarial loss, the main network gradually learns to extract only effective features relevant to liveness detection, while the role of the adversarial network is to decouple and allow these irrelevant features to be stripped away and removed as much as possible.

[0180] Furthermore, in some embodiments, the decoupling of features in the liveness image can also be performed by reconstruction loss, whereby the encoder separates features related to liveness detection from features unrelated to liveness detection through the image decoupling module.

[0181] The encoder extracts two features from the input image: features T1 that are relevant to liveness detection and features T2 that are irrelevant. The decoder takes T2 as input to obtain the reconstructed image. By comparing the differences between the input image and the reconstructed image, the reconstruction loss can be determined. The reconstruction loss is used to ensure that the encoder can correctly extract features that are irrelevant to liveness detection and reconstruct these parts. The liveness detection loss is used for liveness detection tasks (such as cross-entropy loss) to ensure that the relevant features extracted by the encoder can be used to accurately classify live or fake bodies.

[0182] By minimizing the reconstruction loss, the decoder learns how to reconstruct irrelevant parts of the input image from the features, enabling the encoder to separate these irrelevant features and better identify parts unrelated to liveness detection. At the same time, by minimizing the liveness detection loss, the liveness detection branch will focus more on features related to liveness detection, thereby improving the model's detection accuracy. As training progresses, the model gradually learns how to decouple different features in the input image, separate irrelevant features for reconstruction, and use relevant features for liveness detection.

[0183] 170. Based on the first loss and the second loss, update the liveness detection parameters of the text branch and the image branch until the first loss and the second loss obtained again meet the preset loss values.

[0184] After obtaining the first and second losses, the liveness detection parameters of the text and image branches can be updated using these losses. These parameters correspond to the weight coefficients in the text branch and also include the parameters in the image branch. After obtaining the first loss, its gradient relative to the parameters of the image branch is propagated back through backpropagation. Backpropagation calculates the impact of each parameter on the loss, and the parameters of the image branch are updated based on this gradient information, reducing the loss and gradually decreasing the similarity between the first text features and image features. Parameters modified in the image branch include convolutional kernel weights, fully connected layer weights, and biases. During this process, the model continuously trains, constantly obtaining new first and second losses, thus continuously updating the liveness detection parameters until both the first and second losses meet their respective preset loss values. At this point, model training stops, and the liveness detection parameters updated based on the final first and second losses are considered to meet the final requirements.

[0185] By updating the liveness detection parameters of the cross-modal detection model through the first loss and the second loss, the model gradually adjusts its internal parameters through multiple iterations of training using the first loss. This makes the dot product result (i.e., similarity) between the feature vectors generated by the image encoder and the text encoder smaller and smaller, thereby achieving the purpose of decoupling image features from text features. This not only enables the model to learn more information unrelated to liveness detection through text supervision, thereby improving the model's generalization ability, but also continuously improves the accuracy of liveness attack type identification through text supervision learning based on the second loss.

[0186] As can be seen from the above, this embodiment can acquire liveness detection sample images to be trained; based on preset non-liveness attack type text description prompts, the liveness detection sample images are described using an image description model to obtain a first description text; features are extracted from the first description text through the text branch of the cross-modal detection model to obtain a first text feature; features are extracted from the liveness detection sample images through the image branch of the cross-modal detection model to obtain image features; features are extracted from the preset liveness detection attack type text through the text branch to obtain a second text feature; the first text feature and image feature are calculated to obtain a first loss; the second text feature and image feature are calculated to obtain a second loss; and the liveness recognition parameters of the text branch and image branch are updated based on the first loss and the second loss. This application extracts first text features from the first descriptive text of non-liveness attack types through the text branch of a cross-modal detection model, and extracts image features from the liveness detection sample image through the image branch. Then, a first loss is calculated using the first text features and the image features. Next, a second text feature is extracted from the text of liveness detection attack types through the text branch, and a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Thus, in addition to training the model using image features, it can also train the model using first text features that are unrelated to liveness detection. This allows the model to better decouple non-liveness attack content in the image. The smaller the first loss value, the stronger the decoupling ability, and the more accurately the model can exclude non-liveness attack factors in the image. Since the model can learn discriminative features unrelated to liveness detection, it can enhance the model's cross-domain capability, and thus better combine the supervision capability of language text, enabling the model to better process relevant data in conjunction with text and improve the model's generalization ability.

[0187] Furthermore, through the aforementioned method, during model training, by combining non-liveness detection text description prompts to obtain non-liveness detection text features, and by combining liveness detection attack type text features to obtain liveness attack text features, these two are combined with image features, overcoming the limitations of the current single-element prompts. At the same time, by combining fine-grained classification to expand the text, these fine-grained category prompts can provide the model with more forgery information (attack information), enabling the model to learn the subtle differences between different attack types, overcoming the limitations of insufficient precision of coarse-grained elements, thereby enhancing their ability to distinguish them, and thus more fully exploring the potential of language supervision, further improving the model's generalization ability.

[0188] Based on the method described in the preceding embodiments, the following will provide a more detailed explanation by taking the specific integration of the model training device into a terminal as an example.

[0189] This application provides a model training method, such as... Figure 3 As shown, the specific process of this model training method can be as follows:

[0190] 210. The terminal acquires the liveness detection sample images to be trained.

[0191] Before training the model, the first step is to acquire liveness detection sample images for training. Since the model training aims to improve liveness detection, these sample images are all facial images. This can be done using a camera; the captured liveness detection sample images can be real facial images or facial images obtained by capturing images containing facial features. Of course, uploaded images can also be used, but this embodiment does not limit this.

[0192] 220. The terminal uses an image description model to perform text description on the liveness detection sample image based on environmental and facial prompts to obtain the first description text.

[0193] To enable the cross-modal detection model to better identify information unrelated to liveness detection, improve its cross-domain detection capability, and enhance its generalization ability, the image description model can be used to describe the liveness detection sample image using preset text description prompts for non-liveness attack types, thus obtaining the first description text.

[0194] In other words, the image description model generates a text description of the liveness detection sample image based on the text description prompts and according to the logic of the text description prompts, thus obtaining the first description text. The content and format of the first description text all meet the requirements of the text description prompts.

[0195] Since the environmental information and facial semantics of an image are irrelevant to the liveness detection task, an image description model can be used to generate textual descriptions related to the environmental information and facial semantics. The resulting descriptive text is the first descriptive text. Environmental information can include background and lighting, while facial information includes facial features, gender, and expression. Of course, more information can be included in the environmental and facial information during this process, depending on the specific requirements.

[0196] 230. The terminal extracts features from the first descriptive text through the text branch of the cross-modal detection model to obtain the first text features.

[0197] After obtaining the first descriptive text, it is input into the text branch of the cross-modal detection model (such as a pre-trained Transformer model). The model encodes each word or subword into an embedding vector, which is text encoding. After all word vectors are processed, the model generates a global semantic representation, which is a fixed-dimensional feature vector. The generated feature vector will capture the key semantic information in the descriptive text, such as "tranquil outdoor scene", "middle-aged man", "soft sunlight", etc. The obtained feature vectors are the first text features.

[0198] 240. The terminal extracts features from the liveness detection sample image through the image branch of the cross-modal detection model to obtain image features.

[0199] After the liveness detection sample image is input into the cross-modal detection model, the image branch of the cross-modal detection model (such as the ResNet model, VGG model, Vision Transformer (ViT) model, CLIP model, and other visual models) will encode the liveness detection sample image to obtain a high-dimensional feature vector, which represents the global or local features of the image.

[0200] 250. The terminal extracts features from the preset liveness detection attack type text through the text branch to obtain the second text feature.

[0201] The preset liveness detection attack type text is the text corresponding to the attack type mentioned above, such as "A photo of a print face," which corresponds to a printing attack. Feature extraction is performed on the liveness detection attack type text through a text branch to obtain the second text feature. The method for obtaining the second text feature is the same as that for obtaining the first text feature: acquiring a global semantic representation. The global semantic representation is a fixed-dimensional feature vector, and the generated feature vector will capture the key semantic information describing the text.

[0202] 260. The terminal performs a dot product calculation on the text feature vector and the image feature vector to obtain the first loss, and obtains the second loss based on the second text feature and the image feature.

[0203] Vector calculations are performed on the first text features and image features to obtain the first loss, and vector calculations are performed on the second text features and image features to obtain the second loss.

[0204] 270. The terminal updates the liveness detection parameters of the text branch and the image branch according to the first loss and the second loss until the first loss and the second loss obtained again meet the preset loss values.

[0205] After obtaining the first and second losses, the liveness detection parameters of the text and image branches can be updated using these losses. These parameters correspond to the weight coefficients in the text branch and also include the parameters in the image branch. After obtaining the first loss, its gradient relative to the parameters of the image branch is propagated back through backpropagation. Backpropagation calculates the impact of each parameter on the loss, and the parameters of the image branch are updated based on this gradient information, thus reducing the loss and gradually decreasing the similarity between the first text feature and the image feature. Parameters modified in the image branch include convolutional kernel weights, fully connected layer weights, and biases.

[0206] Overall process reference Figure 4 , Figure 4 This is a flowchart illustrating the model training method provided in this application embodiment. As can be seen, this embodiment can acquire liveness detection sample images to be trained via a terminal; based on preset text description prompts for non-liveness attack types, the liveness detection sample images are described using an image description model to obtain a first description text; features are extracted from the first description text using the text branch of a cross-modal detection model to obtain a first text feature; features are extracted from the liveness detection sample images using the image branch of a cross-modal detection model to obtain image features; features are extracted from preset liveness detection attack type text using the text branch to obtain a second text feature; the first text feature and image feature are calculated to obtain a first loss; the second text feature and image feature are calculated to obtain a second loss; and the liveness recognition parameters of the text branch and image branch are updated based on the first loss and the second loss. This application extracts first text features from the first descriptive text of non-liveness attack types through the text branch of a cross-modal detection model, and extracts image features from the liveness detection sample image through the image branch. Then, a first loss is calculated using the first text features and the image features. Next, a second text feature is extracted from the text of liveness detection attack types through the text branch, and a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Thus, in addition to training the model using image features, it can also train the model using first text features that are unrelated to liveness detection. This allows the model to better decouple non-liveness attack content in the image. The smaller the first loss value, the stronger the decoupling ability, and the more accurately the model can exclude non-liveness attack factors in the image. Since the model can learn discriminative features unrelated to liveness detection, it can enhance the model's cross-domain capability, and thus better combine the supervision capability of language text, enabling the model to better process relevant data in conjunction with text and improve the model's generalization ability.

[0207] To better implement the above methods, embodiments of this application also provide a model training device, such as... Figure 5 As shown, the model training device may include an image processing module, a content element decoupling module, and a fine-grained classification element module. The image processing module includes an image acquisition module 310 and an image feature acquisition module 340. The content element decoupling module includes a first text acquisition module 320, a first feature acquisition module 330, and a loss acquisition module 360. The fine-grained classification element module includes a second feature acquisition module 350 and a loss acquisition module 360. The model training device also includes a parameter update module 370, as follows:

[0208] Image acquisition module 310 is used to acquire images of liveness detection samples to be trained;

[0209] The first text acquisition module 320 is used to perform text description on the liveness detection sample image based on the preset text description prompts for non-liveness attack types and through the image description model to obtain the first description text.

[0210] The first feature acquisition module 330 is used to extract features from the first descriptive text through the text branch of the cross-modal detection model to obtain the first text features;

[0211] The image feature acquisition module 340 is used to extract features from the liveness detection sample image through the image branch of the cross-modal detection model to obtain image features;

[0212] The second feature acquisition module 350 is used to extract features from the preset liveness detection attack type text through the text branch to obtain the second text features;

[0213] The loss acquisition module 360 ​​is used to obtain a first loss based on the first text features and image features, and to obtain a second loss based on the second text features and image features;

[0214] The parameter update module 370 is used to update the liveness detection parameters of the text branch and the image branch based on the first loss and the second loss.

[0215] In this process, in addition to training the model using image features, it can also be trained using first text features unrelated to liveness detection. This allows the model to better decouple non-liveness attack content from images. The smaller the first loss value, the stronger the decoupling ability, enabling the model to more accurately exclude non-liveness attack factors from images. Because the model can learn discriminative features unrelated to liveness detection, it enhances the model's cross-domain capabilities, thereby better integrating language text supervision. This allows the model to better process relevant data with text, improving its generalization ability. Furthermore, combining the above model training methods, during text augmentation using fine-grained classification element modules, attack categories can be further subdivided. These fine-grained category cues provide the model with more forgery information (attack information), enabling the model to learn subtle differences between different attack types, thus enhancing its ability to distinguish them.

[0216] As can be seen from the above, in this embodiment, the image acquisition module can acquire the liveness detection sample image to be trained; the first text acquisition module, based on the preset non-liveness attack type text description prompts, performs text description on the liveness detection sample image through the image description model to obtain the first description text; the first feature acquisition module extracts features from the first description text through the text branch of the cross-modal detection model to obtain the first text features; the image feature acquisition module extracts features from the liveness detection sample image through the image branch of the cross-modal detection model to obtain the image features; the second feature acquisition module extracts features from the preset liveness detection attack type text through the text branch to obtain the second text features; the loss acquisition module obtains the first loss based on the first text features and the image features, and obtains the second loss based on the second text features and the image features; the parameter update module updates the liveness recognition parameters of the text branch and the image branch based on the first loss and the second loss. This application extracts first text features from the first descriptive text of non-liveness attack types through the text branch of a cross-modal detection model, and extracts image features from the liveness detection sample image through the image branch. Then, a first loss is calculated using the first text features and the image features. Next, a second text feature is extracted from the text of liveness detection attack types through the text branch, and a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Thus, in addition to training the model using image features, it can also train the model using first text features that are unrelated to liveness detection. This allows the model to better decouple non-liveness attack content in the image. The smaller the first loss value, the stronger the decoupling ability, and the more accurately the model can exclude non-liveness attack factors in the image. Since the model can learn discriminative features unrelated to liveness detection, it can enhance the model's cross-domain capability, and thus better combine the supervision capability of language text, enabling the model to better process relevant data in conjunction with text and improve the model's generalization ability.

[0217] To better implement the above methods, this application also provides a liveness detection method, such as... Figure 6 As shown, the procedure for this liveness detection method is as follows:

[0218] 410. Obtain the image to be detected.

[0219] The image to be detected can be obtained by taking pictures with a camera. For example, in the application of community access control, when a user needs to enter the community, he or she needs to enter by scanning his or her face. At this time, the access control camera can take pictures of the face to obtain the image to be detected. Of course, the image to be detected can also be a photo or other video recording placed on the community access control camera and captured by the camera.

[0220] 420. The image features of the image to be detected are extracted by the image branch of the cross-modal detection model. The cross-modal detection model is trained using the model training method described above.

[0221] The image branch of the cross-modal detection model trained using the above model training method will extract features from the image to be detected, thus obtaining the image features of the image to be detected.

[0222] 430. Obtain different attack texts for attacking the image to be detected.

[0223] During the recognition process of the image to be detected, the text prompts preset in the text branch of the cross-modal detection model generate different attack texts based on the image to be detected. These text prompts include various types, such as printing attacks, replay attacks, and 3D head model attacks; different attack text prompts will generate different attack texts.

[0224] 440. By extracting features from different attack texts through the text branch of the cross-modal detection model, the text features corresponding to different attack texts are obtained.

[0225] Then, the text branch of the cross-modal detection model is used to extract features from different attack texts to obtain the text features corresponding to different attack texts. That is, each type of attack text has its own text features. These text features can be used to determine the correlation between the attack text and the image. The higher the correlation, the more it proves that the image is not a living object.

[0226] 450. Obtain the similarity between image features and each text feature.

[0227] 460. Based on the similarity, obtain the liveness detection results of the image to be detected.

[0228] For example, the cosine similarity between image features and each text feature can be calculated, and then the liveness detection result of the image to be detected can be determined based on the obtained similarity. The higher the similarity between the image to be detected and the attack text, the more it proves that the image to be detected is an attack image, that is, a non-live image.

[0229] Therefore, if the similarity value exceeds the threshold, it proves that the liveness detection result of the image to be detected is not live, and the access control fails. If the similarity value does not exceed the threshold, it proves that the image to be detected is likely live, so the liveness detection result is live, and the access control passes. By modifying the liveness detection method described above to better distinguish between live and non-live individuals, the accuracy and security of liveness detection can be improved.

[0230] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0231] This application also provides an electronic device, such as... Figure 7 The diagram shows a structural schematic of an electronic device involved in an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0232] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will understand that... Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0233] The processor 101 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 102, and calls data stored in the memory 102, to perform various functions and process data. Optionally, the processor 101 may include one or more processing cores; preferably, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 101.

[0234] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0235] The electronic device also includes a power supply 103 that supplies power to the various components. Preferably, the power supply 103 can be logically connected to the processor 101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 103 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0236] The electronic device may also include an input unit 104, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0237] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 101 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 102 according to the following instructions, and the processor 101 runs the applications stored in the memory 102 to realize various functions, as follows:

[0238] Acquire liveness detection sample images to be trained; based on preset text description prompts for non-liveness attack types, perform text descriptions on the liveness detection sample images using an image description model to obtain first description text; extract features from the first description text using the text branch of a cross-modal detection model to obtain first text features; extract features from the liveness detection sample images using the image branch of a cross-modal detection model to obtain image features; extract features from the preset liveness detection attack type text using the text branch to obtain second text features; calculate the first text features and image features to obtain a first loss; calculate the second text features and image features to obtain a second loss; update the liveness recognition parameters of the text branch and image branch based on the first loss and the second loss.

[0239] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0240] As can be seen from the above, this embodiment can acquire liveness detection sample images to be trained; based on preset non-liveness attack type text description prompts, the liveness detection sample images are described using an image description model to obtain a first description text; features are extracted from the first description text through the text branch of the cross-modal detection model to obtain a first text feature; features are extracted from the liveness detection sample images through the image branch of the cross-modal detection model to obtain image features; features are extracted from the preset liveness detection attack type text through the text branch to obtain a second text feature; the first text feature and image feature are calculated to obtain a first loss; the second text feature and image feature are calculated to obtain a second loss; and the liveness recognition parameters of the text branch and image branch are updated based on the first loss and the second loss. This application extracts first text features from the first descriptive text of non-liveness attack types through the text branch of a cross-modal detection model, and extracts image features from the liveness detection sample image through the image branch. Then, a first loss is calculated using the first text features and the image features. Next, a second text feature is extracted from the text of the liveness detection attack type through the text branch, and a second loss is calculated using the second text features and the image features. Finally, the liveness recognition parameters of the text branch and the image branch are updated based on the first loss and the second loss. Thus, in addition to training the model using image features, it can also train the model using text features, which can better combine the supervision capabilities of language text, enabling the model to better process relevant data in conjunction with text, thereby improving the model's generalization ability.

[0241] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0242] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the model training methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0243] Acquire liveness detection sample images to be trained; based on preset non-liveness attack type text description prompts, perform text description on the liveness detection sample images using an image description model to obtain first description text; extract features from the first description text using the text branch of a cross-modal detection model to obtain first text features; extract features from the liveness detection sample images using the image branch of a cross-modal detection model to obtain image features; extract features from the preset liveness detection attack type text using the text branch to obtain second text features; calculate the first text features and image features to obtain a first loss; calculate the second text features and image features to obtain a second loss; update the liveness recognition parameters of the text branch and image branch based on the first loss and the second loss.

[0244] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0245] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0246] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the model training methods provided in the embodiments of this application, the beneficial effects that any of the model training methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0247] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations regarding the above-described content ordering.

[0248] The above provides a detailed description of a model training method and related equipment provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0249] It should be noted that when the above embodiments of this application are applied to specific products or technologies, the relevant data of the object needs to be authorized or agreed to by the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A model training method, characterized in that, include: Obtain the liveness detection sample images to be trained; Based on the preset text description prompts for non-liveness attack types, the liveness detection sample image is described using an image description model to obtain the first description text; The first text features are obtained by extracting features from the first descriptive text through the text branch of the cross-modal detection model; Feature extraction is performed on the liveness detection sample images using the image branch of the cross-modal detection model to obtain image features; The text branch is used to extract features from the preset liveness detection attack type text to obtain the second text feature. The liveness detection attack type text is generated according to the attack type of liveness attack. A first loss is obtained based on the first text features and the image features, and a second loss is obtained based on the second text features and the image features; Based on the first loss and the second loss, update the liveness detection parameters of the text branch and the image branch until the first loss and the second loss are re-acquired and satisfy the preset loss value.

2. The model training method according to claim 1, characterized in that, The text description prompts include environmental prompts and face prompts. The first description text is obtained by performing text descriptions on the liveness detection sample image using an image description model based on preset non-liveness attack type text description prompts, including: Based on the environmental cue words, the liveness detection sample image is described using an image description model to obtain the environmental description text. Based on the facial prompts, the liveness detection sample image is described using an image description model to obtain facial description text; Based on the environmental description text and the face description text, a first description text is obtained.

3. The model training method according to claim 1, characterized in that, The step of obtaining the first loss based on the first text features and the image features includes: Obtain the text feature vector of the first text feature and the image feature vector of the image feature; The first loss is obtained by performing a dot product on the text feature vector and the image feature vector.

4. The model training method according to claim 1, characterized in that, The step of extracting features from the preset liveness detection attack type text through the text branch to obtain the second text features includes: Based on preset text expansion prompts, the text expansion model is used to expand the text of liveness detection attack types, resulting in multiple attack type expanded texts; By extracting features from multiple attack type extended texts through the text branches, a second text feature is obtained.

5. The model training method according to claim 4, characterized in that, The step of extracting features from multiple attack type extended texts through the text branch to obtain second text features includes: By extracting features from multiple attack type extended texts through the text branches, a third text feature for each attack type extended text is obtained; The feature similarity is calculated by combining the third text feature and the image feature. Multiple attack type extended texts that match the feature similarity with a preset similarity are obtained and used as target extended texts; The second text features are obtained by extracting features from multiple target extended texts through the text branches.

6. The model training method according to claim 5, characterized in that, The step of extracting features from multiple target extended texts through the text branches to obtain second text features includes: Obtain the target weight coefficient for each of the target extended texts; Based on the target weight coefficient, feature extraction is performed on multiple target extended texts through the text branch to obtain the second text features.

7. The model training method according to claim 6, characterized in that, The step of obtaining the target weight coefficient for each of the target expanded texts includes: Obtain the initial weight coefficient for each of the target expanded texts; Based on the initial weight coefficients, feature extraction is performed on multiple target extended texts through the text branches to obtain the fourth text feature; The fourth loss is calculated by combining the fourth text features and the image features; Based on the fourth loss and the preset loss, the initial weight coefficients are updated to obtain the target weight coefficients for each target extended text.

8. The model training method according to claim 7, characterized in that, The step of updating the initial weight coefficients based on the fourth loss and the preset loss includes: When the fourth loss does not match the preset loss, the initial weight coefficient is adjusted until the fourth loss matches the preset loss, at which point the update of the initial weight coefficient stops.

9. A method for detecting liveness, characterized in that, include: Acquire the image to be detected; The image features of the image to be detected are obtained by extracting features from the image branch of the cross-modal detection model. The cross-modal detection model is trained using the model training method described in any one of claims 1 to 8. Obtain different attack texts that attack the image to be detected; By extracting features from different attack texts through the text branch of the cross-modal detection model, the text features corresponding to different attack texts are obtained. Obtain the similarity between the image features and each of the text features; Based on the similarity, the liveness detection result of the image to be detected is obtained.

10. A model training device, characterized in that, include: The image acquisition module is used to acquire images of liveness detection samples to be trained. The first text acquisition module is used to perform text description on the liveness detection sample image based on preset text description prompts for non-liveness attack types using an image description model, and obtain the first description text. The first feature acquisition module is used to extract features from the first descriptive text through the text branch of the cross-modal detection model to obtain the first text features; The image feature acquisition module is used to extract features from the liveness detection sample image through the image branch of the cross-modal detection model to obtain image features; The second feature acquisition module is used to extract features from the preset liveness detection attack type text through the text branch to obtain the second text feature, wherein the liveness detection attack type text is generated according to the attack type of liveness attack. The loss acquisition module is used to obtain a first loss based on the first text features and the image features, and to obtain a second loss based on the second text features and the image features; The parameter update module is used to update the liveness detection parameters of the text branch and the image branch according to the first loss and the second loss until the re-obtained first loss and second loss meet the preset loss values.

11. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program within the memory to perform the operations in the model training method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the model training method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the model training method according to any one of claims 1 to 8.

Images