Live detection model training method and apparatus

Abstract

This disclosure provides a method and apparatus for training a liveness detection model, relating to the field of artificial intelligence technology, specifically computer vision, image processing, and deep learning, and applicable to scenarios such as liveness detection. One specific implementation of the method includes: acquiring sample image pairs; training a single-modality liveness detection model based on a first-modality sample image, adjusting the parameters of the single-modality liveness detection model to obtain a first-modality liveness detection model; adding a prompting module and a second-modality classification head to the first-modality liveness detection model; training the first-modality liveness detection model with the added prompting module and second-modality classification head based on a second-modality sample image, adjusting the parameters of the prompting module to obtain a hybrid-modality liveness detection model. This implementation only requires training one hybrid-modality liveness detection model to achieve simultaneous face liveness detection on multi-modal images.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning, and other technical fields, and can be applied to scenarios such as liveness detection. Background Technology

[0002] Face liveness detection, which distinguishes whether an image is taken of a real person, is a fundamental component of face recognition systems, ensuring their security. Face liveness detection algorithms using deep learning technology are currently the mainstream method in this field, offering significant improvements in accuracy compared to traditional algorithms.

[0003] Currently, commonly used face liveness detection algorithms are based on RGB (red, green, blue) images, which can detect live faces in RGB images. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, storage medium, and program product for training a liveness detection model.

[0005] In a first aspect, embodiments of this disclosure propose a method for training a liveness detection model, comprising: acquiring sample image pairs, wherein the sample image pairs consist of a first modality sample image and a second modality sample image containing the same face; training a single-modality liveness detection model based on the first modality sample image, adjusting the parameters of the single-modality liveness detection model to obtain a first modality liveness detection model; adding a prompting module and a second modality classification head to the first modality liveness detection model; and training the first modality liveness detection model with the added prompting module and second modality classification head based on the second modality sample image, adjusting only the parameters of the prompting module to obtain a hybrid modality liveness detection model.

[0006] Secondly, embodiments of this disclosure propose a liveness detection method, comprising: acquiring a target image in a first modality and / or a second modality; inputting the target image into a liveness detection model to obtain a liveness detection result of the target image, wherein the liveness detection model is trained using the method described in the first aspect.

[0007] Thirdly, embodiments of this disclosure propose a liveness detection model training apparatus, comprising: an acquisition module configured to acquire sample image pairs, wherein the sample image pairs consist of a first modality sample image and a second modality sample image containing the same face; a first training module configured to train a single-modality liveness detection model based on the first modality sample image, adjusting the parameters of the single-modality liveness detection model to obtain a first modality liveness detection model; an addition module configured to add a prompting module and a second modality classification head to the first modality liveness detection model; and a second training module configured to train the first modality liveness detection model with the added prompting module and second modality classification head based on the second modality sample image, adjusting only the parameters of the prompting module to obtain a hybrid modality liveness detection model.

[0008] Fourthly, embodiments of this disclosure propose a liveness detection device, comprising: an acquisition module configured to acquire a first modality and / or a second modality of an image to be detected; and a detection module configured to input the image to be detected into a liveness detection model to obtain a liveness detection result of the image to be detected, wherein the liveness detection model is trained using the device described in the third aspect.

[0009] Fifthly, embodiments of this disclosure provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect, or to perform a method as described in any implementation of the second aspect.

[0010] In a sixth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method as described in any implementation of the first aspect, or to perform a method as described in any implementation of the second aspect.

[0011] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any implementation of the first aspect, or implements the method described in any implementation of the second aspect.

[0012] The liveness detection model training method provided in this disclosure first trains a first-modality liveness detection model based on sample data from a first modality. Then, a prompting module and a second-modality classification head are added to the trained first-modality liveness detection model. Finally, when continuing training based on sample data from a second modality, the parameters of the first-modality liveness detection model are kept unchanged, and only the parameters of the prompting module are adjusted to obtain a hybrid-modality liveness detection model. Only one hybrid-modality liveness detection model needs to be trained to simultaneously detect face liveness in images from both the first and second modalities. Face liveness detection based on multimodal images reduces sensitivity to lighting conditions and improves generalization against planar attacks such as those from photos and videos, thereby enhancing the model's application performance. Furthermore, compared to traditional multimodal solutions, it avoids model redundancy and improves model computation speed.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein:

[0015] Figure 1 This is a flowchart of an embodiment of the liveness detection model training method according to the present disclosure;

[0016] Figure 2 This is a flowchart of yet another embodiment of the liveness detection model training method according to the present disclosure;

[0017] Figure 3 This is a flowchart of another embodiment of the liveness detection model training method according to the present disclosure;

[0018] Figure 4 This is a scene diagram illustrating the liveness detection model training method implemented in the embodiments of this disclosure.

[0019] Figure 5 This is a flowchart of one embodiment of the liveness detection method according to the present disclosure;

[0020] Figure 6 This is a schematic diagram of the structure of an embodiment of the liveness detection model training device according to the present disclosure;

[0021] Figure 7 This is a schematic diagram of a structure of an embodiment of the liveness detection device according to the present disclosure;

[0022] Figure 8 This is a block diagram of an electronic device used to implement the liveness detection model training method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 A flowchart 100 of an embodiment of a liveness detection model training method according to the present disclosure is shown. The liveness detection model training method includes the following steps:

[0026] Step 101: Obtain sample image pairs.

[0027] In this embodiment, the entity executing the liveness detection model training method can acquire sample image pairs.

[0028] The sample image pair can consist of a first modality sample image and a second modality sample image containing the same face. The first modality sample image and the second modality sample image can be images of different modalities. For example, the first modality sample image can be an RGB image, and the second modality sample image can be an NIR (Near Infrared) image.

[0029] In some embodiments, to improve the recognition performance of the liveness detection model on images of different modalities, the same face contained in the first modality sample image and the second modality sample image in the sample image pair needs to be aligned. For example, the sample image pair can be obtained through the following steps:

[0030] First, obtain the image pairs.

[0031] Typically, a series of first modality images and second modality images containing a human face are acquired. These first and second modality images are paired images, i.e., image pairs. An image pair may consist of a first modality image and a second modality image containing the same human face. The first and second modality images can be images of different modalities; for example, the first modality image can be an RGB image, and the second modality image can be an NIR image.

[0032] Then, the first modality image and the second modality image are face aligned to obtain sample image pairs.

[0033] Face alignment, in particular, standardizes face images to improve subsequent liveness detection. Specifically, face alignment uses transformations such as rotation, scaling, and translation to homogenize face images under different poses, angles, and lighting conditions to a fixed size and pose. This eliminates information distortion and loss caused by face positional deviations and poses, enhancing the ability of liveness detection algorithms to recognize facial appearance and structural features.

[0034] Step 102: Train the single-modal liveness detection model based on the first modality sample image, adjust the parameters of the single-modal liveness detection model, and obtain the first modality liveness detection model.

[0035] In this embodiment, the aforementioned execution entity can train the single-modal liveness detection model based on the first modal sample image, adjust the parameters of the single-modal liveness detection model, and obtain the first modal liveness detection model.

[0036] Typically, the first modality sample image is labeled with a category label to indicate whether it is a live image. The first modality sample image is input into a single-modality liveness detection model, which outputs the liveness detection result for the first modality sample image. Based on the liveness detection result and the category label, a loss is calculated. The parameters of the single-modality liveness detection model are adjusted based on the loss until the model converges, resulting in the first modality liveness detection model.

[0037] The unimodal liveness detection model can include a feature extraction network and a classification head. The feature extraction network can be, for example, a ViT-base (Vision Transformer base) model, used to extract class token features. The classification head can be used for image classification. Here, the unimodal liveness detection model is trained using sample images from the first modality to obtain the first modality liveness detection model. The feature extraction network of the first modality liveness detection model can be used to extract class token features from the images of the first modality, and the classification head of the first modality liveness detection model can be used to classify the images of the first modality. This classification head is then called the first modality classification head.

[0038] The loss function of the first modality liveness detection model can be, for example, a binary cross-entropy loss function. During training, the binary cross-entropy loss function is used for supervision to obtain the first modality liveness detection model. This model can be used to detect whether an image in the first modality is a live image.

[0039] Step 103: Add a prompting module and a second modality classification head to the first modality liveness detection model.

[0040] In this embodiment, the aforementioned execution entity can add a prompt token and a second modality classification head to the first modality liveness detection model.

[0041] Typically, multiple cue modules are added to each layer of the first modality liveness detection model. Furthermore, a second modality classification head is added to the first modality liveness detection model to classify images of the second modality. This second modality classification head can be a branch parallel to the first modality classification head, following the feature extraction network.

[0042] Step 104: Train the first modality liveness detection model with added prompting module and second modality classification head based on the second modality sample image, and adjust only the parameters of the prompting module to obtain the hybrid modality liveness detection model.

[0043] In this embodiment, the aforementioned execution entity can train a first modality liveness detection model with added prompting module and second modality classification head based on the second modality sample image. The parameters of the first modality liveness detection model are fixed, and only the parameters of the prompting module are adjusted to obtain a hybrid modality liveness detection model.

[0044] Typically, second-modality sample images are labeled with category labels to indicate whether they represent liveness. The second-modality sample images are input into a first-modality liveness detection model incorporating a cueing module and a second-modality classification head. The second-modality classification head outputs the liveness detection result for the second-modality sample images. The loss is calculated based on the liveness detection result and category labels. The parameters of the first-modality liveness detection model are fixed, and only the parameters of the cueing module are adjusted based on the loss until the model converges, resulting in a hybrid-modality liveness detection model. Here, a cueing module and a second-modality classification head are added to the first-modality liveness detection model, introducing only a small number of new parameters. During training, the parameters of the first-modality liveness detection model are kept unchanged, and only the parameters of the cueing module are adjusted, reducing the workload of parameter tuning and improving model training efficiency. Furthermore, the trained hybrid-modality liveness detection model can simultaneously detect face liveness in images from both the first and second modalities.

[0045] The hybrid modality liveness detection model can include a feature extraction network and two classification heads: a first modality classification head and a second modality classification head. The feature extraction network is used to extract class-labeled features. When an image of the first modality is input, the feature extraction network outputs the class-labeled features of the first modality image. When an image of the second modality is input, the feature extraction network outputs the class-labeled features of the second modality image. The first modality classification head is used to classify the first modality image. The second modality classification head is used to classify the second modality image.

[0046] The loss function of the hybrid modality liveness detection model can be, for example, a binary cross-entropy loss function. During training, the binary cross-entropy loss function is used for supervision to obtain the hybrid modality liveness detection model. This model can be used to detect whether images from the first and second modalities represent live images.

[0047] The liveness detection model training method provided in this disclosure first trains a first-modality liveness detection model using only sample data from the first modality. Then, a prompting module and a second-modality classification head are added to the trained first-modality liveness detection model. Finally, when training continues based on sample data from the second modality, the parameters of the first-modality liveness detection model are kept unchanged, and only the parameters of the prompting module are adjusted to obtain a hybrid-modality liveness detection model. Training the hybrid-modality liveness detection model for face liveness detection on multimodal images reduces sensitivity to lighting conditions and improves generalization against planar attacks such as photos and videos, thereby enhancing the model's application performance. Furthermore, training only one hybrid-modality liveness detection model is sufficient to simultaneously detect face liveness in images from both the first and second modalities. Compared to traditional multimodal solutions, this method avoids model redundancy and improves model computation speed.

[0048] The liveness detection model training method provided in this disclosure can effectively defend against most common attacks, accelerate network training convergence speed, and improve the generalization and accuracy of the face liveness detection algorithm. It also enhances the defense effect of the face liveness detection algorithm against unknown attack sample methods.

[0049] The liveness detection model training method provided in this disclosure can be applied to any deep learning neural network model for face liveness detection, helping to improve its performance. This technology, by improving the performance of face liveness detection, can be applied to many applications in the field of face recognition, such as attendance, access control, security, and financial payments. It has wide applications in many current businesses. Using this method can improve the performance of face liveness detection technology, helping many applications based on face liveness detection technology to improve their effectiveness and user experience. This is beneficial for the further promotion of business projects.

[0050] Continue to refer to Figure 2 This illustrates a flow 200 of yet another embodiment of the liveness detection model training method according to the present disclosure. The liveness detection model training method includes the following steps:

[0051] Step 201: Obtain image pairs.

[0052] In this embodiment, the entity executing the liveness detection model training method can acquire image pairs.

[0053] Typically, a series of first modality images and second modality images containing a human face are acquired. These first and second modality images are paired images, i.e., image pairs. An image pair may consist of a first modality image and a second modality image containing the same human face. The first and second modality images can be images of different modalities; for example, the first modality image can be an RGB image, and the second modality image can be an NIR image.

[0054] Step 202: Perform face detection on the first modality image and the second modality image, and extract the face region image of the first modality and the face region image of the second modality.

[0055] In this embodiment, the execution entity can perform face detection on the first modality image and crop the face region image of the first modality. Similarly, it can perform face detection on the second modality image and crop the face region image of the second modality.

[0056] Typically, face detection is performed on a first modality image to obtain the location of the first modality's face region. Cropping this first modality's face region from the first modality's image yields the first modality's face region image. Similarly, face detection is performed on a second modality image to obtain the location of the second modality's face region. Cropping this second modality's face region from the second modality's image yields the second modality's face region image.

[0057] In some embodiments, to improve the accuracy of face detection, a face detection model can be used. Specifically, a first modality image is input into a pre-trained first modality face detection model to obtain the location of the first modality face region. Based on the location of the first modality face region, the first modality image is cropped to obtain a first modality face region image. Similarly, a second modality image is input into a pre-trained second modality face detection model to obtain the location of the second modality face region. Based on the location of the second modality face region, the second modality image is cropped to obtain a second modality face region image. The first modality face detection model can be any neural network model capable of performing face detection on images of the first modality, and the second modality face detection model can be any neural network model capable of performing face detection on images of the second modality; no specific limitations are imposed here.

[0058] It should be understood that when the first modality image is an RGB image, the first modality face detection model can be an RGB face detection model used to detect the location of faces in the RGB image. When the second modality image is an NIR image, the second modality face detection model can be an NIR face detection model used to detect the location of faces in the NIR image.

[0059] Step 203: Perform facial landmark detection on the first modality face region image and the second modality face region image to obtain the coordinates of the first modality facial landmarks and the second modality facial landmarks.

[0060] In this embodiment, the aforementioned execution entity can perform facial landmark detection on the first modality face region image to obtain the coordinates of the first modality facial landmarks. Similarly, it can perform facial landmark detection on the second modality face region image to obtain the coordinates of the second modality facial landmarks.

[0061] Typically, N facial keypoints are predefined. Facial keypoint detection is performed on both the first and second modal face region images, yielding N coordinates for the first modal facial keypoints and N coordinates for the second modal facial keypoints. These facial keypoints can include, but are not limited to, feature points such as eyes, mouth, and nose. N is a positive integer, for example, 72. That is, a face is defined with 72 keypoints, and the coordinates of these 72 keypoints are: (x1, y1), ..., (x...). 72 ,y 72 ).

[0062] In some embodiments, to improve the accuracy of facial landmark detection, a facial landmark detection model can be used. Specifically, a first modality face region image is input into a pre-trained first modality facial landmark detection model to obtain the coordinates of the first modality facial landmarks. Similarly, a second modality face region image is input into a pre-trained second modality facial landmark detection model to obtain the coordinates of the second modality facial landmarks. The first facial landmark detection model can be any neural network model capable of performing facial landmark detection on images of the first modality, and the second facial landmark detection model can be any neural network model capable of performing facial landmark detection on images of the second modality; no specific limitations are imposed here.

[0063] It should be understood that when the first modality image is an RGB image, the first modality facial landmark detection model can be an RGB facial landmark detection model, used to detect the coordinates of facial landmarks in the RGB face image. When the second modality image is an NIR image, the second modality facial landmark detection model can be an NIR facial landmark detection model, used to detect the coordinates of facial landmarks in the NIR face image.

[0064] Step 204: Align the faces in the first modality image and the second modality image according to the coordinates of the first modality face key points and the second modality face key points to obtain the first modality aligned image and the second modality aligned image.

[0065] In this embodiment, the execution entity can perform face alignment on the first modality image and the second modality image based on the coordinates of the first modality face key points and the second modality face key points to obtain the first modality aligned image and the second modality aligned image.

[0066] Face alignment based on facial landmarks is also known as facial landmark localization. By detecting and matching the coordinates of facial landmarks, including feature points such as eyes, mouth, and nose, in the first and second modal images, rotation angles, scaling factors, and translation amounts are calculated. This normalizes faces in the first and second modal images with different poses or expressions to the same size and orientation.

[0067] Step 205: Extract the face regions from the first modality-aligned image and the second modality-aligned image through affine transformation to obtain sample image pairs.

[0068] In this embodiment, the aforementioned execution entity can extract the face region of the first modality-aligned image and the second modality-aligned image through affine transformation to obtain a sample image pair.

[0069] Affine transformation is a mathematical model that performs a series of linear transformations and translations on a two-dimensional image on a plane. It can be represented as a matrix multiplication and vector addition. Through affine transformation, operations such as translation, rotation, and scaling of images can be achieved, effectively adjusting the facial pose and size of the first and second modality aligned images, and cropping them to the same size to obtain sample image pairs.

[0070] For example, determining the maximum and minimum coordinate values ​​from 72 facial landmark coordinates, i.e., x max y max x min y min The face bounding box is determined based on the maximum and minimum coordinate values, such as (x... min ,y min ) and (xmax ,y max The rectangle is defined by its diagonal vertices. The face image is cropped by enlarging the rectangle by a preset factor, such as 3 times. The face image is then adjusted to a preset size, such as 224×224.

[0071] Step 206: Train the single-modal liveness detection model based on the first modality sample image, adjust the parameters of the single-modal liveness detection model, and obtain the first modality liveness detection model.

[0072] Step 207: Add a prompting module and a second modality classification head to the first modality liveness detection model.

[0073] Step 208: Train the first modality liveness detection model with added prompting module and second modality classification head based on the second modality sample image, and adjust only the parameters of the prompting module to obtain the hybrid modality liveness detection model.

[0074] In this embodiment, the specific operations of steps 206-208 have been described. Figure 1 Steps 102-104 in the illustrated embodiments are described in detail and will not be repeated here.

[0075] from Figure 2 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the liveness detection model training method in this embodiment emphasizes the step of acquiring sample image pairs. Therefore, the scheme described in this embodiment improves the liveness detection model's recognition performance for images of different modalities by aligning the first modality sample image and the second modality sample image for facial recognition.

[0076] Further reference Figure 3 The diagram illustrates flow 300 of another embodiment of a liveness detection model training method according to the present disclosure. This liveness detection model training method includes the following steps:

[0077] Step 301: Obtain image pairs.

[0078] In this embodiment, the entity executing the liveness detection model training method can acquire image pairs.

[0079] Typically, a series of first modality images and second modality images containing a human face are acquired. These first and second modality images are paired images, i.e., image pairs. An image pair may consist of a first modality image and a second modality image containing the same human face. The first and second modality images can be images of different modalities; for example, the first modality image can be an RGB image, and the second modality image can be an NIR image.

[0080] Step 302: Align the first modality image and the second modality image for face recognition to obtain a sample image pair.

[0081] In this embodiment, the aforementioned execution entity can perform face alignment between the first modal image and the second modal image to obtain a sample image pair.

[0082] Face alignment, in particular, standardizes face images to improve subsequent liveness detection. Specifically, face alignment uses transformations such as rotation, scaling, and translation to homogenize face images under different poses, angles, and lighting conditions to a fixed size and pose. This eliminates information distortion and loss caused by face positional deviations and poses, enhancing the ability of liveness detection algorithms to recognize facial appearance and structural features.

[0083] Step 303: Preprocess the first modality sample image and the second modality sample image to obtain the first modality preprocessed image and the second modality preprocessed image.

[0084] In this embodiment, the execution entity can preprocess the first modality sample image and the second modality sample image to obtain the first modality preprocessed image and the second modality preprocessed image.

[0085] Image preprocessing can remove irrelevant information from an image, recover useful real information, enhance the detectability of relevant information, and simplify the data to the maximum extent, thereby improving the accuracy and efficiency of subsequent face liveness detection. Preprocessing can include, but is not limited to, at least one of the following: normalization processing, data augmentation processing, etc. Image normalization processing can simplify the data to the maximum extent. Data augmentation processing can enhance the detectability of relevant information, including but not limited to, at least one of the following: random flipping, random scaling, color perturbation, etc.

[0086] For example, image normalization is performed on the first modality sample image and the second modality sample image respectively. The normalized image is then subjected to random data augmentation processing, such as random flipping, random scaling, and color perturbation. Specifically, the image normalization process can be performed by normalizing each pixel in the image sequentially. Specifically, the pixel value of each pixel is reduced by 128 and then divided by 256, so that the pixel value of each pixel is between [-0.5, 0.5].

[0087] Step 304: Cut the first modality preprocessed image and the second modality preprocessed image into multiple first modality image blocks and multiple second modality image blocks.

[0088] In this embodiment, the execution entity can cut the first modality preprocessed image and the second modality preprocessed image into multiple first modality image blocks and multiple second modality image blocks.

[0089] For example, the first modality preprocessed image and the second modality preprocessed image are each cut into 196 image blocks of size 16x16.

[0090] Step 305: Input multiple first-modal image blocks into the single-modal liveness detection model and output the first-modal liveness detection results.

[0091] In this embodiment, the aforementioned execution entity can input multiple first modal image blocks into a single modal liveness detection model and output the first modal liveness detection result.

[0092] The single-modal liveness detection model can include a feature extraction network and a classification head. The feature extraction network can be, for example, a ViT-based model, used to extract class label features. The classification head can be used for image classification. Here, the classification head can be used to classify images of the first modality. In this case, this classification head is the first-modal classification head.

[0093] Step 306: Calculate the first loss based on the first modality liveness detection result and the category label of the first modality sample image.

[0094] In this embodiment, the aforementioned execution entity can calculate the first loss based on the first modality liveness detection result and the category label of the first modality sample image.

[0095] Typically, the first modality sample image is labeled with a category label to characterize whether it is a liveness image. The loss function of the first modality liveness detection model can be, for example, a binary cross-entropy loss function. The first loss is obtained by inputting the first modality liveness detection result and the category label of the first modality sample image into the binary cross-entropy loss function.

[0096] Step 307: Based on the first loss, adjust the parameters of the single-modal liveness detection model to obtain the first-modal liveness detection model.

[0097] In this embodiment, the aforementioned execution entity can use the first loss as supervision to adjust the parameters of the single-modal liveness detection model until the model converges, thereby obtaining the first-modal liveness detection model.

[0098] Step 308: Add a prompting module and a second modality classification head to the first modality liveness detection model.

[0099] In this embodiment, the aforementioned execution entity can add a prompting module and a second modality classification head to the first modality liveness detection model.

[0100] Typically, multiple cue modules are added to each layer of the first modality liveness detection model. Furthermore, a second modality classification head is added to the first modality liveness detection model to classify images of the second modality. This second modality classification head can be a branch parallel to the first modality classification head, following the feature extraction network.

[0101] Step 309: Input multiple second modality image blocks into the first modality liveness detection model with the added prompting module and the second modality classification head, and output the second modality liveness detection result.

[0102] In this embodiment, the execution entity can input multiple second modal image blocks into the first modal liveness detection model of the addition prompt module and the second modal classification head, and the second modal classification head can output the second modal liveness detection result.

[0103] Step 310: Calculate the second loss based on the second modality liveness detection results and the category labels of the second modality sample images.

[0104] In this embodiment, the execution entity can calculate the second loss based on the second modality liveness detection result and the category label of the second modality sample image.

[0105] Typically, second-modality sample images are labeled with category labels to characterize whether they represent liveness. The loss function of the second-modality liveness detection model can be, for example, a binary cross-entropy loss function. The second-modality liveness detection result and the category labels of the second-modality sample images are input into the binary cross-entropy loss function to obtain the second loss.

[0106] Step 311: Based on the second loss, adjust only the parameters of the prompting module to obtain the hybrid modality liveness detection model.

[0107] In this embodiment, the aforementioned execution entity can use the second loss as supervision, adjusting only the parameters of the prompting module until the model converges, thus obtaining a hybrid modality liveness detection model.

[0108] The hybrid modality liveness detection model can include a feature extraction network and two classification heads: a first modality classification head and a second modality classification head. The feature extraction network is used to extract class-labeled features. When an image of the first modality is input, the feature extraction network outputs the class-labeled features of the first modality image. When an image of the second modality is input, the feature extraction network outputs the class-labeled features of the second modality image. The first modality classification head is used to classify the first modality image. The second modality classification head is used to classify the second modality image.

[0109] In step 305, the single-modal liveness detection model outputs the first modality liveness detection result and the first modality class label features simultaneously. Similarly, in step 309, the first modality liveness detection model, with the addition of a prompting module and a second modality classification head, outputs the second modality liveness detection result and the second modality class label features simultaneously. To make the features of the second modality sample images similar to the trained features of the first modality sample images, the first modality sample images from the sample image pairs can be added during the training of the hybrid modality liveness detection model. The first modality class label features of the first modality sample images can be used to provide additional supervision for the second modality class label features of the second modality sample images. For example, based on the first and second modality class label features, a third loss is calculated; based on the second and third losses, only the parameters of the prompting module are adjusted to obtain the hybrid modality liveness detection model.

[0110] Adding a small number of parameters can draw on the features of a large number of first-modality sample images. While maintaining the model's detection performance on first-modality images, the added ability to detect second-modality images greatly improves the accuracy and generalization of the hybrid-modality liveness detection model.

[0111] from Figure 3 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the liveness detection model training method in this embodiment emphasizes image preprocessing and model training steps. Therefore, the scheme described in this embodiment preprocesses sample image pairs before model training, eliminating irrelevant information in the images, restoring useful real information, enhancing the detectability of relevant information, and simplifying the data to the maximum extent, thereby improving the accuracy and efficiency of subsequent face liveness recognition. Supervised training of the first modality liveness detection model and the hybrid modality liveness detection model is achieved based on category labels.

[0112] For ease of understanding, Figure 4 A scene diagram is shown illustrating a possible implementation of the liveness detection model training method according to embodiments of this disclosure. For example... Figure 4As shown, the liveness detection model training method consists of two steps: First, training the first modality liveness detection model; second, training the hybrid modality liveness detection model. Specifically, in the first step: Face detection, face alignment, and image preprocessing are performed sequentially on the RGB images in the paired data; then, the images are input into the Vit-base to obtain RGB features (class tokens). These are then input into the RGB classification head to obtain RGB liveness detection results; finally, the first loss, L1 Loss, is calculated based on the RGB liveness detection results and RGB class labels. The Vit-base is then trained based on the L1 loss. In the second step: Face detection, face alignment, and image preprocessing are performed sequentially on the NIR images in the paired data; then, the images are input into the trained Vit-base with n prompts added to obtain NIR features (class tokens). These are then input into the NIR classification head to obtain NIR liveness detection results; finally, the second loss, L2 Loss, is calculated based on the NIR liveness detection results and NIR class labels, and simultaneously, the third loss, L3 Loss, is calculated based on the NIR features (class tokens) and RGB features (class tokens). Then, based on L2 loss and L3 loss, only the parameters of n prompts are adjusted to obtain a hybrid modality liveness detection model.

[0113] Further reference Figure 5 The diagram illustrates a flow 500 of an embodiment of a liveness detection method according to the present disclosure. The liveness detection method includes the following steps:

[0114] Step 501: Obtain the image to be detected in the first modality and / or the second modality.

[0115] In this embodiment, the liveness detection method can acquire images to be detected in a first modality and / or a second modality.

[0116] The images to be detected in the first mode and the second mode can be images of different modes. For example, the image to be detected in the first mode can be an RGB image, and the image to be detected in the second mode can be an NIR image.

[0117] Step 502: Input the image to be detected into the liveness detection model to obtain the liveness detection result of the image to be detected.

[0118] In this embodiment, the aforementioned execution entity can input the image to be detected into the liveness detection model to obtain the liveness detection result of the image. The liveness detection model can be used to detect whether a live object exists in the images of the first and second modalities, and utilizes... Figures 1-3 The training method provided in any of the embodiments is used, and will not be described again here.

[0119] The liveness detection method provided in this disclosure requires only a single hybrid-modal liveness detection model to simultaneously detect liveness in images of both the first and second modalities. Furthermore, liveness detection based on multimodal images reduces sensitivity to light, improves generalization against planar attacks such as those involving photos and videos, and thus enhances the model's application performance.

[0120] The liveness detection method provided in this disclosure can effectively prevent most common attacks, improving the generalization and accuracy of face liveness detection algorithms. It also enhances the defense effectiveness of face liveness detection algorithms against unknown attack sample methods.

[0121] The liveness detection method provided in this disclosure improves the performance of face liveness detection and can be applied to various applications in the field of face recognition, such as attendance, access control, security, and financial payments. It has wide applications in many current business scenarios. This method can improve the performance of face liveness detection technology and help many applications based on face liveness detection technology improve their effectiveness and user experience. It is beneficial for the further promotion of business projects.

[0122] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a liveness detection model training device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0123] like Figure 6 As shown, the liveness detection model training device 600 of this embodiment may include: an acquisition module 601, a first training module 602, an addition module 603, and a second training module 604. The acquisition module 601 is configured to acquire sample image pairs, wherein the sample image pairs consist of a first modality sample image and a second modality sample image containing the same face. The first training module 602 is configured to train a single-modality liveness detection model based on the first modality sample image, adjusting the parameters of the single-modality liveness detection model to obtain a first modality liveness detection model. The addition module 603 is configured to add a prompting module and a second modality classification head to the first modality liveness detection model. The second training module 604 is configured to train the first modality liveness detection model with the added prompting module and second modality classification head based on the second modality sample image, adjusting only the parameters of the prompting module to obtain a hybrid modality liveness detection model.

[0124] In this embodiment, the specific processing of the acquisition module 601, the first training module 602, the addition module 603, and the second training module 604 in the liveness detection model training device 600, and the resulting technical effects, can be found in the following references: Figure 1The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0125] In some optional implementations of this embodiment, the acquisition module 601 includes: an acquisition submodule configured to acquire an image pair, wherein the image pair consists of a first modal image and a second modal image containing the same face; and an alignment submodule configured to align the first modal image and the second modal image to obtain a sample image pair.

[0126] In some optional implementations of this embodiment, the alignment submodule includes: a face detection unit configured to perform face detection on the first modality image and the second modality image, and to crop the first modality face region image and the second modality face region image; a key point detection unit configured to perform face key point detection on the first modality face region image and the second modality face region image, and to obtain the coordinates of the first modality face key points and the second modality face key points; an alignment unit configured to perform face alignment on the faces in the first modality image and the second modality image according to the coordinates of the first modality face key points and the second modality face key points, and to obtain the first modality aligned image and the second modality aligned image; and a cropping unit configured to crop the face regions of the first modality aligned image and the second modality aligned image through an affine transformation, and to obtain a sample image pair.

[0127] In some optional implementations of this embodiment, the face detection unit is further configured to: input a first modal image into a pre-trained first modal face detection model to obtain the location of the first modal face region, and input a second modal image into a pre-trained second modal face detection model to obtain the location of the second modal face region; crop the first modal image based on the location of the first modal face region to obtain a first modal face region image, and crop the second modal image based on the location of the second modal face region to obtain a second modal face region image.

[0128] In some optional implementations of this embodiment, the key point detection unit is further configured to: input a first modal face region image into a pre-trained first modal face key point detection model to obtain the coordinates of the first modal face key points, and input a second modal face region image into a pre-trained second modal face key point detection model to obtain the coordinates of the second modal face key points.

[0129] In some optional implementations of this embodiment, the liveness detection model training device 600 further includes: a preprocessing module configured to preprocess the first modality sample image and the second modality sample image to obtain a first modality preprocessed image and a second modality preprocessed image; and a cutting module configured to cut the first modality preprocessed image and the second modality preprocessed image into a plurality of first modality image blocks and a plurality of second modality image blocks.

[0130] In some optional implementations of this embodiment, preprocessing includes at least one of the following: normalization processing and data augmentation processing. Data augmentation processing includes at least one of the following: random flipping, random scaling, and color perturbation.

[0131] In some optional implementations of this embodiment, the first training module 602 is further configured to: input multiple first modal image blocks into a single modal liveness detection model and output the first modal liveness detection result; calculate a first loss based on the first modal liveness detection result and the category label of the first modal sample image; and adjust the parameters of the single modal liveness detection model based on the first loss to obtain the first modal liveness detection model.

[0132] In some optional implementations of this embodiment, the second training module 604 includes: an output submodule configured to input multiple second modality image blocks into a first modality liveness detection model with added prompting module and second modality classification head, and output second modality liveness detection results; a calculation submodule configured to calculate a second loss based on the second modality liveness detection results and the category labels of the second modality sample images; and an adjustment submodule configured to adjust only the parameters of the prompting module based on the second loss to obtain a hybrid modality liveness detection model.

[0133] In some optional implementations of this embodiment, the single-modal liveness detection model also outputs a first modality class label feature, and the first modality liveness detection model with added prompting module and second modality classification head also outputs a second modality class label feature; and the adjustment submodule is further configured to: calculate a third loss based on the first modality class label feature and the second modality class label feature; and adjust only the parameters of the prompting module based on the second loss and the third loss to obtain a hybrid modality liveness detection model.

[0134] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a liveness detection device, which is similar to... Figure 5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0135] like Figure 7As shown, the liveness detection device 700 of this embodiment may include an acquisition module 801 and a detection module 802. The acquisition module 701 is configured to acquire a first modality and / or a second modality of the image to be detected; the detection module 702 is configured to input the image to be detected into a liveness detection model to obtain a liveness detection result for the image to be detected, wherein the liveness detection model employs... Figure 1 , Figure 2 and Figure 3 The training method described in any of the embodiments is used to obtain the training.

[0136] In this embodiment, the specific processing of the acquisition module 701 and the detection module 702 in the liveness detection device 700 and the resulting technical effects can be referred to respectively. Figure 5 The relevant descriptions of steps 501-502 in the corresponding embodiments will not be repeated here.

[0137] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0138] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0141] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the liveness detection model training method. For example, in some embodiments, the liveness detection model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the liveness detection model training method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the liveness detection model training method by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0148] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0149] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a live detection model, comprising: obtaining a sample image pair, the sample image pair being composed of a first modality sample image and a second modality sample image containing the same face; training a single modality live detection model based on the first modality sample image, adjusting parameters of the single modality live detection model to obtain a first modality live detection model; adding a prompt module and a second modality classification head to the first modality live detection model; training the first modality live detection model with the prompt module and the second modality classification head added based on the second modality sample image, adjusting parameters of the prompt module to obtain a mixed modality live detection model; preprocessing the second modality sample image to obtain a second modality preprocessed image; cutting the second modality preprocessed image into a plurality of second modality image blocks; the training of the first modality live detection model with the prompt module and the second modality classification head added based on the second modality sample image, and the adjustment of the parameters of the prompt module to obtain the mixed modality live detection model, comprising: inputting the plurality of second modality image blocks into the first modality live detection model with the prompt module and the second modality classification head added, and outputting a second modality live detection result; calculating a second loss based on the second modality live detection result and a class label of the second modality sample image; adjusting the parameters of the prompt module based on the second loss to obtain the mixed modality live detection model.

2. The method of claim 1, wherein, the obtaining of the sample image pair, comprising: obtaining an image pair, wherein the image pair is composed of a first modality image and a second modality image containing the same face; aligning the faces of the first modality image and the second modality image to obtain the sample image pair.

3. The method of claim 2, wherein, the aligning of the faces of the first modality image and the second modality image to obtain the sample image pair, comprising: detecting the faces of the first modality image and the second modality image to obtain a first modality face region image and a second modality face region image; detecting face key points of the first modality face region image and the second modality face region image to obtain first modality face key point coordinates and second modality face key point coordinates; aligning the faces of the first modality image and the second modality image according to the first modality face key point coordinates and the second modality face key point coordinates to obtain a first modality aligned image and a second modality aligned image; obtaining the sample image pair by cutting face regions of the first modality aligned image and the second modality aligned image through affine transformation.

4. The method of claim 3, wherein, the detecting of the faces of the first modality image and the second modality image to obtain the first modality face region image and the second modality face region image, comprising: inputting the first modality image into a pre-trained first modality face detection model to obtain a first modality face region position, and inputting the second modality image into a pre-trained second modality face detection model to obtain a second modality face region position; obtaining a first modality face region image by intercepting the first modality image based on the first modality face region position, and obtaining a second modality face region image by intercepting the second modality image based on the second modality face region position.

5. The method of claim 3, wherein, The face key point detection on the first modality face region image and the second modality face region image to obtain first modality face key point coordinates and second modality face key point coordinates comprises: inputting the first modality face region image into a pre-trained first modality face key point detection model to obtain the first modality face key point coordinates, and inputting the second modality face region image into a pre-trained second modality face key point detection model to obtain the second modality face key point coordinates.

6. The method of claim 1, wherein, The method further comprises: preprocessing the first modality sample image to obtain a first modality preprocessed image; cutting the first modality preprocessed image into a plurality of first modality image blocks.

7. The method of claim 6, wherein, The preprocessing comprises at least one of normalization processing and data enhancement processing, and the data enhancement processing comprises at least one of random flipping, random scaling, and color disturbance.

8. The method of claim 6, wherein, The training of the single-modality living body detection model based on the first modality sample image, adjustment of parameters of the single-modality living body detection model, and obtaining of a first modality living body detection model comprise: inputting the plurality of first modality image blocks into the single-modality living body detection model to output a first modality living body detection result; calculating a first loss based on the first modality living body detection result and a class label of the first modality sample image; adjusting the parameters of the single-modality living body detection model based on the first loss to obtain the first modality living body detection model.

9. The method of claim 8, wherein, The single-modality living body detection model further outputs first modality class label features, and a first modality living body detection model added with the prompt module and the second modality classification head further outputs second modality class label features. And The adjustment of the parameters of the prompt module based on the second loss and obtaining of a mixed-modality living body detection model comprise: calculating a third loss based on the first modality class label features and the second modality class label features; adjusting the parameters of the prompt module based on the second loss and the third loss to obtain the mixed-modality living body detection model.

10. A living body detection method, comprising: obtaining a to-be-detected image of a first modality and / or a second modality; inputting the to-be-detected image into a living body detection model to obtain a living body detection result of the to-be-detected image, wherein the living body detection model is trained by any one of the methods in claims 1-9.

11. A living body detection model training device, comprising: an acquisition module configured to acquire a sample image pair, the sample image pair being composed of a first modality sample image and a second modality sample image containing the same face; a first training module configured to train a single-modality living body detection model based on the first modality sample image, adjust parameters of the single-modality living body detection model, and obtain a first modality living body detection model. The adding module is configured to add a prompt module and a second modality classification head in the first modality live body detection model; The second training module is configured to train the first modality live body detection model to which the prompt module and the second modality classification head are added based on the second modality sample image pair, adjust the parameters of the prompt module, and obtain a mixed modality live body detection model; The first preprocessing module is configured to pre-process the second modality sample image to obtain a second modality pre-processed image; The first cutting module is configured to cut the second modality pre-processed image into a plurality of second modality image blocks; The second training module comprises: The output sub-module is configured to input the plurality of second modality image blocks into the first modality live body detection model to which the prompt module and the second modality classification head are added, and output a second modality live body detection result; The calculation sub-module is configured to calculate a second loss based on the second modality live body detection result and the class label of the second modality sample image; The adjustment sub-module is configured to adjust the parameters of the prompt module based on the second loss, and obtain a mixed modality live body detection model.

12. The apparatus of claim 11, wherein, The acquisition module comprises: The acquisition sub-module is configured to acquire an image pair, wherein the image pair comprises a first modality image and a second modality image containing the same face; The alignment sub-module is configured to perform face alignment on the first modality image and the second modality image to obtain the sample image pair.

13. The apparatus of claim 12, wherein, The alignment sub-module comprises: The face detection unit is configured to perform face detection on the first modality image and the second modality image, and intercept a first modality face region image and a second modality face region image; The key point detection unit is configured to perform face key point detection on the first modality face region image and the second modality face region image to obtain first modality face key point coordinates and second modality face key point coordinates; The alignment unit is configured to perform face alignment on the faces of the first modality image and the second modality image according to the first modality face key point coordinates and the second modality face key point coordinates to obtain a first modality aligned image and a second modality aligned image; The intercepting unit is configured to intercept face regions of the first modality aligned image and the second modality aligned image through affine transformation to obtain the sample image pair.

14. The apparatus of claim 13, wherein, The face detection unit is further configured to: input the first modality image into a pre-trained first modality face detection model to obtain a first modality face region position, and input the second modality image into a pre-trained second modality face detection model to obtain a second modality face region position; intercept the first modality image based on the first modality face region position to obtain the first modality face region image, and intercept the second modality image based on the second modality face region position to obtain the second modality face region image.

15. The apparatus of claim 13, wherein, The key point detection unit is further configured to: inputting the first modality face region image into a pre-trained first modality face key point detection model to obtain the first modality face key point coordinates, and inputting the second modality face region image into a pre-trained second modality face key point detection model to obtain the second modality face key point coordinates.

16. The apparatus of claim 11, wherein, The device further comprises: a second preprocessing module configured to pre-process the first modality sample image to obtain a first modality pre-processed image; a second cutting module configured to cut the first modality pre-processed image into a plurality of first modality image blocks.

17. The apparatus of claim 16, wherein, The preprocessing comprises at least one of normalization processing and data enhancement processing, and the data enhancement processing comprises at least one of random flipping, random scaling and color disturbance.

18. The apparatus of claim 16, wherein, The first training module is further configured to: input the plurality of first modality image blocks into the single modality living body detection model to output a first modality living body detection result; calculate a first loss based on the first modality living body detection result and a class label of the first modality sample image; adjust parameters of the single modality living body detection model based on the first loss to obtain the first modality living body detection model.

19. The apparatus of claim 18, wherein, The single modality living body detection model further outputs a first modality class label feature, and the first modality living body detection model added with the prompt module and the second modality classification head further outputs a second modality class label feature; and The adjusting submodule is further configured to: calculate a third loss based on the first modality class label feature and the second modality class label feature; adjust parameters of the prompt module based on the second loss and the third loss to obtain a mixed modality living body detection model. 20.A living body detection device, comprising: an acquisition module configured to acquire a to-be-detected image of a first modality and / or a second modality; a detection module configured to input the to-be-detected image into a living body detection model to obtain a living body detection result of the to-be-detected image, wherein the living body detection model is trained by the device of any one of claims 11-19. 21.An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-9 or execute the method of claim 10. 22.A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method of any one of claims 1-9 or execute the method of claim 10. 23.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-9 or implements the method of claim 10.

Images