Living body detection model training method, living body detection method, device and electronic equipment
By combining image mask reconstruction and weighted networks, the problems of low training efficiency and insufficient attack defense capabilities of face liveness detection models are solved, achieving efficient and accurate liveness detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2023-01-06
- Publication Date
- 2026-05-01
AI Technical Summary
In existing technologies, it is difficult to achieve optimal performance improvements in face liveness detection in terms of computational resources and time efficiency, and the defense capabilities against different types of attacks are insufficient.
A self-supervised pre-training method for face liveness detection based on image mask reconstruction is adopted. By selecting candidate image patches for masking according to image type, and combining them with a weighted network for pre-training and fine-tuning, the training efficiency and accuracy of the liveness detection model are improved.
It significantly improves the training efficiency and accuracy of the liveness detection model, enhances its defense capabilities against various attack types, and improves the algorithm's generalization and defense effectiveness.
Smart Images
Figure CN116052288B_ABST
Abstract
Description
Liveness detection model training methods, liveness detection methods, devices and electronic equipment Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically deep learning, image processing, and computer vision. Specifically, it relates to a liveness detection model training method, a liveness detection method, an apparatus, and an electronic device. Background Technology
[0002] With the rapid development of artificial intelligence (AI), AI has been widely applied in fields such as computer vision, speech recognition, natural language processing, deep learning, and big data processing.
[0003] Artificial intelligence-based image detection technology has been widely applied in various scenarios. Among them, AI-based face liveness detection technology, used to distinguish whether an image is taken of a real person, is a fundamental component of face recognition systems and ensures the security of these systems. Therefore, improving liveness detection performance is of paramount importance. Summary of the Invention
[0004] This disclosure provides a method for training a liveness detection model, a liveness detection method, an apparatus, and an electronic device.
[0005] According to one aspect of this disclosure, a method for training a liveness detection model is provided, comprising:
[0006] The original sample image is divided into multiple original image blocks;
[0007] Based on the image type of the original sample image, select the target image block to be masked from the original image block;
[0008] The target image patch is masked to obtain a masked sample image, and the masked sample image is input into a deep learning model for pre-training to obtain a pre-trained model for liveness detection.
[0009] The pre-trained model for liveness detection is fine-tuned to obtain the liveness detection model.
[0010] According to another aspect of this disclosure, a liveness detection method is provided, comprising:
[0011] Acquire the target image to be detected;
[0012] The target image is input into the liveness detection model to obtain the predicted liveness information of the target image; the liveness detection model is obtained by the liveness detection model training method disclosed in any embodiment of this disclosure.
[0013] According to another aspect of this disclosure, a liveness detection model training apparatus is provided, comprising:
[0014] The sample segmentation module is used to divide the original sample image into multiple original image blocks;
[0015] The target selection module is used to select the target image block to be masked from the original image block according to the image type of the original sample image;
[0016] The pre-training module is used to mask the target image patch to obtain a masked sample image, and input the masked sample image into the deep learning model for pre-training to obtain a pre-trained model for liveness detection.
[0017] The fine-tuning module is used to fine-tune the pre-trained model for liveness detection to obtain a liveness detection model.
[0018] According to another aspect of this disclosure, a liveness detection device is provided, comprising:
[0019] The target image module is used to acquire the target image to be detected;
[0020] The target liveness prediction module is used to input the target image into the liveness detection model to obtain the predicted liveness information of the target image; the liveness detection model is obtained by the liveness detection model training method disclosed in any embodiment of this disclosure.
[0021] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising:
[0022] At least one processor; and
[0023] A memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods provided in any embodiment of this disclosure.
[0025] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in any embodiment of this disclosure.
[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0028] Figure 1a is a flowchart of a liveness detection model training method according to an embodiment of the present disclosure;
[0029] Figure 1b is a schematic diagram of the principle of a pre-training stage provided according to an embodiment of the present disclosure;
[0030] Figure 2 is a flowchart of another liveness detection model training method provided according to an embodiment of the present disclosure;
[0031] Figure 3a is a flowchart of another liveness detection model training method provided according to an embodiment of the present disclosure;
[0032] Figure 3b is a schematic diagram illustrating the principle of a pre-training stage according to an embodiment of the present disclosure;
[0033] Figure 3c is a schematic diagram illustrating the principle of a fine-tuning stage according to an embodiment of the present disclosure;
[0034] Figure 4 is a flowchart of a liveness detection method provided according to an embodiment of the present disclosure;
[0035] Figure 5 is a schematic diagram of a liveness detection model training device provided according to an embodiment of the present disclosure;
[0036] Figure 6 is a schematic diagram of a liveness detection device according to an embodiment of the present disclosure;
[0037] Figure 7 is a block diagram of an electronic device used to implement the liveness detection model training method or liveness detection method according to the embodiments of the present disclosure. Detailed Implementation
[0038] Figure 1a is a flowchart of a liveness detection model training method according to an embodiment of the present disclosure. This method is applicable to training a liveness detection model. The method can be executed by a liveness detection model training device, which can be implemented in software and / or hardware and integrated into an electronic device. As shown in Figure 1a, the liveness detection model training method of this embodiment may include:
[0039] S101, divide the original sample image into multiple original image blocks;
[0040] S102, Select the target image block to be masked from the original image block according to the image type of the original sample image;
[0041] S103, mask the target image block to obtain a masked sample image, and input the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection;
[0042] S104, Fine-tune the pre-trained model for liveness detection to obtain a liveness detection model.
[0043] In this embodiment, a liveness dataset and an unlabeled dataset can be obtained. The original sample images in the liveness dataset have liveness annotation information, which can be the attack type corresponding to the original sample image. The attack type includes, but is not limited to, no attack (i.e., real person live image), electronic screen attack (i.e., video attack), paper attack (i.e., printed photo attack), head model attack, mask attack, ID photo attack, or facial feature occlusion attack, etc. The original sample images in the unlabeled dataset do not have liveness annotation information.
[0044] The training process for a liveness detection model can include a pre-training phase and a fine-tuning phase. In the pre-training phase, raw sample data from both the liveness dataset (referred to as labeled raw sample images) and the unlabeled dataset (referred to as unlabeled raw sample images) are used to pre-train the deep learning model, resulting in a pre-trained liveness detection model. In the fine-tuning phase, labeled raw sample images are used to fine-tune the pre-trained liveness detection model, resulting in a liveness detection model. Fine-tuning the liveness detection model saves significant computational resources and time compared to training a model from scratch, thus improving the training efficiency of the liveness detection model.
[0045] Figure 1b is a schematic diagram illustrating the principle of a pre-training stage according to an embodiment of this disclosure. Referring to Figure 1b, in the pre-training stage, a target image block to be masked can be selected from the original image blocks according to the image type of the original sample image, and the target image block is masked to obtain a masked sample image. In cases of different image types, especially different attack types, target image blocks to be masked can be selected from different regions in the original sample image. This allows the model to learn the classification features corresponding to each image type during self-supervised pre-training, enabling the pre-trained liveness detection model to focus on the classification features corresponding to different image types, greatly improving the efficiency of liveness pre-training and increasing the accuracy of subsequent liveness detection.
[0046] During the pre-training phase, face detection can be performed on the original sample images to extract the face regions. Specifically, a face detection model can be used to detect faces in the original sample images to obtain the approximate location of the faces. Based on the approximate location of the faces, a facial landmark detection model can be used to detect facial landmarks and obtain their coordinate values. There are 72 pre-set facial landmarks. Face alignment is then performed based on the coordinate values of the facial landmarks to obtain the face region information of the original sample. The face alignment process can be as follows: Based on the coordinates of 72 facial key points, obtain the maximum and minimum values of the horizontal and vertical coordinates. Combine these values to determine the face bounding box. Expand the face bounding box by three parts to extract the face image, and adjust the face image to a fixed size, such as 256×256. Perform image normalization on the face image, for example, subtract 128 from the pixel value of each pixel in the face image and then divide by 256, so that the pixel value of each pixel is between [-0.5, 0.5]. Perform random data augmentation on the normalized sample face region image, and divide the sample face region image into 16×16 original image patches.
[0047] Referring to Figure 1b, a target image patch to be masked can be selected from each original image patch in the original sample image according to the image type of the original sample image, and the target image patch can be masked to obtain a masked sample image. The masked sample image is input into a deep learning model for self-supervised pre-training to obtain a pre-trained model for liveness detection. This embodiment does not specifically limit the network structure of the deep learning model; for example, a VisionTransformer (ViT) structure can be used.
[0048] During the fine-tuning stage, a target image block can be selected from the original image blocks of the original sample image according to the image type of the original sample image, and the target image block is masked to obtain a masked sample image. The masked sample image is then input into the pre-trained model for liveness detection to obtain the predicted liveness information of the original sample image. Based on the labeled liveness information and predicted liveness information of the original sample image, the pre-trained model for liveness detection is fine-tuned to obtain the liveness detection model.
[0049] The technical solution provided in this disclosure, in the pre-training stage, selects the target image block to be masked from the original image block of the original sample image according to the image type of the original sample image, and masks the target image block to obtain the masked sample image. Furthermore, it performs self-supervised pre-training based on the masked sample image, enabling the pre-trained model for liveness detection to pay attention to the classification features corresponding to different image types, which greatly improves the efficiency of liveness pre-training and the accuracy of subsequent liveness detection.
[0050] Figure 2 is a flowchart of another liveness detection model training method provided according to an embodiment of the present disclosure. Referring to Figure 2, the liveness detection model training method of this embodiment may include:
[0051] S201, divide the original sample image into multiple original image blocks;
[0052] S202, Based on the image type of the original sample image, select candidate image blocks that can be masked from the original image blocks;
[0053] S203, randomly select the target image block to be masked from the candidate image blocks;
[0054] S204, Mask the target image block to obtain a masked sample image, and input the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection;
[0055] S205, Fine-tune the pre-trained model for liveness detection to obtain a liveness detection model.
[0056] In this embodiment of the disclosure, the association between image types and candidate image blocks can be pre-constructed based on the attack characteristics of each image type, serving as prior knowledge of the image types. During the pre-training stage, based on the prior knowledge of image types, candidate image blocks that can be masked can be selected from the original image blocks according to the image type of the original sample image, and a preset proportion of candidate image blocks can be randomly selected as target image blocks to be masked. The preset proportion is an empirical value, such as 0.2.
[0057] Prior knowledge differs across image types, especially when considering different attack types, leading to different candidate image patch locations associated with each image type. For example, in the case of no attack or no label, all original image patches in the original sample image can serve as candidate image patches. In the case of head model attacks or mask attacks, the original image patch corresponding to the face region in the original sample image can be used as a candidate image patch, while the original image patch corresponding to the background region (excluding the face region) cannot be used. In the case of electronic screen attacks or paper attacks, the original image patch corresponding to the background region in the original sample image can be used as a candidate image patch, while the original image patch corresponding to the face region cannot be used. By leveraging prior knowledge based on image type, candidate image patches are selected from the original image patches of the original sample image according to its image type. This allows for better learning of the classification features corresponding to each image type during the pre-training stage, further improving the efficiency of liveness pre-training and the accuracy of subsequent liveness detection.
[0058] In one optional implementation, selecting maskable candidate image blocks from the original image blocks according to the image type of the original sample image includes: if the image type of the original sample image is a first attack type, selecting the original image block corresponding to the face region in the original sample image as a candidate image block; the first attack type is used to attack the face region; if the image type of the original sample image is a second attack type, selecting the original image block corresponding to the background region in the original sample image other than the face region as a candidate image block; the second attack type is used to attack the background region.
[0059] In this embodiment of the disclosure, the image type that attacks the face region in the image can be identified as the first attack type, such as head model attack, mask attack, or facial feature occlusion attack; the image type that attacks the background region in the image can be identified as the second attack type, such as video attack, paper attack, or ID photo attack.
[0060] Specifically, when the original sample image belongs to the first attack type, the original image block corresponding to the face region can be used as a candidate image block. Taking an original sample image with 16×16 original image blocks as an example, the candidate image block numbers could be 86-91, 102-107, 118-123, 134-139, and 150-155, for a total of 36 candidate image blocks. When the original sample image belongs to the second attack type, the original image block corresponding to the background region can be used as a candidate image block. Again, taking an original sample image with 16×16 original image blocks as an example, all original image blocks except those numbered 86-91, 102-107, 118-123, 134-139, and 150-155 are candidate image blocks.
[0061] For original sample images targeting facial regions, candidate image patches corresponding to the facial regions are used. This allows the reconstructing of the attacked facial regions during pre-training to learn the attack features at the facial regions. Similarly, for original sample images targeting background regions, candidate image patches corresponding to the background regions are used. This allows the reconstructing of the attacked background regions during pre-training to learn the attack features at the background regions. In other words, by selecting candidate image patches from the attack regions corresponding to the image type, the pre-training process can learn the attack features of the attack regions, thereby learning better classification features, further improving the efficiency of liveness pre-training, and the accuracy of subsequent liveness detection.
[0062] The technical solution provided in this disclosure proposes a self-supervised pre-training method for face liveness detection based on image masking reconstruction. Based on prior knowledge that the focus positions of different image types are inconsistent, candidate image blocks that can be masked are selected from each original image block, and target image blocks to be masked are randomly selected from the candidate image blocks. During the pre-training process, the attack features of different attack types can be considered, thereby further improving the efficiency of liveness pre-training and the accuracy of subsequent liveness detection.
[0063] Figure 3a is a flowchart of another liveness detection model training method provided according to an embodiment of the present disclosure. This embodiment is an optional solution proposed based on the above embodiments. Referring to Figure 3a, the liveness detection model training method of this embodiment may include:
[0064] S301, divide the original sample image into multiple original image blocks;
[0065] S302, Select the target image block to be masked from the original image block according to the image type of the original sample image;
[0066] S303, Mask the target image block to obtain a masked sample image, and input the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection;
[0067] S304, Determine the labeling weights of the original image patches in the original sample image based on the image type of the original sample image;
[0068] S305, Input the original image patch from the original sample image into the weight network to obtain the predicted weight of the original image patch;
[0069] S306, The weight network is pre-trained based on the predicted weights of the original image patches and the labeled weights of the original image patches;
[0070] S307, Fine-tune the pre-trained model and pre-trained weight network for liveness detection to obtain the liveness detection model.
[0071] Figure 3b is a schematic diagram of the pre-training stage according to an embodiment of the present disclosure. Referring to Figure 3b, the liveness detection model further includes a weight network. During the pre-training stage, the annotation weights of each original image patch in the original sample image are determined based on the image type of the original sample image. For example, when the image type is a first attack type, the annotation weight of the original image patch in the face region can be a first value, and the annotation weight of the original image patch in the background region can be a second value; when the image type is a second attack type, the annotation weight of the original image patch in the face region can be the second value, and the annotation weight of the original image patch in the background region can be the first value, where the first value is greater than the second value; when the image type is no attack or no annotation, the annotation weights of each original image patch can be equal, for example, all can be the first value. The first value is greater than the second value, for example, the first value is 1, and the second value is 0.5. By setting annotation weights for the original image patches according to the image type of the original sample image, the weight network can focus more on the features of the attack region during the pre-training stage, thereby further improving the accuracy of liveness detection.
[0072] In this embodiment of the disclosure, each original image block in the original sample image can be input into the weight network to obtain the predicted weight of each original image block. Based on the predicted weight and labeled weight of each original image block, the weight network is pre-trained so that the predicted weight and labeled weight of each original image block continuously approach each other during the pre-training stage.
[0073] In one optional implementation, the step of inputting the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection includes: inputting the masked sample image into the deep learning model to reconstruct the target image patch to obtain a reconstructed image patch; constructing a first loss function based on the target image patch and the reconstructed image patch, and using the first loss function to pre-train the deep learning model to obtain a pre-trained model for liveness detection; the step of pre-training the weight network based on the predicted weights and labeled weights of the original image patch includes: constructing a second loss function based on the predicted weights and labeled weights of the original image patch, and using the second loss function to pre-train the weight network.
[0074] Referring to Figure 3b, during the pre-training phase, the target image patch in the masked sample image can be reconstructed using a deep learning model to obtain the reconstructed image patch. A first loss function L1 is constructed based on the target image patch and the corresponding reconstructed image patch, and the network parameters in the deep learning model are updated using the first loss function L1. Furthermore, a second loss function L2 is constructed based on the predicted weights and labeled weights of the original image patches, and the network parameters in the weight network are updated using the second loss function L2. This ensures that as the target image patch and the corresponding reconstructed image patch become increasingly similar, the predicted weights and labeled weights of each original image patch also become increasingly similar. This enables the subsequent liveness detection model to learn the differential features of different attack types, thereby defending against multiple attack types and improving the accuracy, generalization, and robustness of subsequent liveness detection.
[0075] In one optional implementation, fine-tuning the pre-trained model for liveness detection to obtain a liveness detection model includes: acquiring an original sample image and liveness annotation information labeled on the original sample image; inputting original image blocks from the original sample image into a pre-trained weight network to obtain prediction weights for the original image blocks; weighting the original image blocks according to their prediction weights and inputting the weighted result into the pre-trained model for liveness detection to obtain liveness prediction information for the original sample image; and fine-tuning the pre-trained weight network and the pre-trained model for liveness detection based on the liveness annotation information and the liveness prediction information of the original sample image to obtain a liveness detection model.
[0076] Figure 3c is a schematic diagram of the principle of a fine-tuning stage according to an embodiment of the present disclosure. Referring to Figure 3c, in the fine-tuning stage, the original sample image is a labeled sample image. The liveness labeling information of the original sample image can be the attack type corresponding to the original sample image. The attack type includes, but is not limited to, no attack, electronic screen attack, paper attack, head model attack, mask attack, ID photo attack, or facial feature occlusion attack.
[0077] In the fine-tuning phase, the original sample image is segmented into individual image patches. Each image patch is then input into a pre-trained weight network to obtain its predicted weight. These weighted patches are then weighted to obtain a weighted result, which is input into a pre-trained liveness detection model to obtain liveness prediction information for the original sample image. This liveness prediction information is compared with the labeled prediction information, and a third loss function is constructed based on the comparison results. This third loss function is then used to fine-tune the pre-trained weight network and the pre-trained liveness detection model, resulting in the liveness detection model. In other words, the liveness detection model includes both a deep learning network and a weight network. By introducing a weight network into the liveness detection model, it is possible to better learn the attack characteristics of various attack types, thereby learning the differential characteristics of multiple complex attacks and further improving the performance of liveness detection.
[0078] The technical solution provided in this disclosure, by introducing a weighted network into the liveness detection model, can better learn the attack characteristics of various attack types, thereby learning the differential characteristics of various complex attacks and further improving the performance of liveness detection.
[0079] Figure 4 is a flowchart of a liveness detection method according to an embodiment of the present disclosure. This method is applicable to situations involving liveness detection of images. The method can be executed by a liveness detection device, which can be implemented in software and / or hardware and integrated into an electronic device. As shown in Figure 4, the liveness detection method of this embodiment may include:
[0080] S401, acquire the target image to be detected;
[0081] S402, the target image is input into the liveness detection model to obtain the predicted liveness information of the target image.
[0082] The liveness detection model is obtained using the liveness detection model training method disclosed in any embodiment of this disclosure, and the liveness detection model can use ViT as the base network. During the pre-training stage of the liveness detection model, candidate image blocks that can be masked are selected from each original image block of the original sample image according to the image type of the original sample image, and a target image block to be masked is randomly selected from the candidate image blocks. A masked sample image is obtained by masking the target image block in the original sample image, and the target image block in the masked sample image is reconstructed using the ViT network. Based on the ViT network structure and the prior knowledge that the positions of image blocks of interest differ for different attack types, a self-supervised pre-training method for face liveness detection based on image mask reconstruction is realized. Compared to randomly masking each original image block, this method can better focus on the classification features of different image types, especially the attack features of different attack types, greatly improving the efficiency of liveness pre-training, as well as the accuracy and generalization of the liveness detection algorithm.
[0083] In the fine-tuning phase, the pre-trained liveness detection model is fine-tuned to obtain the liveness detection model. In the prediction phase, the target image is segmented to obtain original image patches, and these original image patches are input into the liveness detection model, for example, into a ViT network, to obtain liveness prediction information for the target image.
[0084] The technical solution provided in this disclosure can learn attack features of different attack types during the pre-training stage of the liveness detection model. Therefore, during the prediction stage of the liveness detection model, it can accurately identify different attack types, effectively defend against multiple attack types, accelerate network training convergence speed, improve the generalization and accuracy of the face liveness detection algorithm, and enhance the defense effect of the face liveness detection algorithm against unknown attack sample methods. It can also improve the performance of face liveness detection technology and help many applications based on face liveness detection technology improve their effectiveness and user experience.
[0085] In one optional implementation, the step of inputting the target image into a liveness detection model to obtain predicted liveness information of the target image includes: dividing the target image into original image blocks; inputting the original image blocks into a weighted network to obtain predicted weights for the original image blocks; weighting the original image blocks in the target image according to their predicted weights, and inputting the weighted result into the liveness detection model to obtain predicted liveness information of the target image.
[0086] In a liveness detection model that includes a ViT network and a weight network, during the prediction phase, each original image patch in the target image is input into the weight network to obtain the predicted weight of each original image patch. Based on the predicted weights of each original image patch, the original image patches in the target image are weighted to obtain a weighted result, which is then input into the ViT network in the liveness detection model to obtain the liveness prediction information for the target image. By introducing a weight network into the liveness detection model, the differential features of various complex attacks can be better detected, further improving the performance of liveness detection.
[0087] Figure 5 is a schematic diagram of a liveness detection model training device according to an embodiment of the present disclosure. This embodiment is applicable to the training of a liveness detection model. The device can be implemented in software and / or hardware. As shown in Figure 5, the liveness detection model training device 500 of this embodiment may include:
[0088] The sample segmentation module 510 is used to divide the original sample image into multiple original image blocks;
[0089] The target selection module 520 is used to select a target image block to be masked from the original image block according to the image type of the original sample image;
[0090] The pre-training module 530 is used to mask the target image block to obtain a masked sample image, and input the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection.
[0091] The fine-tuning module 540 is used to fine-tune the pre-trained model for liveness detection to obtain a liveness detection model.
[0092] In one alternative implementation, the target selection module 520 includes:
[0093] Candidate block unit, used to select maskable candidate image blocks from the original image blocks according to the image type of the original sample image;
[0094] The target block unit is used to randomly select a target image block to be masked from the candidate image blocks.
[0095] In one alternative implementation, the candidate block unit includes:
[0096] The first candidate subunit is used to select the original image block corresponding to the face region in the original sample image as a candidate image block when the image type of the original sample image is the first attack type; the first attack type is used to attack the face region.
[0097] The second candidate subunit is used to select the original image block corresponding to the background region (excluding the face region) in the original sample image as a candidate image block when the image type of the original sample image is the second attack type; the second attack type is used to attack the background region.
[0098] In one optional embodiment, the above-mentioned liveness detection model training device 500 further includes a weight network module, the weight network module comprising:
[0099] The label weight unit is used to determine the label weight of the original image patch in the original sample image based on the image type of the original sample image;
[0100] The prediction weight unit is used to input the original image patch from the original sample image into the weight network to obtain the predicted weight of the original image patch;
[0101] The weight pre-training unit is used to pre-train the weight network based on the predicted weights of the original image patches and the labeled weights of the original image patches.
[0102] In one alternative implementation, the pre-training module 530 includes:
[0103] A reconstruction block unit is used to input the masked sample image into a deep learning model to reconstruct the target image block and obtain a reconstructed image block.
[0104] The liveness pre-training unit is used to construct a first loss function based on the target image patch and the reconstructed image patch, and to pre-train the deep learning model using the first loss function to obtain a pre-trained model for liveness detection.
[0105] The weight pre-training unit is specifically used to construct a second loss function based on the predicted weights and labeled weights of the original image patch, and to pre-train the weight network using the second loss function.
[0106] In one alternative implementation, the fine-tuning module 540 includes:
[0107] The information acquisition unit is used to acquire the original sample image and the liveness annotation information annotated on the original sample image;
[0108] The weight prediction unit is used to input the original image patch in the original sample image into the pre-trained weight network to obtain the predicted weight of the original image patch;
[0109] The prediction information unit is used to weight the original image patch according to the prediction weight of the original image patch, and input the weighting result into the pre-trained model of liveness detection to obtain the liveness prediction information of the original sample image.
[0110] The fine-tuning unit is used to fine-tune the pre-trained weight network and the pre-trained liveness detection model based on the liveness annotation information and liveness prediction information of the original sample image, so as to obtain the liveness detection model.
[0111] The technical solution of this disclosure, based on the ViT network structure, proposes a self-supervised pre-training method for face liveness detection based on image mask reconstruction. By selecting the target image block to be masked according to the image type of the original sample image, it can learn the classification features of various image types, which greatly improves the efficiency of liveness pre-training, as well as the accuracy and generalization of the liveness detection algorithm.
[0112] Figure 6 is a schematic diagram of a liveness detection device according to an embodiment of the present disclosure. This embodiment is applicable to situations involving liveness detection. The device can be implemented using software and / or hardware. As shown in Figure 6, the liveness detection device 600 of this embodiment may include:
[0113] Target image module 610 is used to acquire the target image to be detected;
[0114] The target liveness prediction module 620 is used to input the target image into the liveness detection model to obtain the predicted liveness information of the target image; the liveness detection model is obtained by the liveness detection model training method provided in any embodiment of this disclosure.
[0115] In one optional implementation, the target liveness prediction module 620 includes:
[0116] An image segmentation unit is used to segment the target image to obtain original image blocks in the target image;
[0117] The weight prediction unit is used to input the original image blocks in the target image into the weight network to obtain the predicted weights of the original image blocks in the target image.
[0118] The target liveness prediction unit is used to weight the original image blocks in the target image according to the prediction weights of the original image blocks in the target image, and input the weighting result into the liveness detection model to obtain liveness prediction information of the target image.
[0119] The technical solution of this disclosure, based on the ViT network structure, proposes a self-supervised pre-training method for face liveness detection based on image mask reconstruction. By selecting the target image block to be masked according to the image type of the original sample image, it can learn the classification features of various image types, which greatly improves the efficiency of liveness pre-training, as well as the accuracy and generalization of the liveness detection algorithm.
[0120] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0121] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0122] Figure 7 is a block diagram of an electronic device used to implement the liveness detection model training method or liveness detection method according to the embodiments of the present disclosure.
[0123] Figure 7 illustrates a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0124] As shown in Figure 7, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 can also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0125] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0126] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as a liveness detection model training method or a liveness detection method. For example, in some embodiments, the liveness detection model training method or the liveness detection method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the liveness detection model training method or the liveness detection method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform a liveness detection model training method or a liveness detection method.
[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0128] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0129] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0132] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0133] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.
[0134] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.
[0135] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for training a liveness detection model, comprising: The original sample image is divided into multiple original image blocks; Based on the image type of the original sample image, select maskable candidate image blocks from the original image blocks; Randomly select a target image block to be masked from the candidate image blocks; mask the target image block to obtain a masked sample image, and input the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection; fine-tune the pre-trained model for liveness detection to obtain a liveness detection model; wherein, the image type includes an attack type, and the step of selecting a maskable candidate image block from the original image blocks according to the image type of the original sample image includes: when the image type of the original sample image is a first attack type, selecting the original image block corresponding to the face region in the original sample image as a candidate image block; the first attack type is used to attack the face region; when the image type of the original sample image is a second attack type, selecting the original image block corresponding to the background region other than the face region in the original sample image as a candidate image block; the second attack type is used to attack the background region.
2. The method according to claim 1, further comprising: Based on the image type of the original sample image, determine the label weights of the original image patches in the original sample image; The original image patches from the original sample image are input into the weight network to obtain the predicted weights of the original image patches; the weight network is pre-trained based on the predicted weights and the labeled weights of the original image patches.
3. The method according to claim 2, wherein, The step of inputting the masked sample image into a deep learning model for pre-training to obtain a pre-trained model for liveness detection includes: inputting the masked sample image into the deep learning model to reconstruct the target image patch to obtain a reconstructed image patch; constructing a first loss function based on the target image patch and the reconstructed image patch, and using the first loss function to pre-train the deep learning model to obtain a pre-trained model for liveness detection; the step of pre-training the weight network based on the predicted weights and labeled weights of the original image patch includes: constructing a second loss function based on the predicted weights and labeled weights of the original image patch, and using the second loss function to pre-train the weight network.
4. The method according to claim 2, wherein, The step of fine-tuning the pre-trained model for liveness detection to obtain a liveness detection model includes: acquiring an original sample image and liveness annotation information labeled on the original sample image; inputting original image patches from the original sample image into a pre-trained weight network to obtain prediction weights for the original image patches; weighting the original image patches according to their prediction weights and inputting the weighted result into the pre-trained model for liveness detection to obtain liveness prediction information for the original sample image; and fine-tuning the pre-trained weight network and the pre-trained model for liveness detection based on the liveness annotation information and the liveness prediction information of the original sample image to obtain a liveness detection model.
5. A method for detecting liveness, comprising: Acquire the target image to be detected; input the target image into the liveness detection model to obtain the predicted liveness information of the target image; The liveness detection model is obtained using the liveness detection model training method as described in any one of claims 1-4.
6. The method according to claim 5, wherein, The step of inputting the target image into a liveness detection model to obtain predicted liveness information of the target image includes: dividing the target image into original image blocks; inputting the original image blocks into a weighted network to obtain predicted weights for the original image blocks; weighting the original image blocks in the target image according to their predicted weights, and inputting the weighted result into the liveness detection model to obtain predicted liveness information of the target image.
7. A liveness detection model training device, comprising: The sample segmentation module is used to divide the original sample image into multiple original image blocks; The target selection module is used to select a target image block to be masked from the original image block according to the image type of the original sample image; the pre-training module is used to mask the target image block to obtain a masked sample image, and input the masked sample image into the deep learning model for pre-training to obtain a pre-trained model for liveness detection. A fine-tuning module is used to fine-tune the pre-trained model for liveness detection to obtain a liveness detection model. The target selection module includes: a candidate block unit, used to select maskable candidate image blocks from the original image blocks based on the image type of the original sample image; and a target block unit, used to randomly select target image blocks to be masked from the candidate image blocks. The image type includes an attack type, and the candidate block unit includes: a first candidate subunit, used to select the original image block corresponding to the face region in the original sample image as a candidate image block when the image type of the original sample image is a first attack type; the first attack type is used to attack the face region; and a second candidate subunit, used to select the original image block corresponding to the background region in the original sample image (excluding the face region) as a candidate image block when the image type of the original sample image is a second attack type; the second attack type is used to attack the background region.
8. The apparatus according to claim 7, further comprising a weighted network module, the weighted network module comprising: The label weight unit is used to determine the label weight of the original image patch in the original sample image based on the image type of the original sample image; The prediction weight unit is used to input the original image patch from the original sample image into the weight network to obtain the predicted weight of the original image patch; The weight pre-training unit is used to pre-train the weight network based on the predicted weights of the original image patches and the labeled weights of the original image patches.
9. The apparatus according to claim 8, wherein, The pre-training module includes: a reconstruction block unit, used to input the masked sample image into a deep learning model to reconstruct the target image block to obtain a reconstructed image block; a liveness pre-training unit, used to construct a first loss function based on the target image block and the reconstructed image block, and use the first loss function to pre-train the deep learning model to obtain a pre-trained model for liveness detection; and a weight pre-training unit, specifically used to construct a second loss function based on the predicted weights of the original image block and the labeled weights of the original image block, and use the second loss function to pre-train the weight network.
10. The apparatus according to claim 8, wherein, The fine-tuning module includes: an information acquisition unit for acquiring the original sample image and the liveness annotation information labeled on the original sample image; a weight prediction unit for inputting the original image patch in the original sample image into a pre-trained weight network to obtain the predicted weight of the original image patch; a prediction information unit for weighting the original image patch according to the predicted weight of the original image patch and inputting the weighting result into a pre-trained liveness detection model to obtain liveness prediction information of the original sample image; and a fine-tuning unit for fine-tuning the pre-trained weight network and the pre-trained liveness detection model according to the liveness annotation information and the liveness prediction information of the original sample image to obtain a liveness detection model.
11. A liveness detection device, comprising: The target image module is used to acquire the target image to be detected; The target liveness prediction module is used to input the target image into the liveness detection model to obtain the predicted liveness information of the target image; The liveness detection model is obtained using the liveness detection model training method as described in any one of claims 1-4.
12. The apparatus according to claim 11, wherein, The target liveness prediction module includes: an image segmentation unit for segmenting the target image to obtain original image blocks in the target image; a weight prediction unit for inputting the original image blocks in the target image into a weight network to obtain the prediction weights of the original image blocks in the target image; and a target liveness prediction unit for weighting the original image blocks in the target image according to the prediction weights of the original image blocks in the target image, and inputting the weighting result into the liveness detection model to obtain liveness prediction information of the target image.
13. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.
Citation Information
Patent Citations
Multi-task full-reference image quality evaluation method based on gated convolutional neural network
CN108596902A
Model training method, incomplete face image recognition method, incomplete face image reconstruction method, equipment and medium
CN115035567A
Target detection model training method, target detection method and device
CN115170887A