A method and apparatus for liveness detection and model training

Abstract

The specification discloses a living body detection and model training method and device. Local images with different scales are determined according to a user image, the user image and each local image are input into different feature extraction networks of a living body detection model respectively, global features of the user image and local features of each local image are determined, the global features and the local features are spliced to obtain fusion features, and the fusion features are input into a regression network of the living body detection model to obtain a living body detection result. The living body detection model is trained according to a classification loss determined based on a contrast loss based on the fusion features and a probability corresponding to a label based on a training sample. The fusion features and the probability corresponding to the label are both obtained by inputting the user image as the training sample and corresponding local images into the living body detection model. Global features and local features with different scales can be extracted, the features are extracted comprehensively and sufficiently, and the output of the model is more accurate.

Description

Technical Field

[0001] This specification relates to the field of liveness detection technology, and in particular to a liveness detection and model training method and apparatus. Background Technology

[0002] While facial recognition technology has made life more convenient, it also brings certain risks. For example, if someone maliciously uses a user's photo or a facial mask to successfully perform facial recognition, it could cause harm to the user. Therefore, facial recognition technology is usually used in conjunction with liveness detection technology.

[0003] Liveness detection technology is used to identify whether a face in front of a camera is a real, living face or a non-living face (such as a printed photo, a mask, or a screen photo). Current liveness detection technologies include cooperative and non-cooperative methods. Non-cooperative methods do not require the user to cooperate in facial recognition, thus being more efficient.

[0004] Current non-cooperative liveness detection methods obtain liveness detection results (live or non-live) by acquiring user images and inputting them into a trained model. Because the model's feature extraction is not comprehensive enough, the liveness detection results obtained by current methods are not accurate enough. Summary of the Invention

[0005] This specification provides a method and apparatus for liveness detection and model training, which partially solves the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a liveness detection method. The pre-trained liveness detection model includes a regression network and several feature extraction networks. The method includes:

[0008] Acquire a user image, and determine local images of different scales based on the user image;

[0009] The user image and each local image are input into different feature extraction networks to determine the global features corresponding to the user image and the local features corresponding to each local image.

[0010] The global features and local features are concatenated to obtain the fused features;

[0011] The fused features are input into the regression network to obtain the liveness detection results;

[0012] The liveness detection model is trained using a contrast loss determined by fusion features and a classification loss determined by the probability corresponding to the label of the training sample. The fusion features and the probability corresponding to the label are obtained by inputting the user image as the training sample and its corresponding local image into the liveness detection model.

[0013] This specification provides a model training method. The liveness detection model includes a regression network and several feature extraction networks. The method includes:

[0014] Each user's image is used as a training sample, and whether each training sample is a live person is used as a label.

[0015] For each training sample, determine the local maps of different scales corresponding to that training sample;

[0016] The training sample and each local map are input into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map.

[0017] The global features and local features are concatenated to obtain fused features, and the fused features are input into the regression network to obtain the probability that the training sample belongs to a living or non-living organism.

[0018] The classification loss is determined based on the probability corresponding to the label of each training sample, and the contrast loss is determined based on the fusion feature corresponding to each training sample, so as to determine the total loss.

[0019] The liveness detection model is trained with the goal of minimizing the total loss.

[0020] Optionally, for each training sample, determine local maps of different scales corresponding to that training sample, specifically including:

[0021] For each training sample, the facial region contained in the training sample is determined by the object detection algorithm;

[0022] Determine the facial image based on the obtained facial region;

[0023] Determine the key point regions in the facial image, and from there identify key point regions belonging to a preset type;

[0024] The facial image and the key point region of the preset type are used as local images of different scales corresponding to the training sample.

[0025] Optionally, a facial image is determined based on the obtained facial region, specifically including:

[0026] When there are multiple facial regions, the facial image corresponding to the training sample is determined based on the facial region with the largest area.

[0027] When there is only one facial region, the facial image corresponding to the training sample is determined based on the determined facial region.

[0028] Optionally, key point regions in the facial image are determined to identify key point regions belonging to a preset type, specifically including:

[0029] Facial key point localization is performed on the facial image to determine the key point regions and the type of each key point region in the facial image.

[0030] Based on the type of each key point region, the key point region of the eye is determined from each key point region as the key point region belonging to the preset type.

[0031] Optionally, the contrastive loss is determined based on the fusion features corresponding to each training sample, specifically including:

[0032] Determine the pairwise similarity between the fusion features of each training sample, and use it as the second similarity.

[0033] For each training sample, the similarity between the training sample and the fusion features of other training samples with the same label as the training sample is determined from each second similarity, and this is used as the first similarity.

[0034] Based on the determined first similarity and second similarity scores, the loss corresponding to the training sample is determined.

[0035] The contrast loss is determined based on the loss corresponding to each training sample.

[0036] Optionally, based on the determined first similarity and second similarity, the loss corresponding to the training sample is determined, specifically including:

[0037] Sum the second similarity scores to obtain the comprehensive similarity score.

[0038] For each first similarity, the similarity ratio corresponding to that first similarity is determined based on the ratio of that first similarity to the comprehensive similarity.

[0039] The loss corresponding to the training sample is determined based on the similarity ratio corresponding to each first similarity of the training sample.

[0040] This specification provides a liveness detection device. The pre-trained liveness detection model includes a regression network and several feature extraction networks. The device includes:

[0041] The acquisition module is used to acquire user images and determine local images of different scales based on the user images.

[0042] The first feature extraction module is used to input the user image and each local image into different feature extraction networks to determine the global features corresponding to the user image and the local features corresponding to each local image.

[0043] The fusion module is used to concatenate the global features and the local features to obtain the fused features;

[0044] The detection module is used to input the fused features into the regression network to obtain the liveness detection result;

[0045] The liveness detection model is trained using a contrast loss determined by fusion features and a classification loss determined by the probability corresponding to the label of the training sample. The fusion features and the probability corresponding to the label are obtained by inputting the user image as the training sample and its corresponding local image into the liveness detection model.

[0046] This specification provides a model training device. The liveness detection model includes a regression network and several feature extraction networks. The device includes:

[0047] The sample determination module is used to acquire images of each user as training samples and label whether each training sample is a live person.

[0048] The local map determination module is used to determine the local maps of different scales corresponding to each training sample.

[0049] The second feature extraction module is used to input the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map.

[0050] The regression module is used to concatenate the global features and local features to obtain fused features, and input the fused features into the regression network to obtain the probability that the training sample belongs to a live or non-live organism.

[0051] The loss determination module is used to determine the classification loss based on the probability corresponding to the label of each training sample, and to determine the contrast loss based on the fusion features corresponding to each training sample, so as to determine the total loss.

[0052] The training module is used to train the liveness detection model with the goal of minimizing the total loss.

[0053] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described liveness detection and model training methods.

[0054] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described liveness detection and model training methods.

[0055] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0056] In the liveness detection and model training method provided in this specification, local maps of different scales are determined based on the user image. The user image and each local map are then input into different feature extraction networks within the liveness detection model to determine the global features of the user image and the local features of each local map. These global and local features are then concatenated to obtain fused features. Inputting these fused features into the regression network of the liveness detection model yields the liveness detection result. This liveness detection model is trained using a contrastive loss determined based on the fused features and a classification loss determined based on the probabilities corresponding to the labels of the training samples. Both the fused features and the probabilities corresponding to the labels are obtained by inputting the user image as the training sample and its corresponding local maps into the liveness detection model.

[0057] As can be seen from the above methods, this method can extract global features and local features at different scales, resulting in comprehensive and sufficient feature extraction and making the model output more accurate. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0059] Figure 1 This is a flowchart illustrating a live detection method provided in this specification;

[0060] Figure 2 This is a flowchart illustrating a model training method provided in this specification.

[0061] Figure 3 This is a schematic diagram of model training provided in this specification;

[0062] Figure 4 This is a schematic diagram of a liveness detection device provided in this specification;

[0063] Figure 5 This is a schematic diagram of a model training device provided in this specification;

[0064] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this specification. Detailed Implementation

[0065] Currently, non-cooperative liveness detection methods typically acquire user images for face recognition and input these images into a model to extract global features, thus obtaining a liveness detection result (live or non-live). When a real person performs face recognition, the user image acquired is the image corresponding to a live person. However, when face recognition is performed using other media carrying faces, such as printed photos or masks, the user image acquired on that medium is the image corresponding to a non-live person.

[0066] For some non-living photographs, masks, etc., their local areas possess features that facilitate accurate identification of whether a person is alive. For example, printed photographs often have characteristic paper textures or reflective surfaces. For face masks, especially those with holes, the transition between the mask portion and the actual skin at the holes is usually unnatural. For photographs displayed on a screen, reflections and moiré patterns on the screen may also appear. These features are all local characteristics that contribute to accurate liveness detection. However, existing methods do not consider these local features.

[0067] Because existing methods only focus on global features and cannot extract detailed local features from images, they cannot accurately detect whether a person is alive in non-living media such as high-resolution printed images, face masks, and photos containing faces displayed on screens.

[0068] Furthermore, existing models for liveness detection are typically trained based solely on the labels (live or non-live) of the training samples, with the loss function determined accordingly. Consequently, the generalization ability of the trained models is not good enough.

[0069] To at least partially address the aforementioned problems, this specification provides a liveness detection and model training method. In this method, in addition to extracting global features from the user image using a liveness detection model, local maps at different scales are determined based on the user image. Local features corresponding to these local maps are then extracted using the liveness detection model. This results in a fused feature set that includes both global and local features. This allows the liveness detection model to output more accurate results based on the fused feature set.

[0070] Furthermore, when training the liveness detection model, in addition to determining the classification loss based on the labels of the training samples, a contrastive loss was also determined. Using both contrastive and classification losses to train the liveness detection model enables it to more accurately represent different features, narrowing the distance between similar features in the feature space and widening the distance between dissimilar features, resulting in more accurate liveness detection results.

[0071] Because training a liveness detection model based on contrastive loss can make the fused features corresponding to the same label output by the liveness detection model closer in the feature space, make the fused features corresponding to different labels farther apart in the feature space, make the fused features of the same label more aggregated and concentrated, and make the fused features of different labels more separated.

[0072] This allows for a more accurate identification of which label's fusion feature in the feature space is closer to when encountering non-living user images of unknown class (such as those not included in the training samples, excluding printed photos, screen photos, and face masks). In other words, it more accurately identifies whether the fusion feature is more similar to the fusion feature of the living class or the fusion feature of the non-living class. Furthermore, this enables the liveness detection model to accurately output the probability that the user image of the unknown class belongs to the living class and the probability that it is not living; the higher probability is the liveness detection result. Therefore, training the liveness detection model with contrastive loss can improve its generalization ability.

[0073] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0074] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0075] Figure 1 This is a flowchart illustrating a liveness detection method described in this specification, which specifically includes the following steps:

[0076] S100: Acquire a user image and determine local images of different scales based on the user image.

[0077] In this specification, the liveness detection method can be executed by a server. Of course, it can also be executed by other electronic devices capable of performing liveness detection based on user images using a pre-trained liveness detection model; this specification does not impose any restrictions on this.

[0078] In this specification, the pre-trained liveness detection model includes a regression network and several feature extraction networks. This liveness detection model is trained using a contrastive loss determined based on fused features and a classification loss determined based on the probabilities corresponding to the labels of the training samples. Both the fused features and the probabilities corresponding to the labels are obtained by inputting the user image as the training sample and its corresponding local maps into the liveness detection model. The training process for the liveness detection model will be explained later and will not be elaborated upon here.

[0079] In one or more embodiments of this specification, during liveness detection, the server first acquires a user image. This user image may be acquired via a visual sensor during face recognition.

[0080] After acquiring the user image, in order for the subsequent liveness detection model to extract features at different scales, including global and local features, the server can determine local maps of different scales based on the acquired user image.

[0081] That is, a local area of ​​the user's image is extracted and used as a local image.

[0082] Since this liveness detection method is used to detect whether the user's face is alive or a non-living medium carrying the user's face during face recognition, and the user's face contains a large number of features that help identify whether it is alive, the user's facial image can be used as a local region for liveness detection.

[0083] Therefore, in one or more embodiments of this specification, when determining a local image, specifically, the server can use a target detection algorithm to determine the user's facial region contained in the user image, and determine a facial image based on the obtained facial region.

[0084] In one or more embodiments of this specification, bounding boxes of each face in the user image can be obtained through object detection. The face region can be the image region surrounded by the bounding boxes of the faces obtained through object detection in the user image.

[0085] It should be noted that this manual does not restrict the specific object detection algorithm used; any existing object detection algorithm that can recognize human faces and output bounding boxes can be used.

[0086] In one or more embodiments of this specification, after obtaining a facial image, the server may further determine a partial image of the user image based on the facial image.

[0087] The server can locate facial landmarks to determine the key point regions in the facial image, and then identify key point regions belonging to a preset type. Subsequently, the facial image and the key point regions of the preset type can be used as local images of different scales corresponding to the training sample.

[0088] There is no limit to the number of key point regions of the preset types. For example, the type may include at least one of mouth and eyes. Therefore, key point regions belonging to the preset types may include at least one of the eye key point regions and the mouth key point regions in the user image. Since facial landmark localization is a mature technology, the specific process of performing facial landmark localization will not be described in detail here.

[0089] Because other people may be moving around in the environment when performing facial recognition, the captured user image may contain the faces of multiple people. However, the face closest to the sensor should be the face of the user being recognized, and its area is also the largest among all faces.

[0090] Therefore, in one or more embodiments of this specification, when only one facial region is identified, the server can determine the facial image based on that facial region. That is, the image corresponding to that facial region is extracted from the user image as the facial image.

[0091] When multiple facial regions are identified, the server can determine the area of ​​each facial image and identify the facial image corresponding to the user's image based on the facial region with the largest area.

[0092] The area of ​​the facial region can be the area of ​​the bounding box of the facial region.

[0093] S102: Input the user image and each local image into different feature extraction networks to determine the global features corresponding to the user image and the local features corresponding to each local image.

[0094] In one or more embodiments of this specification, the total number of user images and local images determined is the same as the number of feature extraction networks included in the liveness detection model.

[0095] After determining the user image and its local images, the server can input each image into a different feature extraction network to determine the global features corresponding to the user image and the local features corresponding to each local image. That is, each image in the user image and each local image is input into a feature extraction network to obtain the corresponding global and local features output by the feature extraction network.

[0096] S104: The global features and local features are concatenated to obtain the fused features.

[0097] In one or more embodiments of this specification, after obtaining the global features and each local feature, the server can concatenate the global features and each local feature to obtain the fused features. This enables the liveness detection model to more accurately output the probability that the user image is a live person and the probability that it is not a live person in subsequent steps, based on the fused features containing both global and local features.

[0098] S106: Input the fused features into the regression network to obtain the liveness detection result.

[0099] In one or more embodiments of this specification, the server can input the obtained fused features into a regression network to obtain liveness detection results.

[0100] The liveness detection result may include either a live body or a non-live body.

[0101] Specifically, after inputting the fused features into the regression network, the server can determine the category with the higher probability from the probability of the user image being a live object or the probability of the user image being a non-live object, and use this as the liveness detection result for the user image.

[0102] based on Figure 1 The liveness detection method shown acquires a user image and further determines local maps of different scales based on the user image to be detected. Then, it uses different feature extraction networks of the liveness detection model to extract features from the user image and each local map to obtain global features and local features of different scales. The global features and local features are then fused to obtain fused features. Finally, the fused features are input into the regression network of the liveness detection model, and the liveness detection result is obtained based on the output of the regression network.

[0103] As can be seen from the above methods, this method performs global and local feature extraction on user images, resulting in more comprehensive feature extraction. Based on the fused features obtained by stitching together global features and local features at different scales, it can more accurately determine the liveness detection results.

[0104] Furthermore, this specification also provides a model training method based on which a liveness detection model can be trained. As mentioned above, the liveness detection model includes a regression network and several feature extraction networks. Moreover, the networks included in this liveness detection model are based on... Figure 2 The end-to-end training method shown is used for joint training.

[0105] Specifically, the feature extraction networks included in this liveness detection model can be one or more of the following: Convolutional Neural Network (CNN), Multilayer Perceptron (MLP), Recurrent Neural Network (RNN), and Transformer. For example, all feature extraction networks can be CNNs, or they can be different, with some being CNNs (or other networks listed above) and some being MLPs (or other networks listed above). That is, each feature extraction network can be at least a portion of CNNs, MLPs, RNNs, and Transformers. The regression network of this liveness detection model can be an MLP network.

[0106] Figure 2 This is a flowchart illustrating one model training method described in this specification, which specifically includes the following steps:

[0107] S300: Obtain images of each user as training samples, and label whether each training sample is a live person.

[0108] In one or more embodiments of this specification, the model training method can be executed by a server. When training the entity detection model, the server first acquires pre-collected user images as training samples. For each training sample, whether the training sample is a live object is used as the corresponding label.

[0109] Among the acquired user images are user images classified as live, and user images classified as non-live, such as printed photos of users, masks with user faces printed on them, and user photos displayed on screens.

[0110] S302: For each training sample, determine the local maps of different scales corresponding to that training sample.

[0111] In one or more embodiments of this specification, after determining each training sample and its corresponding label, the server can determine local maps of different scales corresponding to each training sample. This allows the server to obtain global features and local features of different scales through the liveness detection model to be trained in subsequent steps.

[0112] Specifically, for each training sample, the server can use an object detection algorithm to determine the facial regions contained in the training sample, and determine a facial image based on the obtained facial regions. Then, the server can determine the keypoint regions in the facial image, identify keypoint regions belonging to a preset type, and use the facial image and the preset type of keypoint regions as local images of different scales corresponding to the training sample.

[0113] In one or more embodiments of this specification, when determining a facial image, if multiple facial regions are determined, the server can determine the facial image corresponding to the training sample based on the facial region with the largest area; if only one facial region is determined, the facial image is determined based on the single facial region.

[0114] When determining key point regions in a facial image, the server can perform facial key point localization on the facial image, determine each key point region in the facial image and the type of each key point region, and then, based on the type of each key point region, determine the eye key point region from each key point region as a key point region belonging to a preset type.

[0115] Of course, as described in step S100 above, there is no limit to the number of key point regions of the preset type. Key point regions may also include other types of key point regions. Therefore, the server can also determine the mouth key point region from among the key point regions according to the type of each key point region, as a key point region belonging to the preset type. Of course, other key point regions, such as the ear, can also be determined.

[0116] During keypoint localization, the coordinates of various types of keypoints on the user's face can be obtained. Keypoints are the points that define keypoint regions. For example, keypoints for the eyes define the eye's outline. Based on the coordinates of these keypoints, the position of the eyes in the user's image can be determined. This can be achieved by directly determining the eye's bounding box based on the coordinates of these keypoints, and then using the image region corresponding to the bounding box as the keypoint region for the eyes. Alternatively, after determining the eye's bounding box based on the coordinates of the keypoints, the bounding box can be expanded according to a preset expansion distance. The server can then use the expanded bounding box as the keypoint region corresponding to the eyes.

[0117] S304: Input the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map.

[0118] In one or more embodiments of this specification, after determining each local map, the server can input the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map.

[0119] S306: The global features and local features are concatenated to obtain fused features, and the fused features are input into the regression network to obtain the probability that the training sample belongs to a living or non-living organism.

[0120] After obtaining the global features and local features, the server can concatenate the global features and local features to obtain fused features, and input the fused features into the regression network of the liveness detection model to obtain the probability that the training sample belongs to a live body and the probability that it belongs to a non-live body.

[0121] S308: Determine the classification loss based on the probability corresponding to the label of each training sample, and determine the contrast loss based on the fusion feature corresponding to each training sample, so as to determine the total loss.

[0122] In one or more embodiments of this specification, the server can determine the probability corresponding to the label of each training sample from the probability that the training sample belongs to a live organism and the probability that the training sample belongs to a non-live organism output by the regression network.

[0123] The server then determines the classification loss based on the probability corresponding to the label of each training sample, and determines the contrastive loss based on the fusion features corresponding to each training sample. The total loss is then determined based on both the classification loss and the contrastive loss.

[0124] In one or more embodiments of this specification, the formula for determining the classification loss may be as follows:

[0125]

[0126] Here, L1 represents the classification loss, and I is the set of training samples. For example, when determining the loss using training samples from a subset (batch), I is a batch. i This refers to the probability corresponding to the label of the i-th training sample in set I. It can be seen that the probability corresponding to the label of a training sample is negatively correlated with the classification loss.

[0127] In one or more embodiments of this specification, when determining the contrastive loss, specifically, the server can determine the pairwise similarity between the fusion features of each training sample as each second similarity, and for each training sample, determine the similarity between the fusion features of that training sample and other training samples with the same label as that training sample from each second similarity, and use this as a first similarity. Then, the server can determine the loss corresponding to that training sample based on the determined first similarity and second similarity, so as to determine the contrastive loss based on the losses corresponding to each training sample.

[0128] The second similarity used to determine the loss corresponding to the training sample includes the similarity that is also used as the first similarity.

[0129] When determining the loss for each training sample, the server can sum the second similarities to obtain a comprehensive similarity. Then, for each first similarity, the server can determine the similarity ratio corresponding to that first similarity based on the ratio of that first similarity to the comprehensive similarity. Finally, the server can determine the loss for that training sample based on the similarity ratios corresponding to each of the first similarities.

[0130] In one or more embodiments of this specification, the formula for determining the contrast loss may be as follows:

[0131]

[0132] Where L2 is the contrastive loss, and I is the set of training samples. i Let |Q| represent the fusion feature corresponding to the i-th training sample in set I. Let Q be the set of other training samples with the same label as the i-th training sample. |Q| represents the number of training samples contained in set Q. q Let z represent the fusion feature of the q-th training sample in set Q. a Let be the fusion feature of the a-th training sample in set A. Set A may contain the i-th training sample and all other training samples, or it may not contain the i-th training sample and only contain the other training samples.

[0133] That is, set A can be the same as set I, or it can be set I after removing the i-th training sample. τ is a preset adjustment parameter that can be set as needed, for example, it can be 0.1, or it can be other values, depending on the specific requirements.

[0134] That is, a first similarity (the first similarity between the i-th training sample in set I and the q-th training sample in set Q). That is, a second similarity (the second similarity between the i-th training sample in set I and the a-th training sample in set A). That is, to fuse similarity. That is, the similarity ratio between the q-th training sample in set Q and the i-th training sample in set I, corresponding to the first similarity. That is, the loss corresponding to the i-th training sample in set I.

[0135] In one or more embodiments of this specification, the total loss can be obtained by summing the classification loss and the contrast loss. Alternatively, the total loss can be determined by weighted summation of the classification loss and the contrast loss.

[0136] In one or more embodiments of this specification, the formula for determining the total loss may be as follows:

[0137] L 总 =L1+λL2

[0138] Among them, L 总 The total loss is L1, the classification loss is L1, and the contrast loss is L2. λ is a preset weighting coefficient, which can be set as needed, for example, it can be 0.5, or other values, depending on the requirements.

[0139] S310: Train the liveness detection model with the goal of minimizing the total loss.

[0140] After determining the total loss, the server can train the liveness detection model with the goal of minimizing the total loss.

[0141] based on Figure 2 The model training method shown involves using each user image as a training sample, determining local maps of different scales corresponding to that training sample, and inputting the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map of different scales. Then, the fused features obtained by concatenating the global features and the local features are input into a regression network to obtain the probability that the training sample belongs to a live or non-live object. The classification loss is determined based on the probability corresponding to the obtained label, the contrast loss is determined based on the fused features corresponding to each training sample, and the total loss is determined based on the classification loss and the contrast loss. The liveness detection model is trained with the goal of minimizing the total loss.

[0142] As can be seen from the above methods, this approach achieves more comprehensive feature extraction by performing both global and local feature extraction on user images. Based on the fused features obtained by concatenating global and local features, the liveness detection model can output more accurate liveness detection results. Furthermore, when training the liveness detection model, in addition to training with a classification loss determined based on the labels of the training samples, it also trains with a contrastive loss determined based on the fused features corresponding to the training samples. The total loss determined by the classification loss and contrastive loss enables the liveness detection model to output more accurate liveness detection results while exhibiting better generalization ability.

[0143] For ease of understanding, this instruction manual also provides the following: Figure 3 The diagram shown illustrates the model training process.

[0144] like Figure 3As shown, based on the user image used as a training sample, a local map, namely the user's facial image, can be obtained. Further, based on the user's facial image, a local map, namely the user's eye keypoint region, can be obtained. After inputting the user image and each local map into different feature extraction networks of the liveness detection model, the global features corresponding to the user image, the local features corresponding to the first local map (the user's facial image), and the local features corresponding to the second local map (the user's eye keypoint region) can be obtained. By concatenating these local maps, a fused feature can be obtained. Inputting the fused feature into a regression network, the probability that the training sample belongs to a live subject and the probability that it belongs to a non-live subject can be obtained.

[0145] After obtaining the probabilities corresponding to multiple training samples output by the liveness detection model, the classification loss can be determined based on the probability corresponding to the label in each training sample's probability, and the contrastive loss can be determined based on the similarity determined by the fusion features of each training sample (including the similarity determined by the fusion features of training samples with the same label, and the similarity determined by the fusion features of training samples with different labels). The total loss can be obtained based on the classification loss and the contrastive loss. After obtaining the total loss, the liveness detection model can be trained with the goal of minimizing the total loss.

[0146] The above describes one or more embodiments of the liveness detection and model training methods provided in this specification. Based on the same idea, this specification also provides corresponding liveness detection devices, such as... Figure 4 As shown.

[0147] Figure 4 This is a schematic diagram of a liveness detection device provided in this specification. The pre-trained liveness detection model includes a regression network and several feature extraction networks. The device includes:

[0148] The acquisition module 200 is used to acquire a user image and determine local images of different scales based on the user image.

[0149] The first feature extraction module 201 is used to input the user image and each local image into different feature extraction networks to determine the global features corresponding to the user image and the local features corresponding to each local image.

[0150] The fusion module 202 is used to concatenate the global features and the local features to obtain fused features;

[0151] Detection module 203 is used to input the fused features into the regression network to obtain liveness detection results;

[0152] The liveness detection model is trained using a contrast loss determined by fusion features and a classification loss determined by the probability corresponding to the label of the training sample. The fusion features and the probability corresponding to the label are obtained by inputting the user image as the training sample and its corresponding local image into the liveness detection model.

[0153] This manual also provides corresponding model training devices, such as... Figure 5 As shown.

[0154] Figure 5 This is a schematic diagram of a model training device provided in this specification. The liveness detection model includes a regression network and several feature extraction networks. The device includes:

[0155] The sample determination module 400 is used to acquire each user's image as each training sample and to label whether each training sample is a live person.

[0156] The local map determination module 401 is used to determine the local maps of different scales corresponding to each training sample.

[0157] The second feature extraction module 402 is used to input the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map.

[0158] The regression module 403 is used to concatenate the global features and local features to obtain fused features, and input the fused features into the regression network to obtain the probability that the training sample belongs to a living or non-living organism.

[0159] The loss determination module 404 is used to determine the classification loss based on the probability corresponding to the label of each training sample, and to determine the contrast loss based on the fusion feature corresponding to each training sample, so as to determine the total loss.

[0160] Training module 405 is used to train the liveness detection model with the goal of minimizing the total loss.

[0161] Optionally, the local image determination module 401 is further configured to, for each training sample, determine the facial region contained in the training sample through a target detection algorithm, determine the facial image based on the obtained facial region, determine the key point region in the facial image, determine the key point region belonging to a preset type from it, and use the facial image and the key point region of the preset type as local images of different scales corresponding to the training sample.

[0162] Optionally, the local image determination module 401 is further configured to determine the facial image corresponding to the training sample based on the largest facial region when there are multiple facial regions, and to determine the facial image corresponding to the training sample based on the determined facial region when there is only one facial region.

[0163] Optionally, the local image determination module 401 is further configured to locate facial key points in the facial image, determine each key point region and the type of each key point region in the facial image, and determine the eye key point region from each key point region according to the type of each key point region, as a key point region belonging to a preset type.

[0164] Optionally, the loss determination module 404 is further configured to determine the pairwise similarity between the fusion features of each training sample as each second similarity, and for each training sample, determine the similarity between the fusion features of the training sample and other training samples with the same label as the training sample from each second similarity, and at the same time, use it as a first similarity. Based on the determined first similarity and each second similarity, determine the loss corresponding to the training sample, and determine the contrast loss based on the loss corresponding to each training sample.

[0165] Optionally, the loss determination module 404 is further configured to sum the second similarities to obtain a comprehensive similarity, determine the similarity ratio corresponding to each first similarity based on the ratio of the first similarity to the comprehensive similarity, and determine the loss corresponding to the training sample based on the similarity ratios corresponding to each first similarity of the training sample.

[0166] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 , Figure 2 The provided methods include liveness detection and model training.

[0167] This instruction manual also provides Figure 6 The diagram shows the structure of the electronic device. Figure 6 As shown, at the hardware level, this electronic device includes a processor, an internal bus, memory, and non-volatile storage, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile storage into memory and then executes it to achieve the above-mentioned functions. Figure 1 , Figure 2 The provided methods include liveness detection and model training.

[0168] It should be noted that all actions involving the acquisition of signals, information, or data in this manual are performed in accordance with the relevant data protection laws and regulations of the country where the device is located, and with the authorization of the owner of the relevant device.

[0169] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0170] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0171] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0172] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0173] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0174] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0180] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0182] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0184] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0185] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A model training method, characterized in that, The liveness detection model includes a regression network and several feature extraction networks, and the method includes: Each user's image is used as a training sample, and whether each training sample is a live person is used as a label. For each training sample, determine the local maps of different scales corresponding to that training sample; The training sample and each local map are input into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map. The global features and local features are concatenated to obtain fused features, and the fused features are input into the regression network to obtain the probability that the training sample belongs to a living or non-living organism. The classification loss is determined based on the probability corresponding to the label of each training sample, and the contrast loss is determined based on the fusion feature corresponding to each training sample, so as to determine the total loss. The liveness detection model is trained with the goal of minimizing the total loss. Based on the fusion features corresponding to each training sample, the contrastive loss is determined, specifically including: Determine the pairwise similarity between the fusion features of each training sample, and use it as the second similarity. For each training sample, the similarity between the training sample and the fusion features of other training samples with the same label as the training sample is determined from each second similarity, and this is used as the first similarity. Based on the determined first similarity and second similarity scores, the loss corresponding to the training sample is determined. Based on the loss corresponding to each training sample, determine the contrastive loss; based on the determined first similarity and second similarity, determine the loss corresponding to the training sample, specifically including: Sum the second similarity scores to obtain the comprehensive similarity score. For each first similarity, the similarity ratio corresponding to that first similarity is determined based on the ratio of that first similarity to the comprehensive similarity. The loss corresponding to the training sample is determined based on the similarity ratio corresponding to each first similarity of the training sample.

2. The method as described in claim 1, characterized in that, For each training sample, determine the local maps of different scales corresponding to that training sample, specifically including: For each training sample, the facial region contained in the training sample is determined by the object detection algorithm; Determine the facial image based on the obtained facial region; Determine the key point regions in the facial image, and from there identify key point regions belonging to a preset type; The facial image and the key point region of the preset type are used as local images of different scales corresponding to the training sample.

3. The method as described in claim 2, characterized in that, The facial image is determined based on the obtained facial region, specifically including: When there are multiple facial regions, the facial image corresponding to the training sample is determined based on the facial region with the largest area. When there is only one facial region, the facial image corresponding to the training sample is determined based on the determined facial region.

4. The method as described in claim 2, characterized in that, Determining key point regions in the facial image, and from there identifying key point regions belonging to a preset type, specifically includes: Facial key point localization is performed on the facial image to determine the key point regions and the type of each key point region in the facial image. Based on the type of each key point region, the key point region of the eye is determined from each key point region as the key point region belonging to the preset type.

5. A model training device, characterized in that, The liveness detection model includes a regression network and several feature extraction networks, and the device includes: The sample determination module is used to acquire images of each user as training samples and label whether each training sample is a live person. The local map determination module is used to determine the local maps of different scales corresponding to each training sample. The second feature extraction module is used to input the training sample and each local map into different feature extraction networks to determine the global features corresponding to the training sample and the local features corresponding to each local map. The regression module is used to concatenate the global features and local features to obtain fused features, and input the fused features into the regression network to obtain the probability that the training sample belongs to a live or non-live organism. The loss determination module is used to determine the classification loss based on the probability corresponding to the label of each training sample, and to determine the contrast loss based on the fusion features corresponding to each training sample, so as to determine the total loss. The training module is used to train the liveness detection model with the goal of minimizing the total loss. Based on the fusion features corresponding to each training sample, the contrastive loss is determined, specifically including: Determine the pairwise similarity between the fusion features of each training sample, and use it as the second similarity. For each training sample, the similarity between the training sample and the fusion features of other training samples with the same label as the training sample is determined from each second similarity, and this is used as the first similarity. Based on the determined first similarity and second similarity scores, the loss corresponding to the training sample is determined. Based on the loss corresponding to each training sample, determine the contrastive loss; based on the determined first similarity and second similarity, determine the loss corresponding to the training sample, specifically including: Sum the second similarity scores to obtain the comprehensive similarity score. For each first similarity, the similarity ratio corresponding to that first similarity is determined based on the ratio of that first similarity to the comprehensive similarity. The loss corresponding to the training sample is determined based on the similarity ratio corresponding to each first similarity of the training sample.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 4.

Images