Living body detection model training method and device, living body detection method and device and storage medium

Abstract

The invention discloses a living body detection model training method and device, a living body detection method and device and a storage medium. The method comprises the steps that multiple batches of face image data are collected in a data set; the human face image data comprises human face image components of various modes, and labels marked on the human face image data comprise a living body category and various non-living body categories; determining a living body detection model; aiming at the face image data of each batch, inputting the face image components of various modes into an encoder, and encoding the face image components of various modes into multi-mode face features; inputting the multi-modal face features into the head structure, and classifying to obtain a target category; generating a total loss value according to the multi-modal face features, the target category and the tag, so that the total loss value indicates a direction of gathering towards the living body category and being far away from the non-living body category; and training the living body detection model according to the total loss value. According to the embodiment, sample imbalance can be relieved, the robustness of the living body detection model is improved, and therefore the accuracy of living body detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of liveness detection, and in particular to a liveness detection model training and liveness detection method, device and storage medium thereof. Background Art

[0002] In many scenarios such as security, access control, and payment, users are required to undergo liveness detection to authenticate their identities.

[0003] At present, one of the commonly used methods of liveness detection is silent liveness detection. Silent liveness detection does not require users to cooperate in making specific actions, such as blinking, opening mouth, etc. It mainly analyzes subtle dynamic changes in facial image data, such as skin texture, reflective characteristics, micro-expressions, etc., to distinguish real facial images from forged facial images or masks.

[0004] The samples of silent liveness detection are unbalanced. Liveness data of various angles and postures are easier to prepare, while non-liveness data, such as paper attacks, electronic photo attacks, headgear attacks, etc., often require a lot of time to prepare various attack props, resulting in lower robustness of the liveness detection model, and thus lower accuracy of liveness detection. Summary of the invention

[0005] In view of this, the present invention provides a training method, device and storage medium for a liveness detection model, so as to improve the robustness of the liveness detection model and thus improve the accuracy of liveness detection.

[0006] A first aspect of the present invention provides a method for training a liveness detection model, comprising:

[0007] Collecting multiple batches of facial image data in a data set; the facial image data includes facial image components of multiple modalities, and labels annotated on the facial image data include living body categories and multiple non-living body categories;

[0008] Determine a liveness detection model; the liveness detection model includes an encoder and a head structure;

[0009] For each batch of the facial image data, inputting the facial image components of multiple modalities into the encoder to be encoded into multimodal facial features;

[0010] Inputting the multimodal facial features into the head structure to classify the target category;

[0011] Generate a total loss value according to the multimodal facial features, the target category and the label, so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category;

[0012] The living body detection model is trained according to the total loss value.

[0013] A second aspect of the present invention provides a living body detection method, comprising:

[0014] Collecting facial image data of a user; the facial image data includes facial image components of multiple modalities;

[0015] Loading a liveness detection model trained according to the method of the first aspect; the liveness detection model includes an encoder and a head structure;

[0016] Inputting the facial image components of multiple modalities into the encoder to be encoded into multimodal facial features;

[0017] The multimodal facial features are input into the head structure for classification to obtain a target category; the target category includes a living body category or multiple non-living body categories.

[0018] A third aspect of the present invention provides a training device for a liveness detection model, comprising:

[0019] A sample collection module, used to collect multiple batches of face image data in a data set; the face image data includes face image components of multiple modalities, and the labels annotated on the face image data include a living body category and multiple non-living body categories;

[0020] A model determination module, used to determine a liveness detection model; the liveness detection model includes an encoder and a head structure;

[0021] A face encoding module, for inputting the face image components of multiple modes into the encoder for encoding into multi-modal face features for each batch of the face image data;

[0022] A face classification module, used for inputting the multimodal face features into the head structure to classify the target category;

[0023] A total loss value generating module, used for generating a total loss value according to the multimodal facial features, the target category and the label, so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category;

[0024] A model training module is used to train the living body detection model according to the total loss value.

[0025] A fourth aspect of the present invention provides a living body detection device, comprising:

[0026] A face acquisition module, used to acquire face image data of a user; the face image data includes face image components of multiple modalities;

[0027] A model loading module, used to load a liveness detection model trained according to the method of the first aspect; the liveness detection model includes an encoder and a head structure;

[0028] A face encoding module, used for inputting the face image components of multiple modes into the encoder to encode them into multi-modal face features;

[0029] The face classification module is used to input the multimodal face features into the head structure to classify the target category; the target category includes a living body category or multiple non-living body categories.

[0030] A fifth aspect of the present invention provides an electronic device, the electronic device comprising:

[0031] at least one processor; and

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the liveness detection model as described in the first aspect above or the liveness detection method as described in the second aspect above.

[0034] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training method of the liveness detection model as described in the first aspect or the liveness detection method as described in the second aspect.

[0035] The sixth aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training and method of the liveness detection model as described in the first aspect or the liveness detection method as described in the second aspect.

[0036] In this embodiment, multiple batches of face image data are collected in a data set; the face image data includes face image components of multiple modes, and the labels annotated on the face image data include live categories and multiple non-live categories; a liveness detection model is determined; the liveness detection model includes an encoder and a head structure; for each batch of face image data, the face image components of multiple modes are input into the encoder to be encoded into multi-modal face features; the multi-modal face features are input into the head structure to be classified to obtain the target category; a total loss value is generated based on the multi-modal face features, the target category and the label, so that the total loss value indicates the direction of aggregation towards the live category and away from the non-live category; the liveness detection model is trained based on the total loss value. This embodiment constructs a loss function so that the liveness detection model aggregates towards samples of the liveness category and away from samples of the non-liveness category during training, which can alleviate sample imbalance, improve the robustness of the liveness detection model, and thus improve the accuracy of liveness detection.

[0037] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 This is a flow chart of a method for training a living body detection model provided in Example 1 of the present invention.

[0040] Figure 2 It is a structural schematic diagram of a living body detection model provided in Example 1 of the present invention.

[0041] Figure 3 This is a flow chart of a living body detection method provided in Embodiment 2 of the present invention.

[0042] Figure 4 It is a structural schematic diagram of a training device for a living body detection model provided in Embodiment 3 of the present invention.

[0043] Figure 5 It is a structural schematic diagram of a living body detection device provided in Embodiment 4 of the present invention.

[0044] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] Embodiment 1

[0048] See also Figure 1 , shows a flow chart of a method for training a liveness detection model provided by the first embodiment of the present invention. The method can be executed by a training device for a liveness detection model. The training of the liveness detection model and the liveness detection device can be implemented in the form of hardware and / or software. The training device for the liveness detection model can be configured in an electronic device. Figure 1 As shown, the method includes:

[0049] Step 101: Collect multiple batches of face image data in a data set.

[0050] In this embodiment, a data set may be prepared through various channels, and the data set includes a plurality of face image data.

[0051] Among them, Figure 2 As shown, each facial image data is an image set, which includes facial image components of multiple modalities, for example, RGB (red, green, and blue) facial image components, IR (infrared) facial image components, Depth (depth) image components, and so on.

[0052] Generally speaking, if liveness detection is implemented using facial image data according to the specifications of static liveness detection, the facial image components of multiple modalities in the same facial image data are usually facial image components collected from the user at the same time using multiple sensors.

[0053] In addition, each face image data has been labeled with a label Label, and the label Label annotated to the face image data includes a living body category and multiple non-living body categories.

[0054] At this point, the data set can be represented as x i ∈{x1,x2,…,x n-1 ,x n}, where x i ={x i1 ,x i2 ,…,x ij ,…,x im}, x i is the i-th face image data, x ij is the image data component of the jth modality in the i-th face image data, i∈n, n is the number of samples, j∈m, m is the number of modalities, and the label can be expressed as y i ∈{y1,y2,…,y n-1 ,y n},y i is the label of the i-th face image data.

[0055] Furthermore, the live category indicates that the user is a real person, and the non-live category indicates that the user is not a real person, which includes a non-live paper attack category, a non-live electronic photo attack category, a non-live headgear attack category, and the like.

[0056] In this embodiment, a sampling probability may be set for each living body category and each non-living body category, and face image data under corresponding labels are collected in the data set according to the sampling probability to form multiple batches of samples.

[0057] For any facial image component of each facial image data, one or more preprocessings may be performed, such as binarization, Fourier transform, brightness transform, etc. Each preprocessing may add a new facial image component of a modality.

[0058] Step 102: Determine a liveness detection model.

[0059] In this embodiment, a liveness detection model can be constructed in advance based on deep learning, such as Figure 2 As shown, the living body detection model includes an encoder and a head structure Head.

[0060] Among them, the encoder contains structures such as convolutional layers, pooling layers, and attention mechanisms, which are responsible for extracting facial features from facial image components of multiple modalities.

[0061] The head structure includes convolutional layers, pooling layers, fully connected layers, activation functions and other structures, which are responsible for performing live body classification tasks based on facial features.

[0062] In practical applications, the structure of the liveness detection model (including the encoder and the head structure) is not limited to the artificially designed neural network, but can also be a neural network optimized by a model quantization method, a neural network searched for characteristics of static liveness detection by a NAS (Neural Architecture Search) method, and so on. This embodiment does not impose any restrictions on this.

[0063] Step 103: For each batch of facial image data, input facial image components of multiple modalities into an encoder and encode them into multimodal facial features.

[0064] In practical applications, such as Figure 2 As shown, each batch of facial image data (including facial image components of multiple modalities) can be input into the encoder Encoder and encoded into multimodal facial features.

[0065] In one embodiment of the present invention, Figure 2 As shown, the encoder includes multiple unimodal feature extraction modules E1 and a multimodal feature fusion module E2, wherein the multiple unimodal feature extraction modules E1 and the multimodal feature fusion module E2 are encapsulated modules, which include structures such as convolutional layers, pooling layers, and attention mechanisms.

[0066] Then, in this embodiment, step 103 may include the following steps:

[0067] Step 1031: for each batch of facial image data, input facial image components of multiple modalities into multiple single-modal feature extraction modules to extract multiple single-modal facial features.

[0068] In a specific implementation, the number of unimodal feature extraction modules E1 is the same as the number of modes of facial image components in the same facial image data, so that each unimodal feature extraction module E1 is responsible for processing a facial image component of a specific modality and extracting features from a facial image component of a specific modality.

[0069] For example, a certain facial image data includes an RGB facial image component, an IR facial image component, and a Depth image component. Then, three unimodal feature extraction modules E1 can be set in the encoder, one of which is responsible for extracting features from the RGB facial image component, another is responsible for extracting features from the IR facial image component, and another is responsible for extracting features from the Depth facial image component.

[0070] The structures of the unimodal feature extraction modules may be the same or different, and this embodiment does not impose any limitation on this.

[0071] For each batch of face image data, the face image components of multiple modes in the face image data in the batch can be respectively input into the corresponding multiple single-mode feature extraction modules to extract features, and multiple single-mode face features F i ={F i1 ,F i2 ,…,F ij ,…,F im}, where F i is the set of unimodal facial features of the i-th facial image data, F ij is the unimodal facial feature of the image data component of the jth modality in the i-th face image data, j∈m, m is the number of modalities.

[0072] Step 1032: Input multiple single-modal facial features into a multi-modal feature fusion module to fuse them into multi-modal facial features.

[0073] In this embodiment, if Figure 2 As shown, multiple unimodal facial features of the same facial image data are simultaneously input into the multimodal feature fusion module E2, and the multiple unimodal facial features are interacted and fused into multimodal facial features.

[0074] For example, in the multimodal feature fusion module M θ (θ is the parameter of the multimodal feature fusion module), the importance of multiple single-modal facial features is calculated. This process can be expressed as: {E i1 ,E i2 ,…,E ij ,…,E im}=M θ ({F i1 ,F i2 ,…,F ij ,…,F im}), where E ij is the unimodal face feature F ij importance.

[0075] Normalize multiple importance levels. Taking the Softmax function as an example, the normalization can be expressed as: {w i1 ,w i2 ,…,w ij ,…,w im}=softmax({E i1 ,E i2 ,…,E ij ,…,E im}), where w ij The importance level is E ij The normalized value.

[0076] According to the normalized importance levels, multiple single-modal facial features are weighted and summed to obtain multi-modal facial features. Then, the multi-modal facial features can be expressed as: Among them, E i is the multimodal facial feature of the i-th facial image data.

[0077] Step 104: Input the multimodal facial features into the head structure and classify them to obtain the target category.

[0078] In this embodiment, the multimodal facial features corresponding to the facial image data may be input into the head structure Head, and the head structure Head performs live classification on the multimodal facial features to obtain the target category of the facial image data.

[0079] The target category includes a living category or multiple non-living categories.

[0080] For example, Figure 2 As shown, the head structure Head includes a multimodal feature extraction module E3 and one or more fully connected layers FC. The multimodal feature extraction module E3 is an encapsulated module, which includes structures such as convolutional layers, pooling layers, and attention mechanisms.

[0081] In this example, the multimodal facial features can be input into the multimodal feature extraction module E3 to extract the reference facial features, the target facial features can be input into the fully connected layer FC and mapped into the target facial features, the target facial features can be activated using functions such as Sgmoid, the probability of the facial image data belonging to each target category can be obtained, and the target category with the highest probability can be output.

[0082] Step 105: Generate a total loss value based on the multimodal facial features, target categories and labels, so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category.

[0083] In practical applications, a total loss value can be generated based on the differences among multimodal facial features, target categories and labels, so that the total loss value indicates a direction of aggregation toward the living category and away from the non-living category, thereby alleviating the problem of data imbalance.

[0084] In one embodiment of the present invention, step 105 may include the following steps:

[0085] Step 1051: Filter out facial image data marked with living body categories from the same batch of facial image data as living body image data.

[0086] In this embodiment, among the facial image data of the same batch, facial image data marked with living body category can be screened out and recorded as living body image data.

[0087] Step 1052: In each batch of facial image data, iteratively update the multimodal facial features of the living image data to obtain a central feature.

[0088] In this embodiment, each batch of facial image data can be tracked, and in a high-dimensional vector space, the center point of the multimodal facial features of the living image data is iteratively updated between batches, and the center point is smoothed between batches.

[0089] Among them, the dimension of the center point is the same as the dimension of the multimodal facial feature. At this time, the center point is recorded as the center feature to characterize the multimodal facial features in the same batch.

[0090] In a specific implementation, the number N of currently annotated face image data is counted.

[0091] If the number is 0 (N=0), the central feature C of the multimodal face feature of the previous batch of face image data is old Set to the central feature C of the multimodal face feature of the current batch of face image data new .

[0092] If the number is not 0 (N≠0), the average value of the multimodal facial features of the living image data is calculated to obtain the average feature.

[0093] Use a linear method to transform the central feature C of the multimodal face features of the previous batch of face image data old The central feature C of the multimodal face feature of the current batch of face image data is fused with the average feature new .

[0094] For example, the fusion process can be expressed as:

[0095]

[0096] Among them, α is the weight, α∈[0,1].

[0097] Step 1053: Generate an auxiliary loss value based on the central feature.

[0098] When the update of the central feature is completed, an auxiliary loss value is constructed based on the central feature, so that the auxiliary loss value indicates a direction of gathering toward the living body category and away from the non-living body category.

[0099] In the specific implementation, Figure 2 As shown, the distances (such as Euclidean distance, etc.) between the multimodal facial features and the central features of the living image data can be averaged and used as the out-of-bounds distance.

[0100] The distance (such as Euclidean distance, etc.) between the multimodal facial features and the central features of each facial image data (including living image data and non-living image data) is calculated as the monomer distance.

[0101] Substitute the cross-border distance and the monomer distance into the following formula to calculate the auxiliary loss value:

[0102]

[0103] Among them, L aux is the auxiliary loss value, M is the set of face image data in the current batch, X is a Boolean value, when face image data i is marked as a living body, X = 1, when face image data i is marked as a non-living body, X = 0, D i is the monomer distance of face image data i, D thr is the out-of-bounds distance, β is the adjustment coefficient (hyperparameter), and max is the function of taking the maximum value.

[0104] Step 1054: Generate a classification loss value based on the target category and label.

[0105] In this embodiment, if Figure 2 As shown, the target category and label can be substituted into the cross entropy function to calculate the loss in the living body classification and obtain the classification loss value L cross .

[0106] Step 1055: merge the auxiliary loss value and the classification loss value into a total loss value.

[0107] In this embodiment, if Figure 2 As shown, the auxiliary loss value L can be calculated using a linear or nonlinear method. aux And the classification loss value L cross Fusion is the total loss value L all .

[0108] For example, L all =αL cross +βLaux , where α and β are weights.

[0109] Step 106: Train a liveness detection model based on the total loss value.

[0110] In this embodiment, the total loss value can be substituted into an optimization algorithm such as SGD (stochastic gradient descent) and Adam (Adaptive momentum) to calculate the update amplitude of the parameters in the liveness detection model, and the parameters in the liveness detection model are updated according to the update amplitude.

[0111] In each round of iterative training, the sampling probability of each face image data configuration under the living body category is determined, and it is judged whether the total loss value converges.

[0112] The sampling probability is used to collect face image data under the living body category for each batch in the data set; the sampling probability of each face image data under the living body category is initially the same.

[0113] In the process of training the liveness detection model, if the total loss value converges initially, each face image data is traversed, and for the same face image data, the target category is compared with the label, and the probability that the face image data belongs to the target category is compared with the preset classification threshold.

[0114] If the target category is the same as the label and the probability is greater than or equal to the classification threshold, the sampling probability of the face image data is reduced by using a step size a or other methods.

[0115] If the target category is the same as the label and the probability is less than the classification threshold, the sampling probability of the face image data is maintained unchanged.

[0116] If the target category is different from the label, the sampling probability of the face image data is increased by using a step size b or other methods.

[0117] When the sampling probability is less than or equal to the preset sampling threshold, the sampling probability is set to 0, and the face image data will not be sampled.

[0118] During the process of training the liveness detection model, if the total loss value continues to converge, it is determined that the liveness detection model has completed training.

[0119] In the process of training the liveness detection model, the liveness image data is screened, and more valuable liveness image data is adaptively selected, which helps to alleviate the problem of sample imbalance.

[0120] In this embodiment, multiple batches of face image data are collected in a data set; the face image data includes face image components of multiple modes, and the labels annotated on the face image data include live categories and multiple non-live categories; a liveness detection model is determined; the liveness detection model includes an encoder and a head structure; for each batch of face image data, the face image components of multiple modes are input into the encoder to be encoded into multi-modal face features; the multi-modal face features are input into the head structure to be classified to obtain a target category; a total loss value is generated based on the multi-modal face features, the target category and the label, so that the total loss value indicates a direction of aggregation toward the live category and away from the non-live category; the liveness detection model is trained based on the total loss value. This embodiment constructs a loss function so that the liveness detection model aggregates toward samples of the liveness category and away from samples of the non-liveness category during training, which can alleviate sample imbalance, improve the robustness of the liveness detection model, and thus improve the accuracy of liveness detection.

[0121] Embodiment 2

[0122] See also Figure 3 , shows a flow chart of a liveness detection method provided by the first embodiment of the present invention, the method can be executed by a liveness detection device, the training of the liveness detection model and the liveness detection device can be implemented in the form of hardware and / or software, and the liveness detection device can be configured in an electronic device. Figure 3 As shown, the method includes:

[0123] Step 301: Collect facial image data of the user.

[0124] In this embodiment, multiple camera sensors are arranged in the devices on the user side (such as access control, mobile terminals, etc.). When performing liveness detection on the user, multiple camera sensors can be called to collect facial image components of multiple modalities for the user, and the facial image components of multiple modalities are constructed into facial image data, that is, the facial image data contains facial image components of multiple modalities.

[0125] For any facial image component of each facial image data, one or more preprocessings may be performed, such as binarization, Fourier transform, brightness transform, etc. Each preprocessing may add a new facial image component of a modality.

[0126] Step 302: Load the liveness detection model.

[0127] In this embodiment, the living body detection model trained according to the method described in Embodiment 1 can be loaded into the memory for execution.

[0128] Among them, the liveness detection model includes an encoder and a head structure.

[0129] Step 303: Input the facial image components of multiple modalities into an encoder and encode them into multimodal facial features.

[0130] In this embodiment, each batch of facial image data (including facial image components of multiple modalities) can be input into the encoder respectively and encoded into multimodal facial features.

[0131] In a specific implementation, the encoder includes multiple unimodal feature extraction modules and a multimodal feature fusion module; then, during the encoding process, for each batch of facial image data, the facial image components of multiple modalities are input into multiple unimodal feature extraction modules to extract multiple unimodal facial features; the multiple unimodal facial features are input into the multimodal feature fusion module to be fused into multimodal facial features.

[0132] Furthermore, in the fusion process, in the multimodal feature fusion module, the importance of multiple unimodal facial features is calculated, the multiple importances are normalized, and the weighted sum of the multiple unimodal facial features is performed according to the normalized multiple importances to obtain the multimodal facial feature.

[0133] Step 304: Input the multimodal facial features into the head structure and classify them to obtain the target category.

[0134] In this embodiment, the multimodal facial features corresponding to the facial image data may be input into the head structure, and the head structure classifies the multimodal facial features to obtain the target category of the facial image data.

[0135] In the specific implementation, the head structure includes a multimodal feature extraction module and a fully connected layer; then, in the classification process, the multimodal facial features are input into the multimodal feature extraction module to extract the reference facial features; the reference facial features are input into the fully connected layer to be mapped into the target facial features; the target facial features are activated to obtain the probabilities of each target category; and the target category with the highest probability is output.

[0136] The target category includes a living category or multiple non-living categories.

[0137] At this time, business operations can be performed on the user based on the user's target category.

[0138] Generally, when a user is identified as a living person, the facial image data can be used to further perform operations such as facial recognition in order to decide whether to allow or prohibit the user from performing the intended business operations, such as door unlocking, payment, etc.

[0139] In addition, when the user is identified as a non-living category, an alarm operation can be performed.

[0140] In this embodiment, facial image data is collected from a user; the facial image data includes facial image components of multiple modes; a liveness detection model is loaded; the liveness detection model includes an encoder and a head structure; the facial image components of multiple modes are input into the encoder to be encoded into multi-modal facial features; the multi-modal facial features are input into the head structure to be classified to obtain a target category. This embodiment gathers samples of the liveness category and stays away from samples of the non-liveness category when training the liveness detection model, which can alleviate sample imbalance, improve the robustness of the liveness detection model, and thus improve the accuracy of liveness detection.

[0141] Embodiment 3

[0142] See also Figure 4 , shows a schematic diagram of the structure of a training device for a living body detection model provided in Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0143] The sample collection module 401 is used to collect multiple batches of face image data in the data set; the face image data includes face image components of multiple modalities, and the labels annotated on the face image data include a living body category and multiple non-living body categories;

[0144] A model determination module 402 is used to determine a liveness detection model; the liveness detection model includes an encoder and a head structure;

[0145] A face encoding module 403 is used for inputting the face image components of multiple modes into the encoder for encoding into multi-modal face features for each batch of the face image data;

[0146] A face classification module 404, configured to input the multimodal face features into the head structure to classify the head structure into a target category;

[0147] A total loss value generating module 405 is used to generate a total loss value according to the multimodal facial features, the target category and the label, so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category;

[0148] The model training module 406 is used to train the living body detection model according to the total loss value.

[0149] In one embodiment of the present invention, the encoder includes a plurality of single-modal feature extraction modules and a multi-modal feature fusion module; the head structure includes a multi-modal feature extraction module and a fully connected layer;

[0150] The face encoding module 403 includes:

[0151] A single-modal facial feature extraction module, used for inputting the facial image components of multiple modalities into multiple single-modal feature extraction modules to extract multiple single-modal facial features for each batch of facial image data;

[0152] A multimodal facial feature fusion module, used for inputting a plurality of the single-modal facial features into the multimodal feature fusion module to fuse them into a multimodal facial feature;

[0153] A multimodal facial feature extraction module, used for inputting the multimodal facial features into the multimodal feature extraction module to extract the multimodal facial features;

[0154] The face classification module 404 includes:

[0155] A reference facial feature extraction module, used for inputting the multimodal facial features into the multimodal feature extraction module to extract reference facial features;

[0156] A target facial feature mapping module, used to input the reference facial features into the fully connected layer and map them into target facial features;

[0157] A probability calculation module, used to activate the target facial features to obtain the probability of each target category;

[0158] The target category output module is used to output the target category with the highest probability.

[0159] In one embodiment of the present invention, the multimodal facial feature fusion module is further used for:

[0160] In the multimodal feature fusion module, the importance of the multiple single-modal facial features is calculated, the multiple importances are normalized, and the weighted sum of the multiple single-modal facial features is performed according to the normalized importances to obtain a multimodal facial feature.

[0161] In one embodiment of the present invention, the total loss value generating module 405 includes:

[0162] A living body image data screening module, used for screening out the face image data marked with the living body category from the face image data of the same batch as the living body image data;

[0163] A center updating module, used for iteratively updating a center feature of the multimodal face feature of the living body image data in each batch of the face image data;

[0164] An auxiliary loss value generating module, used to generate an auxiliary loss value according to the central feature; the auxiliary loss value indicates a direction of gathering toward the living body category and away from the non-living body category;

[0165] A classification loss value generating module, used to generate a classification loss value according to the target category and the label;

[0166] The total loss value fusion module is used to fuse the auxiliary loss value and the classification loss value into a total loss value.

[0167] In one embodiment of the present invention, the central update module includes:

[0168] A quantity statistics module, used for counting the quantity of the facial image data currently annotated;

[0169] a central feature setting module, configured to set the central feature of the multimodal facial features of the previous batch of facial image data as the central feature of the multimodal facial features of the current batch of facial image data if the number is 0;

[0170] an average feature calculation module, configured to calculate an average value of the multimodal facial features of the living body image data to obtain an average feature if the number is not 0;

[0171] The central feature fusion module is used to fuse the central feature of the multimodal facial features of the previous batch of facial image data with the average feature into the central feature of the multimodal facial features of the current batch of facial image data.

[0172] In one embodiment of the present invention, the auxiliary loss value generating module includes:

[0173] an out-of-bounds distance calculation module, configured to average the distances between the multimodal facial features of the living body image data and the central features as the out-of-bounds distance;

[0174] A monomer distance calculation module, used for calculating the distance between the multimodal facial feature and the central feature of each facial image data as a monomer distance;

[0175] The auxiliary loss value calculation module is used to substitute the cross-border distance and the monomer distance into the following formula to calculate the auxiliary loss value:

[0176]

[0177] Among them, L aux is the auxiliary loss value, M is the set of face image data in the current batch, when the face image data i is labeled with the living body category, X=1, when the face image data i is labeled with the non-living body category, X=0, D i is the monomer distance of the face image data i, D thris the out-of-bounds distance, β is the adjustment coefficient, and max is the function of taking the maximum value.

[0178] In one embodiment of the present invention, the model training module 406 includes:

[0179] A sampling probability determination module, used to determine the sampling probability configured for each of the face image data under the living body category; the sampling probability is used to collect the face image data under the living body category for each batch in the data set; the sampling probability of each of the face image data under the living body category is initially the same;

[0180] A classification statistics module, for comparing the target category with the label for the same face image data, and comparing the probability that the face image data belongs to the target category with a preset classification threshold;

[0181] a sampling probability reduction module, configured to reduce the sampling probability of the face image data if the target category is the same as the label and the probability is greater than or equal to the classification threshold;

[0182] A sampling probability maintaining module, configured to maintain the sampling probability of the face image data unchanged if the target category is the same as the label and the probability is less than the classification threshold;

[0183] A sampling probability increasing module, configured to increase the sampling probability of the face image data if the target category is different from the label;

[0184] A sampling probability zeroing module, used for setting the sampling probability to 0 when the sampling probability is less than or equal to a preset sampling threshold;

[0185] The training completion determination module is used to determine that the training of the liveness detection model is completed if the total loss value continues to converge during the training of the liveness detection model.

[0186] The training device for the liveness detection model provided in the embodiment of the present invention can execute the training method for the liveness detection model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the training method for the liveness detection model.

[0187] Embodiment 4

[0188] See also Figure 5 , shows a schematic diagram of the structure of a living body detection device provided by Embodiment 4 of the present invention. Figure 5 As shown, the device comprises:

[0189] The face acquisition module 501 is used to acquire face image data of the user; the face image data includes face image components of multiple modes;

[0190] A model loading module 502 is used to load a liveness detection model trained according to the method described in Embodiment 1; the liveness detection model includes an encoder and a head structure;

[0191] A face encoding module 503, used for inputting the face image components of multiple modes into the encoder to be encoded into multi-modal face features;

[0192] The face classification module 504 is used to input the multimodal face features into the head structure to classify the target category; the target category includes a living body category or multiple non-living body categories.

[0193] In one embodiment of the present invention, the encoder includes a plurality of single-modal feature extraction modules and a multi-modal feature fusion module; the head structure includes a multi-modal feature extraction module and a fully connected layer;

[0194] The face encoding module 503 includes:

[0195] A single-modal facial feature extraction module, used for inputting the facial image components of multiple modalities into multiple single-modal feature extraction modules to extract multiple single-modal facial features for each batch of facial image data;

[0196] A multimodal facial feature fusion module, used for inputting a plurality of the single-modal facial features into the multimodal feature fusion module to fuse them into a multimodal facial feature;

[0197] A multimodal facial feature extraction module, used for inputting the multimodal facial features into the multimodal feature extraction module to extract the multimodal facial features;

[0198] The face classification module 504 includes:

[0199] A reference facial feature extraction module, used for inputting the multimodal facial features into the multimodal feature extraction module to extract reference facial features;

[0200] A target facial feature mapping module, used to input the reference facial features into the fully connected layer and map them into target facial features;

[0201] A probability calculation module, used to activate the target facial features to obtain the probability of each target category;

[0202] The target category output module is used to output the target category with the highest probability.

[0203] In one embodiment of the present invention, the multimodal facial feature fusion module is further used for:

[0204] In the multimodal feature fusion module, the importance of the multiple single-modal facial features is calculated, the multiple importances are normalized, and the weighted sum of the multiple single-modal facial features is performed according to the normalized importances to obtain a multimodal facial feature.

[0205] The liveness detection device provided in the embodiment of the present invention can execute the liveness detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the liveness detection method.

[0206] Embodiment 5

[0207] See also Figure 6 , showing a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0208] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0209] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0210] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a training method for a liveness detection model or a liveness detection method.

[0211] In some embodiments, the training method of the liveness detection model or the liveness detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the liveness detection model or the liveness detection method described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the liveness detection model or the liveness detection method by any other appropriate means (e.g., by means of firmware).

[0212] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0213] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0214] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0216] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0217] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0218] Embodiment 6

[0219] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training method of the liveness detection model or the liveness detection method provided in any embodiment of the present invention.

[0220] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0221] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0222] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training a liveness detection model, characterized in that: include: Collecting multiple batches of facial image data in a data set; the facial image data includes facial image components of multiple modalities, and labels annotated on the facial image data include living body categories and multiple non-living body categories; Determine a liveness detection model; the liveness detection model includes an encoder and a head structure; For each batch of the facial image data, inputting the facial image components of multiple modalities into the encoder to be encoded into multimodal facial features; Inputting the multimodal facial features into the head structure to classify the target category; Generate a total loss value according to the multimodal facial features, the target category and the label, so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category; The living body detection model is trained according to the total loss value.

2. The method according to claim 1, characterized in that The encoder includes a plurality of single-modal feature extraction modules and a multi-modal feature fusion module; the head structure includes a multi-modal feature extraction module and a fully connected layer; For each batch of the facial image data, inputting the facial image components of multiple modes into the encoder to be encoded into multi-modal facial features, including: For each batch of the facial image data, inputting the facial image components of multiple modalities into multiple single-modal feature extraction modules to extract multiple single-modal facial features; Inputting the plurality of single-modal facial features into the multi-modal feature fusion module to fuse them into multi-modal facial features; The step of inputting the multimodal facial features into the head structure and classifying the target category includes: Inputting the multimodal facial features into the multimodal feature extraction module to extract reference facial features; Inputting the reference facial features into the fully connected layer and mapping them into target facial features; Activating the target facial features to obtain the probability of each target category; Output the target category with the highest probability.

3. The method according to claim 2, characterized in that The step of inputting the plurality of single-modal facial features into the multi-modal feature fusion module to fuse them into multi-modal facial features comprises: In the multimodal feature fusion module, the importance of the multiple single-modal facial features is calculated, the multiple importances are normalized, and the multiple single-modal facial features are weighted and summed according to the normalized importances to obtain a multimodal facial feature.

4. The method according to any one of claims 1 to 3, characterized in that The generating a total loss value according to the multimodal facial features, the target category and the label so that the total loss value indicates a direction of gathering toward the living body category and away from the non-living body category, comprises: From the facial image data of the same batch, the facial image data marked with the living body category is selected as the living body image data; In each batch of the facial image data, iteratively updating a central feature of the multimodal facial features of the living body image data; Generate an auxiliary loss value according to the central feature; the auxiliary loss value indicates a direction of gathering toward the living body category and away from the non-living body category; Generating a classification loss value according to the target category and the label; The auxiliary loss value and the classification loss value are fused into a total loss value.

5. The method according to claim 4, characterized in that The iterative updating of the central feature of the multimodal face feature of the living body image data in each batch of the face image data includes: Counting the number of face image data currently annotated; If the number is 0, the central feature of the multimodal facial features of the previous batch of facial image data is set as the central feature of the multimodal facial features of the current batch of facial image data; If the number is not 0, calculating an average value of the multimodal facial features of the living body image data to obtain an average feature; The central feature of the multimodal facial feature of the previous batch of facial image data and the average feature are fused to form the central feature of the multimodal facial feature of the current batch of facial image data.

6. The method according to claim 5, characterized in that The generating of the auxiliary loss value according to the central feature comprises: Taking an average of the distances between the multimodal facial features of the living body image data and the central feature as the out-of-bounds distance; Calculating the distance between the multimodal facial features of each facial image data and the central feature as a monomer distance; Substitute the cross-border distance and the monomer distance into the following formula to calculate the auxiliary loss value: Among them, L aux is the auxiliary loss value, M is the set of face image data in the current batch, when the face image data i is labeled with the living body category, X=1, when the face image data i is labeled with the non-living body category, X=0, D i is the monomer distance of the face image data i, D thr is the out-of-bounds distance, β is the adjustment coefficient, and max is the function of taking the maximum value.

7. The method according to any one of claims 1-3, 5-6, characterized in that: The step of training the living body detection model according to the total loss value comprises: Determine the sampling probability configured for each of the face image data under the living body category; the sampling probability is used to collect the face image data under the living body category for each batch in the data set; the sampling probability of each of the face image data under the living body category is initially the same; In the process of training the liveness detection model, if the total loss value is initially converged, then for the same face image data, the target category is compared with the label, and the probability that the face image data belongs to the target category is compared with a preset classification threshold; If the target category is the same as the label and the probability is greater than or equal to the classification threshold, reducing the sampling probability of the face image data; If the target category is the same as the label and the probability is less than the classification threshold, maintaining the sampling probability of the face image data unchanged; If the target category is different from the label, increasing the sampling probability of the face image data; When the sampling probability is less than or equal to a preset sampling threshold, setting the sampling probability to 0; During the process of training the liveness detection model, if the total loss value continues to converge, it is determined that the liveness detection model has completed training.

8. A method for detecting a living body, characterized in that: include: Collecting facial image data of a user; the facial image data includes facial image components of multiple modalities; Loading a liveness detection model trained by the method according to any one of claims 1 to 7; the liveness detection model includes an encoder and a head structure; Inputting the facial image components of multiple modalities into the encoder to be encoded into multimodal facial features; The multimodal facial features are input into the head structure for classification to obtain a target category; the target category includes a living body category or multiple non-living body categories.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the liveness detection model as described in any one of claims 1 to 7 or the liveness detection method as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the liveness detection model according to any one of claims 1 to 7 or the liveness detection method according to claim 8 is implemented.

Images