Live detection method, live detection network model training method and device

Abstract

The present invention provides a liveness detection method, a liveness detection network model training method, and equipment, belonging to the field of intelligent recognition technology. The liveness detection method includes: acquiring multiple modal images of an object to be detected; utilizing a liveness detection network model to extract features from each of the multiple modal images, performing weighted fusion on the features extracted from the multiple modal images, and outputting a classification result based on the fused features. The present invention performs liveness detection based on multimodal images, using cross-modal feature weighted fusion to enhance the effective features of features from different modalities. This method can effectively determine whether the acquired image is real, thereby achieving the purpose of liveness detection, improving liveness detection accuracy, and enhancing the ability to defend against attacks.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a liveness detection method, a liveness detection network model training method and a device. Background Art

[0002] In various face recognition application scenarios, liveness detection algorithms play a very important role. They are used to judge the differences between real people and live images, and prevent criminals from using reproduced images to impersonate human faces to attack the face recognition system.

[0003] Liveness detection methods can be categorized as dynamic liveness detection and silent liveness detection. Current silent liveness detection methods rely on single-frame RGB (R: Red, G: Green, B: Blue) images. However, these methods suffer from low detection accuracy and poor attack protection. Summary of the Invention

[0004] In view of this, the present invention provides a liveness detection method, a liveness detection network model training method and a device, which are used to solve the problems of low detection accuracy and poor attack defense capability in the current silent liveness detection method.

[0005] To solve the above technical problems, in a first aspect, the present invention provides a liveness detection method, comprising:

[0006] Acquire multiple modal images of the object to be detected;

[0007] Using a liveness detection network model, feature extraction is performed on the images of the multiple modalities respectively, and weighted fusion is performed on the features extracted from the images of the multiple modalities, and a classification result is output based on the fused features.

[0008] Optionally, the liveness detection network model is used to extract features from the images of the multiple modalities respectively, and the features extracted from the images of the multiple modalities are weightedly fused, and a classification result is output based on the fused features, including:

[0009] Performing shallow feature extraction on the images of the multiple modalities respectively;

[0010] Performing weighted fusion on shallow features of the images of the multiple modalities;

[0011] Perform deep feature extraction on the features after weighted fusion of shallow features;

[0012] Based on the extracted deep features, the classification results are output.

[0013] Optionally, the liveness detection network model is a squeezenet network model;

[0014] The respectively performing shallow feature extraction on the images of the multiple modalities includes:

[0015] Using the shallow feature extraction network in the squeezenet network model to perform shallow feature extraction on the images of the multiple modalities respectively;

[0016] The deep feature extraction of the features after weighted fusion of shallow features includes:

[0017] The deep feature extraction network in the squeezenet network model is used to extract deep features from the features obtained by weighted fusion of the shallow features.

[0018] Optionally, before performing feature extraction on the images of the multiple modalities using the living body detection network model, the method further includes:

[0019] Preprocessing the images of the multiple modalities respectively, wherein the preprocessing includes face detection;

[0020] For each of the images, performing weighted processing on the face region and the background region in the image to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region;

[0021] The extracting features from the images of the multiple modalities respectively includes:

[0022] Perform feature extraction on the image after the weighted processing.

[0023] Optionally, the background area is an area in the image within a preset range from the face area.

[0024] Optionally, the weighted fusion of shallow features of the images of the multiple modalities includes:

[0025] After mean pooling, the shallow features of each modality image are mapped to a shallow feature value;

[0026] Performing a softmax operation on all the shallow feature values and mapping them into a first value, where the first value is a value between 0 and 1;

[0027] For each modality image, a shallow feature weight value is determined according to the corresponding first value.

[0028] In a second aspect, the present invention further provides a method for training a liveness detection network model, comprising:

[0029] Acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities;

[0030] Based on the model training set, a liveness detection network model is trained and obtained, and the liveness detection network model is any of the liveness detection network models mentioned above.

[0031] Optionally, the training of a liveness detection network model based on the model training set includes:

[0032] Based on the model training set, the liveness detection network model is trained using a hierarchical loss function; wherein,

[0033] When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is:

[0034]

[0035] in:

[0036]

[0037] Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN is the average of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set, and p is the predicted value output by the liveness detection network model based on the image of the first detection object;

[0038] When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is:

[0039]

[0040] in:

[0041]

[0042] Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN' is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as attacks in the model training set, and p' is the predicted value output by the liveness detection network model based on the images of the second detection object.

[0043] In a third aspect, the present invention further provides a living body detection device, comprising:

[0044] An acquisition module, used to acquire images of multiple modalities of the object to be detected;

[0045] The detection module is used to use a liveness detection network model to extract features from the images of the multiple modalities respectively, perform weighted fusion on the features extracted from the images of the multiple modalities, and output a classification result based on the fused features.

[0046] Optionally, the detection module includes:

[0047] A shallow feature extraction unit, configured to perform shallow feature extraction on the images of the multiple modalities respectively;

[0048] A shallow feature weighted fusion unit, configured to perform weighted fusion on the shallow features of the images of the multiple modalities;

[0049] The deep feature extraction unit is used to extract deep features from the features after weighted fusion of shallow features;

[0050] The classification unit is used to output classification results based on the extracted deep features.

[0051] Optionally, the liveness detection network model is a squeezenet network model;

[0052] The shallow feature extraction unit is used to respectively perform shallow feature extraction on the images of the multiple modalities using the shallow feature extraction network in the squeezenet network model;

[0053] The deep feature extraction unit is used to use the deep feature extraction network in the squeezenet network model to perform deep feature extraction on the features obtained by weighted fusion of the shallow features.

[0054] Optionally, the living body detection device further includes:

[0055] A preprocessing module, configured to preprocess the images of the multiple modalities respectively, wherein the preprocessing includes face detection;

[0056] a weighted processing module, configured to perform weighted processing on the face region and the background region in each of the images to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region;

[0057] The detection module is used to extract features from the image after the weighted processing.

[0058] Optionally, the background area is an area in the image within a preset range from the face area.

[0059] Optionally, the shallow feature weighted fusion unit includes:

[0060] The first mapping subunit is used to map the shallow features of the image of each modality into a shallow feature value through mean pooling;

[0061] A second mapping subunit is configured to perform a softmax operation on all the shallow feature values and map them into a first value, where the first value is a value between 0 and 1;

[0062] The weighted weight determination subunit is used to determine the shallow feature weight value according to the corresponding first numerical value for each modality image.

[0063] In a fourth aspect, the present invention further provides a living body detection network model training device, comprising:

[0064] A training set acquisition module is used to acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities;

[0065] A training module is used to train a liveness detection network model based on the model training set, where the liveness detection network model is any of the liveness detection network models mentioned above.

[0066] Optionally, the training module is used to train the living body detection network model based on the model training set using a hierarchical loss function; wherein,

[0067] When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is:

[0068]

[0069] in:

[0070]

[0071] Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN is the average of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set, and p is the predicted value output by the liveness detection network model based on the image of the first detection object;

[0072] When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is:

[0073]

[0074] in:

[0075]

[0076] Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN' is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as attacks in the model training set, and p' is the predicted value output by the liveness detection network model based on the images of the second detection object.

[0077] In a fifth aspect, the present invention also provides an electronic device comprising a memory, a processor, and a program stored on the memory and runnable on the processor; when the processor executes the program, it implements the steps of any one of the above-mentioned liveness detection methods or the steps of any one of the above-mentioned liveness detection network model training methods.

[0078] In a sixth aspect, the present invention also provides a readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in any of the above-mentioned liveness detection methods or the steps in any of the above-mentioned liveness detection network model training methods.

[0079] The beneficial effects of the above technical solution of the present invention are as follows:

[0080] In an embodiment of the present invention, liveness detection is performed based on multimodal images, and the effective features of different modal features are enhanced by using cross-modal feature weighted fusion. This can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 Schematic diagram of a flow chart of a liveness detection method in Embodiment 1 of the present invention;

[0082] Figure 2 Schematic diagram of a process of performing liveness detection using a liveness detection network model in an embodiment of the present invention;

[0083] Figure 3 Schematic diagram of another liveness detection method according to an embodiment of the present invention;

[0084] Figure 4 This is a flow chart of a method for training a liveness detection network model in the second embodiment of the present invention;

[0085] Figure 5 Schematic diagram of another method for training a liveness detection network model in an embodiment of the present invention;

[0086] Figure 6 This is a schematic structural diagram of a living body detection device in Embodiment 3 of the present invention;

[0087] Figure 7 This is a structural diagram of a living body detection network model training device in embodiment 4 of the present invention;

[0088] Figure 8 This is a schematic structural diagram of an electronic device in a fifth embodiment of the present invention;

[0089] Figure 9 This is a structural diagram of an electronic device in Example 6 of the present invention. DETAILED DESCRIPTION

[0090] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0091] In the related art, a method for detecting live faces based on near-infrared and visible light binocular cameras is disclosed, belonging to the field of receiver vision and security. This method, belonging to the field of computer multimedia technology, is a binocular liveness detection method based on visible light and near-infrared cameras, comprising the following steps: S1: extracting illumination robust features, dividing the images received by the VIS and NIR cameras into blocks, calculating a histogram for each image block, and obtaining the histogram features of each image block; for each image block, learning its projection direction to maximize the correlation coefficient, and calculating its projection size and correlation; establishing a correlation confidence map to automatically remove image blocks that are of no value to the anti-spoofing detection system. This invention determines whether the detected object is a real face based on the correlation shown by the detection object under the VIS and NIR spectra, thereby effectively resisting face spoofing attacks and being able to effectively deal with various forms of face spoofing attacks, such as photos, video playback, and three-dimensional face masks. However, this solution only uses simple means to determine liveness detection, resulting in poor performance.

[0092] Another related technology involves a binocular camera-based face liveness detection method. This method uses a Fourier transform of infrared facial images to calculate the average value for liveness detection, while a deep neural network performs face recognition in a serial process. This solution uses the Fourier transform average method for liveness detection, resulting in a relatively primitive and simple calculation process, making it a simple liveness detection task.

[0093] In view of the shortcomings of the liveness detection methods in the related art, an embodiment of the present invention provides a method for liveness detection based on multimodal images.

[0094] See also Figure 1 , Figure 1 A flow chart of a liveness detection method provided in Example 1 of the present invention includes the following steps:

[0095] Step 11: Acquire multiple modal images of the object to be detected;

[0096] Step 12: Using the liveness detection network model, extract features from the images of the multiple modalities respectively, perform weighted fusion on the features extracted from the images of the multiple modalities, and output a classification result based on the fused features.

[0097] The multiple modal images include at least an original RGB image (specifically, a high-dynamic range (HDR) image), which is also a color image, and at least one of an infrared (IR) image and a depth map. The IR image may be an estimated IR image based on the original RGB image, and the depth map may be an estimated depth map based on the original RGB image. The infrared image may be a near-infrared image.

[0098] The images of multiple modalities of the object to be detected can be acquired simultaneously. For example, the original RGB image and the image of another modality can be acquired simultaneously based on a binocular camera. In addition, in addition to being images of the object to be detected acquired at the same time, the images of the object to be detected of the multiple modalities can also be images of the object to be detected acquired by an image acquisition device at the same position, angle, and / or height. For example, the images of multiple modalities of the object to be detected acquired at the same time by a camera at the same position, angle, and / or height.

[0099] In the embodiment of the present invention, the classification result may be a probability value, such as the probability of being a real person.

[0100] The present invention provides a multimodal image-based liveness detection method that uses weighted cross-modal feature fusion to enhance the effective features of different modalities. This method effectively determines whether the captured image is real, thereby achieving the purpose of liveness detection, improving liveness detection accuracy, and enhancing the ability to defend against attacks. It also facilitates system integration.

[0101] The liveness detection method may be to perform liveness detection on a human face.

[0102] The following example illustrates the above-mentioned liveness detection method.

[0103] Optionally, the liveness detection network model is used to extract features from the images of the multiple modalities respectively, and the features extracted from the images of the multiple modalities are weightedly fused, and a classification result is output based on the fused features, including:

[0104] Performing shallow feature extraction on the images of the multiple modalities respectively;

[0105] Performing weighted fusion on shallow features of the images of the multiple modalities;

[0106] Perform deep feature extraction on the features after weighted fusion of shallow features;

[0107] Based on the extracted deep features, the classification results are output.

[0108] That is to say, using the liveness detection network model, shallow feature extraction is performed on multiple modal images of the detection object respectively, and then the shallow features extracted from the multiple modal images are weightedly fused, and then deep feature extraction is performed. Finally, classification is performed based on the extracted deep features to obtain the classification results.

[0109] Further optionally, the liveness detection network model is a squeezenet network model;

[0110] The respectively performing shallow feature extraction on the images of the multiple modalities includes:

[0111] Using the shallow feature extraction network in the squeezenet network model to perform shallow feature extraction on the images of the multiple modalities respectively;

[0112] The deep feature extraction of the features after weighted fusion of shallow features includes:

[0113] The deep feature extraction network in the squeezenet network model is used to extract deep features from the features obtained by weighted fusion of the shallow features.

[0114] In the embodiment of the present invention, squeezenet is used as the basic network for liveness detection. Specifically, Figure 2 As shown, the shallow feature extraction network includes modules fire1 to fire3 in the Fire structure of the squeezenet network model, and the deep feature extraction network includes modules fire4 to fire8 in the Fire structure of the squeezenet network model. In other words, in this embodiment of the application, the Fire structure of the basic network squeezenet is used for feature extraction to construct a liveness detection network.

[0115] When the object to be detected has images of two modalities, two shallow feature extraction networks are used to perform shallow feature extraction on the two modal images respectively. In other words, if the object to be detected has images of several modalities, several shallow feature extraction networks are used to perform shallow feature extraction on the images of each modality respectively.

[0116] The liveness detection method provided by the embodiment of the present invention can be run in a terminal side environment or in a central processing unit (CPU) side environment.

[0117] Among them, shallow feature extraction can also be called first-layer feature extraction, and deep feature extraction can also be called second-layer feature extraction.

[0118] Of course, the liveness detection network model is not limited to the network implemented using the Fire structure, and can also be implemented using, for example, a common convolutional layer or a resnet block structure.

[0119] Optionally, before performing feature extraction on the images of the multiple modalities using the living body detection network model, the method further includes:

[0120] Preprocessing the images of the multiple modalities respectively, wherein the preprocessing includes face detection;

[0121] For each of the images, performing weighted processing on the face region and the background region in the image to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region;

[0122] The extracting features from the images of the multiple modalities respectively includes:

[0123] Perform feature extraction on the image after the weighted processing.

[0124] Wherein, the preprocessing also includes face alignment.

[0125] Specifically, before respectively performing shallow feature extraction on the images of the multiple modalities, the method further includes:

[0126] For each image, preprocessing including face detection is performed, and weighted processing is performed on the face area and background area in the image.

[0127] Optionally, the background area is an area in the image that is within a preset range from the face area, that is, the background area is the face edge area, or the area surrounding the face.

[0128] When using weighted convolution to weight the facial image in the first convolution, you need to specify the parameter board. The value of board represents the proportion of the facial edge area in the image. For example, the value of board can be between 0.1 and 0.2. Weighted convolution generates a single channel of weight information and multiplies it with the corresponding pixel position.

[0129] Furthermore, the preprocessing also includes face alignment. Specifically, during preprocessing, the detected face area can be scaled to a preset size and the face area can be adjusted to the center of the image. Then, the value of the parameter board is specified. Finally, the weighted convolution generates a single channel weight information and multiplies it with the corresponding position pixel, and the weight of the area of height*(1-board*2)*width*(1-board*2) pixels in the center of the image (where height and width are the height and width of the image respectively) is set to 1, and the weight of the background area is set to a value less than 1 but greater than 0, for example (1-2*board).

[0130] The face region parameter board is adjusted based on the image. Specifically, it can vary depending on the size of the face in the image. Therefore, during image preprocessing, it is necessary to record the proportion of the face region in the image so that the board value can be set accordingly.

[0131] In another optional specific implementation, during preprocessing, after the face is detected, the size of the face area in the image can be adjusted and the image can be cropped so that the face area in the preprocessed image is located in the center and the width and height of the face area in the image are both (1-board*2), that is, the width and height ratios of other areas except the face area are both the values of the board. Thus, the weighted convolution generates a single channel of weight information and multiplies it with the pixels at the corresponding position, and the weight of the area of height*(1-board*2)*width*(1-board*2) pixels in the center of the image (where height and width are the height and width of the image, respectively) is set to 1, and the weights of other pixels are set to values less than 1 but greater than 0, such as (1-2*board).

[0132] In this embodiment of the present invention, the weight w of the background area, i.e., the area surrounding the face, is dynamically adjusted based on the proportion of the face area. The adjustment method is w = 1 - (2 * board). However, this parameter adjustment method is not limited to this method and can be modified based on the characteristics and distribution of the image.

[0133] Of course, in other optional specific implementations, the images of the multiple modalities may be preprocessed separately before the liveness detection network model is used to extract features from the images of the multiple modalities.

[0134] See also Figure 2 ,The features output by the weighted convolution layer are calculated by three cascaded Fire modules to obtain shallow features.

[0135] In the embodiment of the present invention, a face region weighting method is used to enable the network model to pay more attention to the image information of the face part while referring to the background information.

[0136] Optionally, the weighted fusion of shallow features of the images of the multiple modalities includes:

[0137] After average pooling, the shallow features of each modality image are mapped to a shallow feature value;

[0138] Performing a softmax operation on all the shallow feature values and mapping them into a first value, where the first value is a value between 0 and 1;

[0139] For each modality image, a shallow feature weight value is determined according to the corresponding first value.

[0140] In an embodiment of the present invention, the specific process of cross-modal feature weighted fusion includes: after obtaining the shallow features of images of multiple modalities, each feature map is mapped to a numerical value through average pooling, all feature values are mapped to (0, 1) through softmax operation, and then the mean of the image features from each modality is calculated, and the calculated mean values are used as the weighted weight values of the shallow features of the corresponding modality images. For example, when the images of the multiple modalities include images of two modalities, the original RGB image and the IR image, the mean m1 of the original RGB image features and the mean m2 of the IR image features are calculated respectively. Then, the shallow features of the original RGB image are multiplied by m1, and the shallow features of the IR image are multiplied by m2, and then a concat operation (for connecting two or more arrays) is performed and sent to the deep feature extraction network to obtain the classification result.

[0141] See also Figure 3 The following uses the two-modal images of the object to be detected acquired by a binocular camera as an example to illustrate the above-mentioned living body detection method:

[0142] S101: A binocular camera of a binocular camera acquires an image, where the binocular camera includes a color camera and a near-infrared camera;

[0143] S102: Determine whether there is a face in the image captured by the color camera. If not, go to S103; if yes, go to S104;

[0144] S103: Continue waiting;

[0145] S104: Determine whether there is a human face in the image captured by the near-infrared camera. If not, go to S105. If yes, go to S106.

[0146] S105: Abandon the previously acquired image and continue waiting; specifically, the image previously acquired by the color camera and the image acquired by the near-infrared camera may be abandoned at the same time;

[0147] S106: Preprocessing the two images acquired by the binocular camera, specifically, detecting facial parts in the two images, aligning the detected faces, and / or cropping the images;

[0148] S107: Input the two pre-processed images into the living body detection network model for classification judgment;

[0149] S108: Output the probability that the object to be detected in the two images is a living body.

[0150] In an embodiment of the present invention, the liveness detection network model is a convolutional neural network, which has the following advantages: according to the characteristics of the face liveness detection task, the face area and the background area are distinguished by weighted convolution, and the face area in the image is focused on; the cross-modal feature weighted fusion method is used to assign different weights to the features of different modalities, so that the modal features that are easy to distinguish and / or more important can better play a judgment role.

[0151] See also Figure 4 , Figure 4 : This is a flow chart of a method for training a liveness detection network model provided in the second embodiment of the present invention, which includes the following steps:

[0152] Step 41: Acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities;

[0153] Step 42: Based on the model training set, a liveness detection network model is trained to obtain the liveness detection network model, where the liveness detection network model is any one of the liveness detection network models described in the above embodiments.

[0154] Regarding detection objects, it's important to note that even for the same physical object, images captured at different times can be considered different detection objects. For example, if two sets of images of the same person's face are captured at different times (one set of images includes images from multiple modalities), the detection objects of these two sets of images can be considered different.

[0155] In an embodiment of the present invention, the model training data used when training the liveness detection network model includes images of multiple modalities of multiple detection objects. The trained liveness detection network model can use cross-modal feature weighted fusion to enhance the effective features of different modal features, and can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks.

[0156] Optionally, the training of a liveness detection network model based on the model training set includes:

[0157] Based on the model training set, the liveness detection network model is trained using a hierarchical loss function; wherein,

[0158] When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is:

[0159]

[0160] in:

[0161]

[0162] Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set (that is, the maximum predicted value in the same batch), MEAN is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set (that is, the average value of the predicted values marked as real people in the same batch), and p is the predicted value output by the liveness detection network model based on the image of the first detection object;

[0163] When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is:

[0164]

[0165] in:

[0166]

[0167] Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value of the predicted value output by the liveness detection network model based on the data in the model training set (that is, the minimum predicted value in the same batch), MEAN' is the average value of the predicted value output by the liveness detection network model based on the image of the detection object marked as an attack in the model training set (that is, the average value of the predicted value marked as an attack in the same batch), and p' is the predicted value output by the liveness detection network model based on the image of the second detection object.

[0168] In liveness detection scenarios, difficult samples can easily cause neural networks to have difficulty converging. The hierarchical loss function proposed in the embodiments of the present invention distinguishes between training samples of different confidence levels, effectively solving the problem of network convergence difficulties after long-term training. In other words, the hierarchical loss function can effectively reduce the impact of hard samples on loss, accelerate the convergence of the network model, and reduce the risk of model generalization.

[0169] In other words, the embodiment of the present invention uses a hierarchical loss function to train the network model to address the problem that the overall classification effect deteriorates due to excessive focus on difficult examples (i.e., samples that are difficult to distinguish) during network model training.

[0170] The loss function for the training data when the prediction is wrong is the cross entropy loss.

[0171] Before model training, it is necessary to count the proportion of positive examples in the model training set (i.e., training data) to obtain the split point s. For example, when the positive examples account for 50%, the split point s = 0.5. In addition, the detection objects in the model training set need to be labeled to distinguish between real people and attacks, for example, real people are marked as "1" and attacks are marked as "0", that is, each detection object image in the model training set is labeled with a label. Positive examples, that is, the detection objects in the model training set are real people; attacks, that is, the detection objects in the model training set are not real people, but fake people, such as photos, videos, three-dimensional models, etc. Predicted as real people, that is, the predicted value output by the network model indicates that the corresponding detection object is a real person; predicted as attack, that is, the predicted value output by the network model indicates that the corresponding detection object is an attack, not a real person.

[0172] See also Figure 5 The main processes of the liveness detection network model training method provided by the embodiment of the present invention are as follows:

[0173] S201: Obtain a model training set, where the model training set includes images of multiple detection objects, and each detection object image includes images of multiple modalities. For example, when using a binocular camera to capture images of the detection objects, a pair of images of different modalities may be captured for each detection object.

[0174] S202: Preprocess the images in the model training set and label the images in the model training set as real or attack according to the actual situation;

[0175] S203: Using the pre-processed and labeled images in the model training set, the liveness detection network model is trained to obtain a trained liveness detection network model.

[0176] The trained liveness detection network model can be used to implement the liveness detection method in the above embodiment.

[0177] In the embodiment of the present invention, the specific process of the liveness detection network model training method may be:

[0178] Step 1: Obtain a model training set (or model training data set), which includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities; images of the same physical object collected at different times or at different angles can be considered as different detection objects.

[0179] Step 2: Preprocess the images in the model training set, including face detection, alignment, and cropping and / or resizing the images. The face area in the preprocessed images is located in the middle. The images in the model training set are labeled according to the actual situation. For example, "1" is used to indicate that the image is an image of a real person, and "0" is used to indicate that the image is an image of a fake person. At the same time, it is also necessary to record the proportional relationship of the face area in the image. If the proportional relationship of the face areas of the images of different detection objects in the model training set is different, it is necessary to record the proportional relationship of the face areas of the images of each detection object separately. For example, the proportional relationship can be used as data label information to record the proportional relationship of the face areas of the images of each detection object.

[0180] Step 3: Set the board value of the area surrounding the face (i.e., the background area), that is, set the proportion of the area surrounding the face in the image. Specifically, the board value can be set according to the proportional relationship of the face area in the image. If the proportion of the face area of the images of different detection objects in the model training set is different, then the board value of the images of different detection objects needs to be set according to the data label information that records the proportional relationship of the face area of the images of the corresponding detection objects. Please refer to Figure 2 , weighted convolution is used in the first convolution layer of the pre-built liveness detection network model to weight the face area in the image and give higher weight to the middle face area.

[0181] Step 4: Please refer to Figure 2 ,Use the shallow feature extraction network (fire1~fire3) in the pre-constructed liveness detection network model to perform shallow feature extraction on the image after weighted convolution processing of the first convolutional layer.

[0182] Step 5: Please refer to Figure 2 , set the weight value of the image of each modality, perform cross-modal feature weighted fusion according to the set weight value, input the shallow feature fusion of each modality into the deep feature extraction network, and finally train a globally optimal classification network model through the hierarchical loss function to use the classification network model for liveness detection.

[0183] The embodiments of the present invention enhance the learning ability of the liveness detection network model (a neural network) through the methods of weighted convolution, cross-modal feature weighted fusion and hierarchical loss function, reduce the risk of model generalization, and effectively improve the effect of liveness detection.

[0184] The embodiment of the present invention provides a technical solution corresponding to the above-mentioned embodiment 1, having the same inventive concept, and can achieve the same technical effect. For details, please refer to the above-mentioned embodiment 1, which will not be repeated here.

[0185] See also Figure 6 , Figure 6 : is a schematic structural diagram of a liveness detection device provided in Embodiment 3 of the present invention. The liveness detection device 60 includes:

[0186] An acquisition module 61 is used to acquire images of multiple modalities of the object to be detected;

[0187] The detection module 62 is used to use the liveness detection network model to extract features from the images of the multiple modalities respectively, perform weighted fusion on the features extracted from the images of the multiple modalities, and output a classification result based on the fused features.

[0188] In an embodiment of the present invention, liveness detection is performed based on multimodal images, and the effective features of different modal features are enhanced by using cross-modal feature weighted fusion. This can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks.

[0189] Optionally, the detection module 62 includes:

[0190] A shallow feature extraction unit, configured to perform shallow feature extraction on the images of the multiple modalities respectively;

[0191] A shallow feature weighted fusion unit, configured to perform weighted fusion on the shallow features of the images of the multiple modalities;

[0192] The deep feature extraction unit is used to extract deep features from the features after weighted fusion of shallow features;

[0193] The classification unit is used to output classification results based on the extracted deep features.

[0194] Optionally, the liveness detection network model is a squeezenet network model;

[0195] The shallow feature extraction unit is used to respectively perform shallow feature extraction on the images of the multiple modalities using the shallow feature extraction network in the squeezenet network model;

[0196] The deep feature extraction unit is used to use the deep feature extraction network in the squeezenet network model to perform deep feature extraction on the features obtained by weighted fusion of the shallow features.

[0197] Optionally, the living body detection device further includes:

[0198] A preprocessing module, configured to preprocess the images of the multiple modalities respectively, wherein the preprocessing includes face detection;

[0199] a weighted processing module, configured to perform weighted processing on the face region and the background region in each of the images to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region;

[0200] The detection module 62 is used to perform feature extraction on the image after the weighted processing.

[0201] Optionally, the background area is an area in the image within a preset range from the face area.

[0202] Optionally, the shallow feature weighted fusion unit includes:

[0203] The first mapping subunit is used to map the shallow features of the image of each modality into a shallow feature value through mean pooling;

[0204] A second mapping subunit is configured to perform a softmax operation on all the shallow feature values and map them into a first value, where the first value is a value between 0 and 1;

[0205] The weighted weight determination subunit is used to determine the shallow feature weight value according to the corresponding first numerical value for each modality image.

[0206] The embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 1, so it will not be described in detail here. Please refer to the above-mentioned embodiment 1 for details.

[0207] See also Figure 7 , Figure 7 : is a structural diagram of a liveness detection network model training device provided by a fourth embodiment of the present invention. The liveness detection network model training device 70 includes:

[0208] A training set acquisition module 71 is configured to acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities;

[0209] The training module 72 is configured to train a liveness detection network model based on the model training set, where the liveness detection network model is any of the liveness detection network models mentioned above.

[0210] In an embodiment of the present invention, the model training data used when training the liveness detection network model includes images of multiple modalities of multiple detection objects. The trained liveness detection network model can use cross-modal feature weighted fusion to enhance the effective features of different modal features, and can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks.

[0211] Optionally, the training module 72 is used to train the living body detection network model based on the model training set using a hierarchical loss function; wherein,

[0212] When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is:

[0213]

[0214] in:

[0215]

[0216] Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN is the average of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set, and p is the predicted value output by the liveness detection network model based on the image of the first detection object;

[0217] When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is:

[0218]

[0219] in:

[0220]

[0221] Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN' is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as attacks in the model training set, and p' is the predicted value output by the liveness detection network model based on the images of the second detection object.

[0222] This embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 2, so it will not be described in detail here. Please refer to the above-mentioned embodiment 2 for details.

[0223] See also Figure 8 , Figure 88 is a schematic diagram of the structure of an electronic device provided in a fifth embodiment of the present invention. The electronic device 80 includes a processor 81, a memory 82, and a program stored in the memory 82 and executable on the processor 81. When the processor 81 executes the program, the following steps are implemented:

[0224] Acquire multiple modal images of the object to be detected;

[0225] Using a liveness detection network model, feature extraction is performed on the images of the multiple modalities respectively, and weighted fusion is performed on the features extracted from the images of the multiple modalities, and a classification result is output based on the fused features.

[0226] In an embodiment of the present invention, liveness detection is performed based on multimodal images, and the effective features of different modal features are enhanced by using cross-modal feature weighted fusion. This can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks.

[0227] Optionally, the processor 81 may further implement the following steps when executing the program:

[0228] The liveness detection network model is used to extract features from the images of the multiple modalities, perform weighted fusion on the features extracted from the images of the multiple modalities, and output classification results based on the fused features, including:

[0229] Performing shallow feature extraction on the images of the multiple modalities respectively;

[0230] Performing weighted fusion on shallow features of the images of the multiple modalities;

[0231] Perform deep feature extraction on the features after weighted fusion of shallow features;

[0232] Based on the extracted deep features, the classification results are output.

[0233] Optionally, the liveness detection network model is a squeezenet network model; when the processor 81 executes the program, the following steps may also be implemented:

[0234] The respectively performing shallow feature extraction on the images of the multiple modalities includes:

[0235] Using the shallow feature extraction network in the squeezenet network model to perform shallow feature extraction on the images of the multiple modalities respectively;

[0236] The deep feature extraction of the features after weighted fusion of shallow features includes:

[0237] The deep feature extraction network in the squeezenet network model is used to extract deep features from the features obtained by weighted fusion of the shallow features.

[0238] Optionally, the processor 81 may further implement the following steps when executing the program:

[0239] Before extracting features from the images of the multiple modalities using the living body detection network model, the method further includes:

[0240] Preprocessing the images of the multiple modalities respectively, wherein the preprocessing includes face detection;

[0241] For each of the images, performing weighted processing on the face region and the background region in the image to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region;

[0242] The extracting features from the images of the multiple modalities respectively includes:

[0243] Perform feature extraction on the image after the weighted processing.

[0244] Optionally, the background area is an area in the image within a preset range from the face area.

[0245] Optionally, the processor 81 may further implement the following steps when executing the program:

[0246] The weighted fusion of shallow features of the images of the multiple modalities includes:

[0247] After mean pooling, the shallow features of each modality image are mapped to a shallow feature value;

[0248] Performing a softmax operation on all the shallow feature values and mapping them into a first value, where the first value is a value between 0 and 1;

[0249] For each modality image, a shallow feature weight value is determined according to the corresponding first value.

[0250] The specific working process of the embodiment of the present invention is consistent with that of the above-mentioned method embodiment 1, so it will not be repeated here. For details, please refer to the description of the method steps in the above-mentioned embodiment 1.

[0251] See also Figure 9 , Figure 91 is a schematic diagram of the structure of an electronic device provided in a sixth embodiment of the present invention. The electronic device 90 includes a processor 91, a memory 92, and a program stored in the memory 92 and executable on the processor 91. When the processor 91 executes the program, the following steps are implemented:

[0252] Acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities;

[0253] Based on the model training set, a liveness detection network model is trained and obtained, and the liveness detection network model is any of the liveness detection network models mentioned above.

[0254] In an embodiment of the present invention, the model training data used when training the liveness detection network model includes images of multiple modalities of multiple detection objects. The trained liveness detection network model can use cross-modal feature weighted fusion to enhance the effective features of different modal features, and can effectively determine whether the acquired image is a real person, thereby achieving the purpose of liveness detection, improving the accuracy of liveness detection, and enhancing the ability to defend against attacks.

[0255] Optionally, the processor 91 may further implement the following steps when executing the program:

[0256] The training of a liveness detection network model based on the model training set includes:

[0257] Based on the model training set, the liveness detection network model is trained using a hierarchical loss function; wherein,

[0258] When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is:

[0259]

[0260] in:

[0261]

[0262] Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN is the average of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set, and p is the predicted value output by the liveness detection network model based on the image of the first detection object;

[0263] When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is:

[0264]

[0265] in:

[0266]

[0267] Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN' is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as attacks in the model training set, and p' is the predicted value output by the liveness detection network model based on the images of the second detection object.

[0268] The specific working process of the embodiment of the present invention is consistent with that of the above-mentioned method embodiment 2, so it will not be repeated here. Please refer to the description of the method steps in the above-mentioned embodiment 2 for details.

[0269] Embodiment 7 of the present invention provides a readable storage medium having a program stored thereon. When executed by a processor, the program implements the steps of any of the liveness detection methods in Embodiment 1 or the steps of any of the liveness detection network model training methods in Embodiment 2. For details, please refer to the descriptions of the method steps in the corresponding embodiments above.

[0270] The terminals in the embodiments of the present invention can be either wireless or wired terminals. A wireless terminal can be a device that provides voice and / or other service data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. A wireless terminal can communicate with one or more core networks via a Radio Access Network (RAN). A wireless terminal can be a mobile terminal, such as a mobile phone (also known as a "cellular" phone) or a computer with a mobile terminal. For example, a wireless terminal can be a portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile device that exchanges voice and / or data with a radio access network. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, and Personal Digital Assistants (PDAs). A wireless terminal may also be referred to as a system, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote terminal, access terminal, user terminal, user agent, user device or user equipment, without limitation herein.

[0271] The above-mentioned readable storage media include computer-readable storage media. Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0272] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting a living body, characterized in that: include: Acquire multiple modal images of the object to be detected; Using a liveness detection network model, extracting features from the images of the multiple modalities respectively, performing weighted fusion on the features extracted from the images of the multiple modalities, and outputting a classification result based on the fused features; Before extracting features from the images of the multiple modalities using the living body detection network model, the method further includes: preprocessing the images of the multiple modalities, wherein the preprocessing includes face detection; For each of the images, performing weighted processing on the face region and the background region in the image to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region; The extracting features from the images of the multiple modalities respectively includes: Performing feature extraction on the image after the weighted processing; The preprocessing also includes face alignment. During preprocessing, the detected face area is scaled to a preset size and adjusted to the center of the image. Then, the value of the parameter board is specified. Finally, weighted convolution generates a single channel of weight information and multiplies it with the corresponding position pixels. The weight of the area with a size of height*(1-board*2)*width*(1-board*2) pixels in the center of the image is set to 1, and the weight of the background area is set to a value less than 1 but greater than 0. Among them, the parameter board is adjusted according to different images; it varies according to the size of the facial part in the image; in image preprocessing, the proportional relationship of the facial area in the image is also recorded so that the board value can be set according to the proportion of the facial area in the image.

2. The method according to claim 1, characterized in that The liveness detection network model is used to extract features from the images of the multiple modalities, perform weighted fusion on the features extracted from the images of the multiple modalities, and output classification results based on the fused features, including: Performing shallow feature extraction on the images of the multiple modalities respectively; Performing weighted fusion on shallow features of the images of the multiple modalities; Perform deep feature extraction on the features after weighted fusion of shallow features; Based on the extracted deep features, the classification results are output.

3. The method according to claim 2, characterized in that The liveness detection network model is a squeezenet network model; The respectively performing shallow feature extraction on the images of the multiple modalities includes: Using the shallow feature extraction network in the squeezenet network model to perform shallow feature extraction on the images of the multiple modalities respectively; The deep feature extraction of the features after weighted fusion of shallow features includes: The deep feature extraction network in the squeezenet network model is used to extract deep features from the features obtained by weighted fusion of the shallow features.

4. The method according to claim 1, wherein The background area is an area in the image that is within a preset range from the face area.

5. The method according to claim 2 or 3, characterized in that The weighted fusion of shallow features of the images of the multiple modalities includes: After mean pooling, the shallow features of each modality image are mapped to a shallow feature value; Performing a softmax operation on all the shallow feature values and mapping them into a first value, where the first value is a value between 0 and 1; For each modality image, a shallow feature weight value is determined according to the corresponding first value.

6. A method for training a liveness detection network model, characterized in that: include: Acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities; Based on the model training set, a liveness detection network model is trained and obtained, wherein the liveness detection network model is the liveness detection network model according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The training of a liveness detection network model based on the model training set includes: Based on the model training set, the liveness detection network model is trained using a hierarchical loss function; wherein, When the predicted and true labels are both real people, the loss function corresponding to the image of the first detection object is: in: Wherein, the first detection object is any detection object marked as a real person in the model training set, s is a separation point obtained by counting the proportion of positive examples in the model training set, MAX is the maximum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN is the average of the predicted values output by the liveness detection network model based on the images of the detection objects marked as real people in the model training set, and p is the predicted value output by the liveness detection network model based on the image of the first detection object; When the predicted and true labels are both attacks, the loss function corresponding to the image of the second detection object is: in: Among them, the second detection object is any detection object marked as an attack in the model training set, s is the separation point obtained by counting the proportion of positive examples in the model training set, MIN is the minimum value among the predicted values output by the liveness detection network model based on the data in the model training set, MEAN′ is the average value of the predicted values output by the liveness detection network model based on the images of the detection objects marked as attacks in the model training set, and p′ is the predicted value output by the liveness detection network model based on the images of the second detection object.

8. A living body detection device, characterized in that: include: An acquisition module, used to acquire images of multiple modalities of the object to be detected; a detection module, configured to extract features from the images of the multiple modalities using a liveness detection network model, perform weighted fusion on the features extracted from the images of the multiple modalities, and output a classification result based on the fused features; Before extracting features from the images of the multiple modalities using the living body detection network model, the method further includes: preprocessing the images of the multiple modalities, wherein the preprocessing includes face detection; For each of the images, performing weighted processing on the face region and the background region in the image to obtain a weighted image, wherein the weight value of the face region is greater than the weight value of the background region; The extracting features from the images of the multiple modalities respectively includes: Performing feature extraction on the image after the weighted processing; The preprocessing also includes face alignment. During preprocessing, the detected face area is scaled to a preset size and adjusted to the center of the image. Then, the value of the parameter board is specified. Finally, weighted convolution generates a single channel of weight information and multiplies it with the corresponding position pixels. The weight of the area with a size of height*(1-board*2)*width*(1-board*2) pixels in the center of the image is set to 1, and the weight of the background area is set to a value less than 1 but greater than 0. Among them, the parameter board is adjusted according to different images; it varies according to the size of the facial part in the image; in image preprocessing, the proportional relationship of the facial area in the image is also recorded so that the board value can be set according to the proportion of the facial area in the image.

9. A living body detection network model training device, characterized in that: include: A training set acquisition module is used to acquire a model training set, wherein the model training set includes images of multiple detection objects, and each image of the detection object includes images of multiple modalities; A training module is used to train a liveness detection network model based on the model training set, wherein the liveness detection network model is the liveness detection network model according to any one of claims 1 to 5.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor; When the processor executes the program, the steps of the liveness detection method according to any one of claims 1 to 5 are implemented, or the steps of the liveness detection network model training method according to any one of claims 6 to 7 are implemented.

11. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the liveness detection method according to any one of claims 1 to 5 or the steps of the liveness detection network model training method according to any one of claims 6 to 7 are implemented.

Images