Live detection method, device, apparatus and storage medium

Abstract

The application discloses a kind of living body detection method, device, equipment and storage medium, comprising: obtaining the face image to be identified, and the face image to be identified is converted into thumbnail face image;According to the image area information of local detail image in thumbnail face image determination face image to be identified;According to image area information from face image to be identified extraction local detail image;According to local detail image determination face image to be identified corresponding local texture feature and timing feature;According to local texture feature and timing feature determination face image to be identified living body prediction probability.The application simultaneously takes face image to be identified and thumbnail face image as input, according to the image area information of thumbnail face image extraction local detail image, according to local detail image determination local texture feature and timing feature, according to local texture feature and timing feature determination living body prediction probability, while reducing the data amount of calculation, guarantee the accuracy of living body detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of living body detection, and in particular to a living body detection method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of deep learning technology, face recognition technology has been widely used in security authentication, financial payment and other fields, and the security of face recognition is more important. Face spoofing attack is a major security threat to the development of face recognition technology. Current face spoofing attack methods mainly include: photo printing, video replay, simulation mask / mask and 3D face mold attack methods. Especially, some attackers use carefully designed simulation masks, which have very high deception and are difficult to detect living bodies. SUMMARY

[0003] The main purpose of the present application is to provide a living body detection method, device, equipment and storage medium, which aims to solve the technical problem that the accuracy and data processing amount of living body detection in the prior art are difficult to balance.

[0004] To achieve the above purpose, the present application provides a living body detection method, which comprises the following steps:

[0005] Obtain a face image to be recognized, and convert the face image to be recognized into a thumbnail face image;

[0006] Determine image region information of a local detail image in the face image to be recognized according to the thumbnail face image;

[0007] Extract the local detail image from the face image to be recognized according to the image region information;

[0008] Determine local texture features and time sequence features corresponding to the face image to be recognized according to the local detail image;

[0009] Determine a living body prediction probability of the face image to be recognized according to the local texture features and the time sequence features.

[0010] Optionally, the step of determining the image region information of the local detail image in the face image to be recognized according to the thumbnail face image comprises:

[0011] Feature extraction is performed on the thumbnail face image by an initial feature extraction module to obtain an initial feature extraction image;

[0012] Determine image position encoding according to the initial feature extraction image;

[0013] The encoder in the space-time attention module outputs feature encoding data according to the initial feature extraction image and the image position encoding.

[0014] outputting, by a decoder in the spatio-temporal attention module, a thumbnail feature vector according to the feature encoding data, the image position encoding and a previous thumbnail feature vector;

[0015] outputting, by a first feedforward neural network, image region information of a local detail image in the face image to be recognized according to the thumbnail feature vector.

[0016] Optionally, the image position encoding is determined according to the initial feature extraction map, comprising:

[0017] determining a vertical direction position encoding according to a vertical direction coordinate of the initial feature extraction map;

[0018] determining a horizontal direction position encoding according to a horizontal direction coordinate of the initial feature extraction map;

[0019] determining the image position encoding according to the vertical direction position encoding and the horizontal direction position encoding.

[0020] Optionally, the local texture feature and the temporal feature corresponding to the face image to be recognized are determined according to the local detail image, comprising:

[0021] extracting, by a local texture feature extraction module, a feature of the local detail image, and outputting the local texture feature corresponding to the face image to be recognized;

[0022] comparing, by a temporal feature extraction module, the local texture feature with a previous local texture feature, and outputting the temporal feature of the face image to be recognized.

[0023] Optionally, the temporal feature of the face image to be recognized is outputted by the temporal feature extraction module by comparing the local texture feature with the previous local texture feature, comprising:

[0024] determining, by the temporal feature extraction module, a local edge feature according to the local texture feature;

[0025] determining, by the temporal feature extraction module, a previous local edge feature according to the previous local texture feature, and determining a difference feature according to the local edge feature and the previous local edge feature;

[0026] outputting, by the temporal feature extraction module, the temporal feature of the face image to be recognized according to the local texture feature, the local edge feature, the previous local edge feature, the difference feature and a previous temporal feature.

[0027] Optionally, before the outputting the local texture feature corresponding to the to-be-identified face image by the local texture feature extraction module, the method further comprises:

[0028] extracting a local detail sample image from a face image data sample in a face image data sample set, and constructing an initial training data set according to the local detail sample image;

[0029] performing data enhancement processing on the local detail sample image in the initial training data set to obtain a target training data set;

[0030] training an initial local texture feature extraction module by using the target training data set to obtain a local texture feature extraction module.

[0031] Optionally, the determining the living body prediction probability of the to-be-identified face image according to the local texture feature and the time sequence feature comprises:

[0032] outputting a first living body prediction value of the to-be-identified face image according to the local texture feature by using a second feedforward neural network;

[0033] outputting a second living body prediction value of the to-be-identified face image according to the time sequence feature by using a third feedforward neural network;

[0034] determining the living body prediction probability of the to-be-identified face image according to the first living body prediction value and the second living body prediction value.

[0035] In addition, in order to achieve the above object, the application further provides a living body detection device, which comprises:

[0036] a conversion module, configured to acquire a to-be-identified face image and convert the to-be-identified face image into a thumbnail face image;

[0037] an information determination module, configured to determine image region information of a local detail image in the to-be-identified face image according to the thumbnail face image;

[0038] an extraction module, configured to extract the local detail image from the to-be-identified face image according to the image region information;

[0039] a feature determination module, configured to determine a local texture feature and a time sequence feature corresponding to the to-be-identified face image according to the local detail image;

[0040] an output module, configured to determine a living body prediction probability of the to-be-identified face image according to the local texture feature and the time sequence feature.

[0041] In addition, to achieve the above object, the present application further provides a living body detection device, which comprises a memory, a processor and a living body detection program stored in the memory and executable on the processor, and the living body detection program is configured to implement the steps of the living body detection method as described above.

[0042] In addition, to achieve the above object, the present application further provides a storage medium, which stores a living body detection program, and the living body detection program is executed by a processor to implement the steps of the living body detection method as described above.

[0043] The present application obtains a to-be-recognized face image, converts the to-be-recognized face image into a thumbnail face image, determines image region information of a local detail image in the to-be-recognized face image according to the thumbnail face image, extracts the local detail image from the to-be-recognized face image according to the image region information, determines a local texture feature and a time sequence feature corresponding to the to-be-recognized face image according to the local detail image, and determines a living body prediction probability of the to-be-recognized face image according to the local texture feature and the time sequence feature. The present application simultaneously takes the to-be-recognized face image and the thumbnail face image as inputs, extracts the local detail image from the to-be-recognized face image according to the image region information of the local detail image determined according to the thumbnail face image, determines the local texture feature and the time sequence feature according to the local detail image, and then determines the living body prediction probability of the to-be-recognized face image according to the local texture feature and the time sequence feature, thereby reducing the data calculation amount while ensuring the accuracy of living body detection. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 FIG. 1 is a structural schematic diagram of a living body detection device of a hardware running environment related to an embodiment scheme of the present application;

[0045] Figure 2 FIG. 2 is a flowchart of a first embodiment of a living body detection method of the present application;

[0046] Figure 3 FIG. 3 is a structural schematic diagram of a space-time attention module in an embodiment of the living body detection method of the present application;

[0047] Figure 4 FIG. 4 is a flowchart of a second embodiment of the living body detection method of the present application;

[0048] Figure 5 FIG. 5 is a whole structural schematic diagram of a model in an embodiment of the living body detection method of the present application;

[0049] Figure 6 FIG. 6 is a structural schematic diagram of a multi-dimensional feature extraction module in an embodiment of the living body detection method of the present application;

[0050] Figure 7A flowchart of a third embodiment of the living body detection method of the present application is shown in the figure.

[0051] Figure 8 A structure block diagram of a first embodiment of the living body detection device of the present application is shown in the figure.

[0052] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0054] Reference Figure 1 , Figure 1 A structure diagram of a living body detection device related to the hardware running environment of the embodiment of the present application is shown in the figure.

[0055] As Figure 1 shown, the living body detection device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art can understand Figure 1 that the structure shown in the figure does not constitute a limitation on the living body detection device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0057] As Figure 1 shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a living body detection program.

[0058] In Figure 1The network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 can be arranged in the live body detection device, and the live body detection program stored in the memory 1005 is called by the processor 1001, and the live body detection method provided by the embodiment of the application is executed.

[0059] The embodiment of the application provides a live body detection method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the live body detection method is shown in the figure.

[0060] In the embodiment, the live body detection method comprises the following steps:

[0061] Step S10: obtaining a to-be-recognized face image, and converting the to-be-recognized face image into a thumbnail face image.

[0062] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone and the like, or an electronic device, a live body detection device and the like capable of realizing the above functions. In the following, the live body detection device is taken as an example to illustrate the embodiment and the following embodiments.

[0063] It can be understood that the to-be-recognized face image can be a face image that needs to be subjected to live body detection, and a high-definition face image can be read from a camera at a preset frame interval as the to-be-recognized face image; the thumbnail face image can be a face image obtained by performing thumbnail processing on the to-be-recognized face image, and the to-be-recognized face image can be thumbnai led to a preset resolution to obtain the thumbnail face image.

[0064] As an implementation manner, a high-definition face image is read from a camera at a preset frame interval, and the read high-definition face image is taken as the to-be-recognized face image, and the to-be-recognized face image is thumbnai led to a preset resolution to obtain the thumbnail face image.

[0065] Step S20: determining image region information of a local detail image in the to-be-recognized face image according to the thumbnail face image.

[0066] It can be understood that the local detail image can be an image used for feature extraction and determination in the to-be-recognized face image, and the local detail image includes but is not limited to an eye image, a corner of eye image, a mouth image, a corner of mouth image, a face image and the like; the image region information can be coordinate position information of the local detail image.

[0067] Step S30: extracting the local detail image from the to-be-recognized face image according to the image region information.

[0068] It can be understood that the local detail image in the to-be-identified face image is located according to the image region information, and the local detail image in the to-be-identified face image is extracted according to the positioning result.

[0069] Step S40: determining local texture features and timing features corresponding to the to-be-identified face image according to the local detail image.

[0070] It can be understood that the local texture features can be features representing spatial characteristics such as face texture, shadow and depth; and the timing features can be features representing micro-expression actions and consistency of physiological features of the face skin.

[0071] Step S50: determining a living body prediction probability of the to-be-identified face image according to the local texture features and the timing features.

[0072] In a specific implementation, a high-definition face image captured by a camera is read as a to-be-identified face image at an interval of a preset number of frames, the to-be-identified face image is scaled to a preset resolution to obtain a scaled face image, the to-be-identified face image and the scaled face image are taken as inputs, the scaled face image is focused by a space-time attention module, thereby determining image region information of a local detail image in the to-be-identified face image, the local detail image is extracted from the to-be-identified face image according to the image region information, and the local detail image is obtained; local texture features of the local detail image are extracted, the local texture features of the local detail image are compared with previous local texture features, timing features are obtained, and a living body prediction probability of the to-be-identified face image is determined according to the local texture features and the timing features, wherein the previous local texture features are local texture features obtained by performing local texture feature extraction on a local detail image extracted from a previous to-be-identified face image.

[0073] Further, in order to reduce the calculation amount in the living body detection process and improve the efficiency of the living body detection, the step S20 includes: performing feature extraction on the scaled face image by an initial feature extraction module to obtain an initial feature extraction image; determining image position encoding according to the initial feature extraction image; outputting feature encoding data according to the initial feature extraction image and the image position encoding by an encoder in the space-time attention module; outputting a thumbnail feature vector according to the feature encoding data, the image position encoding and a previous thumbnail feature vector by a decoder in the space-time attention module; and outputting image region information of a local detail image in the to-be-identified face image according to the thumbnail feature vector by a first feedforward neural network.

[0074] It can be understood that the initial feature extraction module can be a module for extracting features from the thumbnail face image, the initial feature extraction module can be a convolutional neural network, for example, a resnet50 network; the initial feature extraction map can be a feature map extracted from the thumbnail face image by the initial feature extraction module; the spatio-temporal attention module adopts a transformer structure, and the spatio-temporal attention module includes an encoder and a decoder; the feature encoding data can be data output by the encoder according to the initial feature extraction image and image position encoding; the previous thumbnail feature vector can be a thumbnail feature vector corresponding to the thumbnail face image of the previous frame; and the first feedforward neural network can be an FFN.

[0075] Further, in order to reduce the calculation amount in the living body detection process and improve the efficiency of the living body detection, the image position encoding is determined according to the initial feature extraction map, including: determining a vertical direction position encoding according to a vertical direction coordinate of the initial feature extraction map; determining a horizontal direction position encoding according to a horizontal direction coordinate of the initial feature extraction map; and determining the image position encoding according to the vertical direction position encoding and the horizontal direction position encoding.

[0076] It can be understood that the vertical direction coordinate can be a coordinate in the vertical direction of the initial feature extraction map; the horizontal direction coordinate can be a coordinate in the horizontal direction of the initial feature extraction map; and the vertical direction position encoding and the horizontal direction position encoding constitute the image position encoding.

[0077] In a specific implementation, every 5 frames, a high-definition face image is read from a camera as a to-be-identified face image, the to-be-identified face image is reduced to a preset resolution to obtain a thumbnail face image, the thumbnail face image is input into an initial feature extraction module, the initial feature extraction module uses a resnet50 network (without the last two layers of Avgpool and Linear), and the thumbnail face image is input into the initial feature extraction module to obtain a first feature extraction map f∈R C×H×W wherein C = 2048, The first feature extraction map is reduced to 256 by using a 1x1conv convolutional network to obtain an initial feature extraction map z0∈R 256×H×W wherein wherein H0 is the height of the to-be-identified face image, and W0 is the width of the to-be-identified face image; the main body of the spatio-temporal attention module adopts a transformer structure, and the structure is as shown in Figure 3 ​As shown, the space-time attention module includes an encoder and a decoder. The input of the encoder is the feature map z0 output by the initial feature extraction module and the image position encoding PE. The input of the decoder is the output of the encoder, the image position encoding, and the thumbnail feature vector output by the decoder in the previous frame space-time attention module.

[0078] (1) The image position encoding PE calculation method can be: for the initial feature extraction map z0∈R 256×H×W The corresponding image position encoding is PE∈R 256×H×W wherein the first 128 dimensions represent vertical direction position encoding, and the last 128 dimensions represent horizontal direction position encoding. The calculation formula is:

[0079]

[0080] The horizontal direction position encoding is The calculation formula is:

[0081]

[0082] wherein h is the vertical direction coordinate of the initial feature extraction map, w is the horizontal direction coordinate of the initial feature extraction map, d h , d w are the dimensions of the vertical and horizontal directions, respectively.

[0083] (2) When the initial feature extraction map and the image position encoding are input into the encoder, the initial feature extraction map and the image position encoding need to be preprocessed. The initial feature extraction map is processed into feature map encoding src_embed, and the image position encoding is processed into position encoding pos_embed. In addition to the feature encoding data output by the encoder, the decoder also needs to input the previous thumbnail feature vector query_embed output by the decoder in the previous frame. Specifically: ① feature map encoding src_embed: through reshape operation, the z0 output by the initial feature extraction module is transformed into a tensor with dimensions of [H*W, 256], which is src_embed; ② position encoding pos_embed: through reshape operation, the image position encoding PE is transformed into a tensor with dimensions of [H*W, 256], which is pos_embed; before inputting into the encoder, the three variables Q (Query), K (Key), and V (Value) specific to the transformer need to be calculated, and the calculation formula is:

[0084] Query=Key=src_embed+pos_embed

[0085] Value=src_embed

[0086] Preceding thumbnail feature vector query_embed: refers to the feature vector output by the decoder of the previous frame, with a dimension of [N ROI ,256], where N ROI is the number of high-definition local image (i.e., local detail image) hyperparameters, and if it is the first frame, all are initialized to 0.

[0087] (3) The output of the encoder: the encoder is inferred through the self-attention module and the FFN forward network, and the output dimension is consistent with the input dimension, which is [H*W, 256], and the output feature encoding data is directly input to the decoder for the next inference.

[0088] (4) Decoder output processing method: the decoder is inferred through the multi-head attention module and the FFN forward network, and the output dimension is [N ROI ,256] of the thumbnail feature vector (here the first dimension represents that the maximum number of local detail images that can be focused on is N ROI ), which is the input of the decoder of the next frame; the feature vector output by the decoder also needs to pass through the first feedforward neural network FFN to obtain the coordinate prediction value of the local detail image, where the FFN adopts a fully connected layer linear, the hidden layer dimension is 512, and the output dimension is [N ROI ,4]; the local detail image is extracted from the face image to be recognized through the RoIAlign module according to the image region information; the RoIAlign module mainly includes an RoIAlign function, which adopts bilinear interpolation according to the face image to be recognized and the ROI region information (i.e., image region information) output by the spatio-temporal attention module; the input of the RoIAlign module is: the face image to be recognized scr, with a dimension of [3, H', W']; the ROI region information output by the spatio-temporal attention module, with a dimension of [N ROI ,4], where H', W' are the height and width of the face image to be recognized scr, and N ROI is the number of local detail images; the output of the RoIAlign module is a multi-dimensional tensor containing local detail image information, with a dimension of [N ROI *3, H s , W s ], where H s , W s are the height and width of the high-definition local image preset size, and N ROI is the number of local detail images (i.e., high-definition local detail images); the local texture features corresponding to the face image to be recognized are determined according to the local detail images, the time sequence features are determined according to the local texture features and the preceding local texture features, and the live prediction probability of the face image to be recognized is determined according to the time sequence features and the local texture features.

[0089] The embodiment obtains a to-be-identified face image, converts the to-be-identified face image into a thumbnail face image, determines image region information of a local detail image in the to-be-identified face image according to the thumbnail face image, extracts the local detail image from the to-be-identified face image according to the image region information, determines a local texture feature and a time sequence feature corresponding to the to-be-identified face image according to the local detail image, and determines a living body prediction probability of the to-be-identified face image according to the local texture feature and the time sequence feature. The embodiment simultaneously takes the to-be-identified face image and the thumbnail face image as inputs, extracts the local detail image from the to-be-identified face image according to the image region information of the local detail image determined according to the thumbnail face image, determines the local texture feature and the time sequence feature according to the local detail image, and then determines the living body prediction probability of the to-be-identified face image according to the local texture feature and the time sequence feature, thereby reducing the data calculation amount while ensuring the accuracy of living body detection.

[0090] Reference Figure 4 , Figure 4 FIG. 2 is a flowchart of a second embodiment of the living body detection method of the present application.

[0091] Based on the above first embodiment, in the present embodiment, the step S40 comprises:

[0092] Step S401: performing feature extraction on the local detail image by a local texture feature extraction module, and outputting a local texture feature corresponding to the to-be-identified face image.

[0093] It can be understood that the local texture feature extraction module can be a module that mainly determines spatial features such as face texture, shadow and depth to infer whether the face corresponding to the to-be-identified face image is a real face.

[0094] Step S402: performing feature comparison on the local texture feature and a previous local texture feature by a time sequence feature extraction module, and outputting a time sequence feature of the to-be-identified face image.

[0095] It can be understood that the time sequence feature extraction module can be a module that mainly determines time sequence features such as the consistency of micro-expression actions of important parts of a face and physiological features of face skin to infer whether the face corresponding to the to-be-identified face image is a real face, and the micro-expression actions of important parts of a face include but are not limited to changes in eye corners, mouth corners and facial wrinkles; the local texture feature extraction module and the time sequence feature extraction module constitute a multi-dimensional feature extraction module.

[0096] Further, in order to improve the accuracy of the living body detection, the step S402 comprises: determining, by a time sequence feature extraction module, a local edge feature according to the local texture feature; determining, by the time sequence feature extraction module, a previous local edge feature according to the previous local texture feature, and determining a difference feature according to the local edge feature and the previous local edge feature; and outputting, by the time sequence feature extraction module, the time sequence feature of the face image to be recognized according to the local texture feature, the local edge feature, the previous local edge feature, the difference feature and a previous time sequence feature.

[0097] In a specific implementation, referring to Figure 5 , Figure 5 is a schematic diagram of the overall structure of the model, a high-definition image frame (i.e., the face image to be recognized) is acquired at T+1 time, the high-definition image frame is reduced to a thumbnail image frame, feature extraction is performed on the thumbnail image frame by an initial feature extraction module to obtain an initial feature extraction image; the encoder in the space-time attention module outputs feature encoding data to the decoder of the space-time attention module according to the initial feature extraction image and image position encoding; the decoder outputs a thumbnail feature vector according to the feature encoding data, the image position encoding and a previous thumbnail feature vector output by the decoder corresponding to the high-definition image frame at T time; the first feedforward neural network outputs image region information of a local detail image according to the thumbnail feature vector; the RoiAlign module extracts the local detail image from the high-definition image frame according to the image region information; the local texture feature extraction module outputs a local texture feature according to the input local detail image; the time sequence feature extraction module outputs a time sequence feature according to the local texture feature and a previous local texture feature corresponding to the high-definition image frame at T time; the local texture feature is input to the second feedforward neural network (i.e., the FNN module), and the second feedforward neural network outputs a first living body prediction value; the time sequence feature is input to the third feedforward neural network (i.e., the FNN module), and the third feedforward neural network outputs a second living body prediction value; and finally, whether the face image to be recognized is a living body is determined according to the first living body prediction value and the second living body prediction value.

[0098] In one example, referring to Figure 6 , Figure 6A structural diagram of the multi-dimensional feature extraction module, (1) a calculation method of the local texture feature extraction module for extracting local texture features: the local texture feature extraction module mainly determines spatial features such as face texture, shadow and depth, and infers whether it is a real face. The local texture feature extraction module is divided into four stages, and each stage is internally composed of a series of Residual Models (abbreviated as ResModel). The number of ResModel in each stage is [4, 3, 3, 3]. The features output by the fourth stage are processed by an FFN module to infer whether it is a face living body; the input of the local texture feature extraction module is a multi-dimensional tensor containing local detail image information output by the RoIAlign module, with a dimension of [N ROI *3,H s ,W s ], where H s , W s are the height and width of the preset size of the local detail image, and N ROI is the number of local detail images. The output of each stage of the local texture feature extraction module is: the feature dimension output by the first stage is [240, H s / 2, W s / 2], the feature dimension output by the second stage is [320, H s / 4, W s / 4], the feature dimension output by the third stage is [640, H s / 8, W s / 8], and the feature dimension output by the fourth stage is [1280, H s / 16, W s / 16]; the FFN structure: a fully connected layer linear with a hidden layer dimension of 512, and then a softmax layer directly outputs the prediction value. The loss function of the local texture feature extraction module is: where batch_size is the number of samples calculated in each batch, k is the type of spoofing attack (if the attack category does not need to be determined, k = 1), y ji is the i-th value in the j-th image label vector, and p ji is the i-th value in the j-th image softmax output vector. (2) a calculation method of the time sequence feature extraction module for extracting time sequence features: the time sequence feature extraction module mainly determines time sequence features such as important part micro-expression actions (such as eye corners, mouth corners and facial wrinkle changes) and consistency of human face skin physiological features, and infers whether it is a real face; the time sequence feature extraction module is composed of 4 feature comparison sub-modules and 1 FFN network, corresponding to the 4 stages of the local texture feature extraction module; the specific calculation steps of each feature comparison sub-module are: ① the feature FS i(t), compress the number of channels by a 1x1 convolution to obtain input feature F i (t);②input feature F i (t) is extracted by a sobel operator to obtain pre-order local edge feature Sobel i (t);③input feature F i (t+Δt) is extracted by a sobel operator to obtain local edge feature Sobel i (t+Δt);④the difference feature Sub is obtained by performing a Subtration operation on the pre-order local edge feature at time t and the local edge feature at time t+Δt i (t);⑤the local texture feature F i (t+Δt), the pre-order local edge feature Sobel i (t), the local edge feature Sobel i (t+Δt), the difference feature Sub i (t) and the pre-order temporal feature Fout i-1 (t) output by the i-1 stage are spliced, and a 3x3 convolution is performed to output the temporal feature Fout i (t) of the i stage; FFN structure: a fully connected layer linear is used, the hidden layer dimension is 512, and then a softmax layer is used to directly output the prediction value. The loss function of the temporal feature extraction module is: where batch_size is the number of samples calculated per batch, k is the type of spoofing attack (if the attack category does not need to be determined, then k=1), y ji refers to the i-th value in the j-th image label vector, p ji is the i-th value in the j-th image softmax output vector. The overall loss function of the model for performing liveness detection is: L=αL s (t)+βL s (t+Δt)+γL t , where L s (t) is the loss value of the local texture feature extraction module at time t, L s (t+Δt) is the loss value of the local texture feature extraction module at time t+Δt, L t is the loss value of the temporal feature extraction module, and α, β, γ are the weighting values, which can be determined according to the specific scene.

[0099] Further, in order to accelerate the convergence speed of model training, before the feature extraction of the local detail image by the local texture feature extraction module and the output of the local texture feature corresponding to the to-be-identified face image, the method further comprises the following steps: extracting a local detail sample image from a face image data sample in a face image data sample set, and constructing an initial training data set according to the local detail sample image; performing data enhancement processing on the local detail sample image in the initial training data set to obtain a target training data set; training an initial local texture feature extraction module by using the target training data set to obtain the local texture feature extraction module.

[0100] It can be understood that the embodiment adopts a step-by-step model training method, which can first train an initial local texture feature extraction module by using a target training data set to obtain the local texture feature extraction module, then train an initial spatio-temporal attention module to obtain the spatio-temporal attention module, and then train the overall model to obtain the final model for performing live detection.

[0101] In a specific implementation, considering that the model structure is relatively complex, in order to accelerate the convergence speed of model training, a distributed model training method is adopted, which can be specifically as follows: first step: local texture feature extraction module parameter training. First, skip the spatio-temporal attention module, directly take samples from the high-definition local image data set, and pre-process them to send them into the initial local texture feature extraction module for training. (1) Construct a data set. Select a certain number of samples from the key regions such as eyes, eyebrows, eye corners, mouth corners, and hair in the high-definition data set sample, and then randomly intercept face local detail images to complete the local detail image list. Each high-definition sample corresponds to a local detail image list, which contains K local detail images (K is a hyperparameter of the number of local detail images); (2) Data enhancement. New samples are generated by randomly crossing positive and negative samples, that is, a local detail image list is randomly selected from the positive and negative sample set, and no more than 1 / 3 of the local detail images are randomly exchanged to generate new samples. The new samples are mixed in the original positive and negative sample set for training. Through data enhancement, the problem of excessive proportion of individual detail features reducing the robustness of the model can be effectively alleviated; (3) Model training. The enhanced data is used for training to obtain the final parameters of the local texture feature extraction module. Second step: spatio-temporal attention module pre-training. Input the original high-definition face image and the thumbnail, lock the parameters of the local texture feature extraction module, skip the time sequence feature extraction module, use the loss function of the local texture feature extraction module for training, mainly train and update the spatio-temporal attention model parameters, and stop after 10 epochs. Third step: overall model training. Group the original samples according to the same face time sequence, each group input contains adjacent or interval time sequence original high-definition face image and thumbnail, use the overall model loss function L for training, and obtain the final model for live detection.

[0102] The embodiment extracts features of the local detail image through a local texture feature extraction module, and outputs local texture features corresponding to the to-be-identified face image; and extracts features of the local texture features and the previous local texture features through a time sequence feature extraction module, and outputs time sequence features of the to-be-identified face image. The embodiment determines the living body prediction probability of the to-be-identified face image by reasoning the local detail image from two dimensions of space and time sequence, thereby improving the accuracy of living body detection.

[0103] Reference Figure 7 , Figure 7 The flowchart of the third embodiment of the living body detection method is shown.

[0104] Based on the above embodiments, in the embodiment, the step S50 comprises:

[0105] Step S501: outputting a first living body prediction value of the to-be-identified face image according to the local texture features through a second feedforward neural network.

[0106] It can be understood that the local texture features output by the local texture feature extraction module are input to the second feedforward neural network, and the second feedforward neural network directly outputs the first living body prediction value of the to-be-identified face image.

[0107] Step S502: outputting a second living body prediction value of the to-be-identified face image according to the time sequence features through a third feedforward neural network.

[0108] It can be understood that the time sequence features output by the time sequence feature extraction module are input to the third feedforward neural network, and the third feedforward neural network directly outputs the second living body prediction value of the to-be-identified face image.

[0109] Step S503: determining a living body prediction probability of the to-be-identified face image according to the first living body prediction value and the second living body prediction value.

[0110] In a specific implementation, the local texture feature extraction module is divided into four stages, each stage is internally formed by a series of Residual Model (abbreviated as ResModel) and stacked, and the number of ResModel in each stage is [4, 3, 3, 3]. The features output by the fourth stage are subjected to a second feedforward neural network FFN to infer whether it is a human face living body. The FFN structure adopts a fully connected layer linear with a hidden layer dimension of 512, and then a softmax layer is used to directly output a first living body prediction value. The time sequence feature extraction module is composed of four feature comparison sub-modules, which correspond to the four stages of the local texture feature extraction module. The output of the fourth feature comparison sub-module is subjected to a third feedforward neural network FFN. The FFN structure adopts a fully connected layer linear with a hidden layer dimension of 512, and then a softmax layer is used to directly output a second living body prediction value. The final living body prediction value is determined according to the first living body prediction value and the second living body prediction value, and whether the to-be-recognized face image is a living body is determined according to the living body prediction value. For example, if the living body prediction value is greater than or equal to a preset value, the to-be-recognized face image is determined to be a living body, otherwise it is determined to be a non-living body.

[0111] In the embodiment, a second feedforward neural network is used to output a first living body prediction value of the to-be-recognized face image according to the local texture features, a third feedforward neural network is used to output a second living body prediction value of the to-be-recognized face image according to the time sequence features, and a living body prediction probability of the to-be-recognized face image is determined according to the first living body prediction value and the second living body prediction value. In the embodiment, the living body prediction probability of the to-be-recognized face image is determined according to the first living body prediction value and the second living body prediction value, thereby improving the accuracy of living body detection.

[0112] In addition, the embodiment of the present application further provides a storage medium, and the storage medium stores a living body detection program. When the living body detection program is executed by a processor, the steps of the living body detection method described above are implemented.

[0113] Reference Figure 8 , Figure 8 is a structural block diagram of a first embodiment of the living body detection device of the present application.

[0114] As Figure 8 shown, the living body detection device provided by the embodiment of the present application comprises:

[0115] A conversion module 10 is configured to acquire a to-be-recognized face image and convert the to-be-recognized face image into a thumbnail face image.

[0116] An information determination module 20 is configured to determine image region information of a local detail image in the to-be-recognized face image according to the thumbnail face image.

[0117] extracting the local detail image from the to-be-identified face image according to the image region information;

[0118] determining the local texture feature and the timing feature corresponding to the to-be-identified face image according to the local detail image;

[0119] outputting the living body prediction probability of the to-be-identified face image according to the local texture feature and the timing feature.

[0120] The embodiment obtains a to-be-identified face image, converts the to-be-identified face image into a thumbnail face image, determines image region information of a local detail image in the to-be-identified face image according to the thumbnail face image, extracts the local detail image from the to-be-identified face image according to the image region information, determines a local texture feature and a timing feature corresponding to the to-be-identified face image according to the local detail image, and determines a living body prediction probability of the to-be-identified face image according to the local texture feature and the timing feature. The embodiment simultaneously takes the to-be-identified face image and the thumbnail face image as inputs, extracts the local detail image from the to-be-identified face image according to the image region information of the local detail image determined according to the thumbnail face image, determines the local texture feature and the timing feature according to the local detail image, and then determines the living body prediction probability of the to-be-identified face image according to the local texture feature and the timing feature, thereby reducing the data calculation amount and ensuring the accuracy of the living body detection.

[0121] Based on the first embodiment of the living body detection device, the second embodiment of the living body detection device is provided.

[0122] In the embodiment, the information determination module 20 is further configured to: perform feature extraction on the thumbnail face image by an initial feature extraction module to obtain an initial feature extraction image; determine image position encoding according to the initial feature extraction image; output feature encoding data according to the initial feature extraction image and the image position encoding by an encoder in a space-time attention module; output a thumbnail feature vector according to the feature encoding data, the image position encoding, and a previous thumbnail feature vector by a decoder in the space-time attention module; and output image region information of a local detail image in the to-be-identified face image according to the thumbnail feature vector by a first feedforward neural network.

[0123] The information determination module 20 is further configured to: determine vertical direction position encoding according to a vertical direction coordinate of the initial feature extraction image; determine horizontal direction position encoding according to a horizontal direction coordinate of the initial feature extraction image; and determine the image position encoding according to the vertical direction position encoding and the horizontal direction position encoding.

[0124] The feature determination module 40 is further configured to perform feature extraction on the local detail image by a local texture feature extraction module to output a local texture feature corresponding to the face image to be recognized; and perform feature comparison on the local texture feature and a previous local texture feature by a temporal feature extraction module to output a temporal feature of the face image to be recognized.

[0125] The feature determination module 40 is further configured to determine a local edge feature according to the local texture feature by the temporal feature extraction module; determine a previous local edge feature according to the previous local texture feature by the temporal feature extraction module, and determine a difference feature according to the local edge feature and the previous local edge feature; and output the temporal feature of the face image to be recognized according to the local texture feature, the local edge feature, the previous local edge feature, the difference feature and a previous temporal feature by the temporal feature extraction module.

[0126] The feature determination module 40 is further configured to extract a local detail sample image from a face image data sample in a face image data sample set, and construct an initial training data set according to the local detail sample image; perform data enhancement processing on the local detail sample image in the initial training data set to obtain a target training data set; and train an initial local texture feature extraction module by the target training data set to obtain a local texture feature extraction module.

[0127] The output module 50 is further configured to output a first living body prediction value of the face image to be recognized according to the local texture feature by a second feedforward neural network; output a second living body prediction value of the face image to be recognized according to the temporal feature by a third feedforward neural network; and determine a living body prediction probability of the face image to be recognized according to the first living body prediction value and the second living body prediction value.

[0128] Other embodiments or specific implementations of the living body detection device can refer to the above-mentioned method embodiments, which will not be described here.

[0129] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0130] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0131] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but many cases the former is the better implementation. Based on such understanding, the technical solutions of the present application essentially or say the part of the contribution to the prior art can be embodied in the form of software products, the computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0132] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting liveness, characterized in that, The method includes: Acquire a face image to be identified, and convert the face image to be identified into a thumbnail face image, wherein the thumbnail face image is an image obtained by shrinking the face image to be identified to a preset resolution; Based on the thumbnail face image, determine the image region information of the local detail image in the face image to be identified; Extract the local detail image from the face image to be identified based on the image region information; Based on the local detail image, determine the local texture features and temporal features corresponding to the face image to be identified; The liveness prediction probability of the face image to be identified is determined based on the local texture features and the temporal features. The step of determining the image region information of the local detail image in the face image to be identified based on the thumbnail face image includes: The initial feature extraction module extracts features from the thumbnail face image to obtain an initial feature extraction map. The image location code is determined based on the initial feature extraction map; The encoder in the spatiotemporal attention module outputs feature-encoded data based on the initial feature extraction map and the image position encoding. The decoder in the spatiotemporal attention module outputs a thumbnail feature vector based on the feature encoding data, the image position encoding, and the preceding thumbnail feature vector. The preceding thumbnail feature vector is the thumbnail feature vector corresponding to the thumbnail face image of the previous frame. The first feedforward neural network outputs image region information of local detail images in the face image to be identified based on the thumbnail feature vector; The step of determining the local texture features and temporal features corresponding to the face image to be identified based on the local detail image includes: The local texture feature extraction module extracts features from the local detail image and outputs the local texture features corresponding to the face image to be identified. The temporal feature extraction module compares the local texture features with the preceding local texture features to output the temporal features of the face image to be identified. The step of comparing the local texture features and the preceding local texture features through the temporal feature extraction module to output the temporal features of the face image to be identified includes: The local edge features are determined by the temporal feature extraction module based on the local texture features; The temporal feature extraction module determines the preceding local edge features based on the preceding local texture features, and determines the difference features based on the local edge features and the preceding local edge features; The temporal feature extraction module outputs the temporal features of the face image to be identified based on the local texture features, the local edge features, the preceding local edge features, the difference features, and the preceding temporal features.

2. The method as described in claim 1, characterized in that, The step of determining the image location encoding based on the initial feature extraction map includes: The vertical position code is determined based on the vertical coordinates of the initial feature extraction map; The horizontal position code is determined based on the horizontal coordinates of the initial feature extraction map; The image position code is determined based on the vertical position code and the horizontal position code.

3. The method as described in claim 1, characterized in that, Before extracting features from the local detail image using the local texture feature extraction module and outputting the local texture features corresponding to the face image to be identified, the method further includes: Local detail sample images are extracted from face image data samples in the face image data sample set, and an initial training dataset is constructed based on the local detail sample images; Data augmentation processing is performed on the local detail sample images in the initial training dataset to obtain the target training dataset; The initial local texture feature extraction module is trained using the target training dataset to obtain the local texture feature extraction module.

4. The method as described in claim 1 or 2, characterized in that, Determining the liveness prediction probability of the face image to be identified based on the local texture features and the temporal features includes: The second feedforward neural network outputs the first liveness prediction value of the face image to be identified based on the local texture features; The third feedforward neural network outputs a second liveness prediction value of the face image to be identified based on the temporal features; The liveness prediction probability of the face image to be identified is determined based on the first liveness prediction value and the second liveness prediction value.

5. A liveness detection device, characterized in that, The device includes: The conversion module is used to acquire a face image to be recognized and convert the face image to be recognized into a thumbnail face image, wherein the thumbnail face image is an image obtained by shrinking the face image to be recognized to a preset resolution; The information determination module is used to determine the image region information of the local detail image in the face image to be identified based on the thumbnail face image; The extraction module is used to extract the local detail image from the face image to be identified based on the image region information; The feature determination module is used to determine the local texture features and temporal features corresponding to the face image to be identified based on the local detail image; The output module is used to determine the liveness prediction probability of the face image to be identified based on the local texture features and the temporal features. The information determination module is further configured to extract features from the thumbnail face image using the initial feature extraction module to obtain an initial feature extraction map; determine the image position code based on the initial feature extraction map; output feature encoding data using the encoder in the spatiotemporal attention module based on the initial feature extraction map and the image position code; output a thumbnail feature vector using the decoder in the spatiotemporal attention module based on the feature encoding data, the image position code, and the preceding thumbnail feature vector, wherein the preceding thumbnail feature vector is the thumbnail feature vector corresponding to the previous frame thumbnail face image; and output image region information of local detail images in the face image to be identified using the first feedforward neural network based on the thumbnail feature vector. The feature determination module is further configured to extract features from the local detail image through the local texture feature extraction module and output the local texture features corresponding to the face image to be identified; and to compare the local texture features with the preceding local texture features through the temporal feature extraction module and output the temporal features of the face image to be identified. The feature determination module is further configured to determine local edge features based on the local texture features through the temporal feature extraction module; determine preceding local edge features based on the preceding local texture features through the temporal feature extraction module, and determine difference features based on the local edge features and the preceding local edge features; and output the temporal features of the face image to be identified through the temporal feature extraction module based on the local texture features, the local edge features, the preceding local edge features, the difference features, and the preceding temporal features.

6. A liveness detection device, characterized in that, The device includes: a memory, a processor, and a liveness detection program stored in the memory and executable on the processor, the liveness detection program being configured to implement the steps of the liveness detection method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a liveness detection program, which, when executed by a processor, implements the steps of the liveness detection method as described in any one of claims 1 to 4.

Images