Human face liveness detection method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for face liveness detection, and relates to the field of image processing. The method comprises: generating an image sequence according to a face image, the image sequence comprising a face image and n image local blocks, the image local blocks being obtained by segmenting the face image, and n being a positive integer; generating a target liveness feature vector based on the image sequence, the target liveness feature vector being generated based on the interaction of liveness feature information between the face image and the image local blocks and the interaction of liveness feature information between the image local blocks; classifying the target liveness feature vector to obtain a prediction score; in response to the prediction score being higher than a judgment threshold, determining that the face in the face image is live. The present application will extract liveness feature information between the whole and the local part of the face image, as well as liveness feature information between the local parts, so that the face liveness detection result is closer to the actual situation, and improves the detection accuracy and correctness.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, equipment and medium for detecting liveness of a human face. Background Art

[0002] Face liveness detection is used to detect whether the face in the image is alive. Liveness refers to living objects. In actual use, liveness detection is usually used in combination with other technologies, and is often used in scenarios such as face payment, remote identity authentication, and face payment.

[0003] The relevant technology is a model based on a convolutional neural network. The convolution kernel is set on the input face image. Through the sliding window algorithm, local information on the face image is aggregated to obtain a good feature representation. Then, the pooling operation is used to suppress smaller responses and highlight higher responses. A new round of convolution and pooling is then performed. After several rounds of repeated operations, a higher-level feature representation is obtained. Then, specific task processing is performed to obtain the result of face liveness detection.

[0004] However, related technologies focus on the aggregation of local features, and the detection results are less accurate. Summary of the invention

[0005] The embodiment of the present application provides a method, device, equipment and medium for detecting liveness of a face, which divides an input face image into several parts, and extracts liveness feature vectors from the original face image and the divided image local blocks, thereby determining whether the face in the face image is live. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for detecting liveness of a face is provided, the method comprising:

[0007] Generate an image sequence according to a face image, wherein the image sequence includes the face image and n image local blocks, wherein the image local blocks are obtained by segmenting the face image, and n is a positive integer;

[0008] Based on the image sequence, a target living body feature vector is generated, wherein the target living body feature vector is generated based on the interaction of living body feature information between the face image and the local block of the image and the interaction of living body feature information between the local blocks of the image;

[0009] Classifying the target living body feature vector to obtain a prediction score;

[0010] In response to the prediction score being higher than a judgment threshold, it is determined that the human face in the facial image is a living body.

[0011] According to another aspect of the present application, a human face liveness detection device is provided, the device comprising:

[0012] A sequence generation module, used to generate an image sequence according to a face image, wherein the image sequence includes the face image and n image local blocks, wherein the image local blocks are obtained by segmenting the face image, and n is a positive integer;

[0013] A feature extraction module, used to generate a target live feature vector based on the image sequence, wherein the target live feature vector is generated based on the interaction of live feature information between the face image and the local block of the image and the interaction of live feature information between the local blocks of the image;

[0014] The classification module is used to classify the living body feature vector to obtain a prediction score.

[0015] A judgment module is used to determine that the face in the face image is alive in response to the prediction score being higher than a judgment threshold.

[0016] In an optional design of the present application, the feature extraction module is also used to linearly map the image sequence into an input vector group, the input vector group includes n+1 one-dimensional vectors; input the input vector group into the encoder model, output the liveness feature vector group of the face image, the liveness feature vector group includes n+1 liveness feature vectors; and use the liveness feature vector corresponding to the face image in the liveness feature vector group as the target liveness feature vector.

[0017] In an optional design of the present application, the feature extraction module is also used to input the input vector group into the self-attention layer, and generate a key-value vector group, a query vector group and a numerical vector group based on the input vector group; generate a liveness score array based on the product of the key-value vector group and the query vector group; calculate the weight value of the numerical vector group based on the liveness score array; perform dot multiplication of the weight value and the numerical vector group and sum them to generate the liveness feature vector group of the face image.

[0018] In an optional design of the present application, the feature extraction module is also used to calculate the key-value vector group based on the product of the input vector group and the key-value matrix; calculate the query vector group based on the product of the input vector group and the query matrix; calculate the numerical vector group based on the product of the input vector group and the numerical matrix.

[0019] In an optional design of the present application, the feature extraction module is further used to normalize the liveness score array according to the dimension of the one-dimensional vector in the input vector group; perform exponential normalization on the normalized liveness score array to calculate the weight value.

[0020] In an optional design of the present application, the sequence generation module is also used to divide the facial image into n equal-sized image local blocks through a grid structure; adjust the size of the facial image to be the same as the image local blocks; arrange the n image local blocks with the resized facial image as the first column to generate the image sequence.

[0021] In an optional design of the present application, the classification module is further used to input the live feature vector into a classification network and output the prediction score, and the classification network is used to classify the live feature vector according to the live information in the live feature vector.

[0022] In an optional design of the present application, the judgment module is also used to input the target living feature vector into an image depth network and output a depth map of the face image; in response to the prediction score being higher than a judgment threshold and the depth map being not an all-zero map, it is determined that the face in the face image is alive, and the all-zero map indicates that the pixel values ​​of all pixels in the image are zero.

[0023] The device also includes: a training module.

[0024] In an optional design of the present application, the training module is used to construct a training data set, which includes target training samples and corresponding pseudo depth maps; input the target training samples in the training data set into the encoder model, and output a training live feature vector; input the training live feature vector into an image depth network, and output the training image depth of the target training sample; and train the encoder model based on the error between the training image depth and the pseudo depth map.

[0025] In an optional design of the present application, the training module is also used to construct a training data set, which includes target training samples and corresponding living body real annotations; input the target training samples in the training data set into the encoder model, and output a training living body feature vector; input the training living body feature vector into a classification network, and output a pseudo-annotation of the target training sample; and train the encoder model based on the error between the pseudo-annotation and the living body real annotation.

[0026] According to another aspect of the present application, a computer device is provided, which includes: a processor and a memory, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the face liveness detection method described above.

[0027] According to another aspect of the present application, a computer storage medium is provided, in which at least one program code is stored. The program code is loaded and executed by a processor to implement the face liveness detection method as described above.

[0028] According to another aspect of the present application, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the face liveness detection method described in the above aspect.

[0029] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0030] The input face image is divided into several parts, and the live feature vector is extracted from the original face image and the local blocks of the segmented image, so as to determine whether the face in the face image is alive. This method obtains the overall live feature information through the face image, obtains the local live feature information through the local blocks of the image, introduces the relationship between the whole and the local, and the local and the local, and can perform liveness detection from a more comprehensive perspective. The obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application;

[0033] Figure 2 It is a flowchart of a method for detecting live faces provided by an exemplary embodiment of the present application;

[0034] Figure 3 It is a flowchart of a method for detecting live faces provided by an exemplary embodiment of the present application;

[0035] Figure 4 is a flowchart of an exemplary implementation scenario provided by an exemplary embodiment of the present application;

[0036] Figure 5is a schematic diagram of image sequence generation provided by an exemplary embodiment of the present application;

[0037] Figure 6 is a schematic diagram of an exemplary encoder model provided by an exemplary embodiment of the present application;

[0038] Figure 7 is a flowchart of an encoder model training method provided by an exemplary embodiment of the present application;

[0039] Figure 8 is a flowchart of an encoder model training method provided by an exemplary embodiment of the present application;

[0040] Fig. 9 is a schematic diagram of an encoder model provided by an exemplary embodiment of the present application;

[0041] Fig.10 is a flowchart of an exemplary implementation scenario provided by an exemplary embodiment of the present application;

[0042] Fig.11 is a flowchart of an exemplary implementation scenario provided by an exemplary embodiment of the present application;

[0043] Fig.12 is a schematic diagram of a face liveness detection device provided by an exemplary embodiment of the present application;

[0044] Fig.13 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0046] First, the nouns involved in the embodiments of the present application are introduced:

[0047] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0048] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0049] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (optical character recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0050] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0051] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0052] Liveness detection: It is a method to determine the physiological characteristics of an object in some identity verification scenarios. In face recognition applications, liveness detection can verify whether the user is a real live person by combining actions such as blinking, opening the mouth, shaking the head, and nodding, using technologies such as facial key point positioning and face tracking. In actual scenarios, it can effectively resist common attack methods such as photos, face swapping, masks, occlusion, and screen reshoots, thereby helping users identify fraudulent behavior and protect their interests.

[0053] Transformer model: It includes a complete Encoder-Decoder framework, which is mainly composed of the attention mechanism. For example, it consists of 6 encoding modules, each of which is divided into two parts: the self-attention layer and the forward neural network. The self-attention layer is mainly responsible for calculating the relationship between each input, then weighting it, and obtaining the result as the output, which is then sent to the classification module for classification.

[0054] Attention mechanism: It is a method of solving problems proposed by imitating human attention. It can quickly filter out high-value information from a large amount of information. It is usually used in encoder + decoder models. The attention mechanism can help the model assign different weights to each part of the input, extract more critical and important information, and enable the model to make more accurate judgments without causing greater overhead in the model's calculation and storage. For example, when the encoder + decoder model is used for translation, the input sentence and the output sentence often correspond to one or several input words corresponding to one or several output words. If each word in the sentence is given the same weight, it is unreasonable. Therefore, different weight values ​​are given to different words to distinguish the important parts of the sentence. Suppose the input sentence is "Today, Mingruns", the output sentence is "Today, Xiao Ming is running", and the words "today", "Xiao Ming" and "running" can be extracted from the translated sentence. Obviously, in the translated sentence, the three words have different importance. Among them, the importance of the word "today" is not as high as that of the words "Xiao Ming" and "running". Therefore, the weight value of "today" can be set to 0.2, and the weight values ​​of the words "Xiao Ming" and "running" can be set to 0.4 to increase the importance of the words "Xiao Ming" and "running".

[0055] All-zero image: When using RGB values ​​(RGB values ​​are the changes in the three color channels of red (R, Red), green (G, Green), and blue (B, Blue) and their superposition to obtain various colors, RGB represents the colors of the three channels of red, green, and blue) to assign values ​​to the pixels in the image, the RGB values ​​of all pixels in the image are assigned to 0 to obtain the image. Since when the RGB value is 0, the corresponding pixel does not carry any color information, the pixel appears black, so the all-zero image can also be regarded as a completely black image, and all pixels in the image appear black.

[0056] Figure 1 The schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a terminal 120 and a server 140 .

[0057] An application related to liveness detection is installed on the terminal 120. The application can be a small program in an app (application), a dedicated application, or a web client. Exemplarily, a user performs operations related to liveness detection on the terminal 120. For example, the user is preparing to use the face payment function. In order to ensure the safety and reliability of the payment process, it is necessary to first perform liveness detection on the face image obtained by the terminal 120 to prevent transactions caused by illegal attacks and protect the interests of individuals and the public. The terminal 120 is at least one of a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop computer, and a desktop computer.

[0058] The terminal 120 is connected to the server 140 via a wireless network or a wired network.

[0059] The server 140 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 140 is used to provide background services for the application of liveness detection and send the results of liveness detection to the terminal 120. Optionally, the server 140 undertakes the main computing work and the terminal 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work and the terminal 120 undertakes the main computing work; or, the server 140 and the terminal 120 both adopt a distributed computing architecture for collaborative computing.

[0060] Figure 2FIG. 1 is a flow chart showing a method for detecting a living face according to an exemplary embodiment of the present invention. Figure 1 The terminal 120 shown executes the method, which includes the following steps:

[0061] Step 202: Generate an image sequence according to the face image, the image sequence includes the face image and n image local blocks, the image local blocks are obtained by segmenting the face image, and n is a positive integer.

[0062] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0063] The method for acquiring the face image includes at least one of: acquiring by real-time shooting with a camera, acquiring by accessing a local storage space, acquiring by sending from other terminals, and acquiring by downloading from the Internet. This application does not limit the method for acquiring the face image.

[0064] Exemplarily, the terminal captures a facial video of the user through a camera, and captures one or more frames of images in the facial video as the facial image.

[0065] Optionally, after acquiring the image, the face region in the image is acquired, and the face region is enlarged according to a preset ratio coefficient, and the enlarged face region is used as the face image. Figure 3 As shown, after obtaining an image, the face region 301 is determined by human face detection technology, and then the face region 301 is enlarged to obtain the enlarged face region 302. Since the face detection technology will give priority to selecting the facial features, such as the mouth, eyes, nose, ears, etc., the edge of the face may be omitted. For example, when using the face detection technology, the image of the chin and hair may be omitted, resulting in the omission of part of the face image, so that the final face liveness detection result has a large deviation from the actual situation.

[0066] The image sequence is composed of face images and local image blocks, wherein the local image blocks are obtained by segmenting the face images.

[0067] Optionally, the sizes of the local blocks of the image are the same, and a complete face image can be obtained by splicing.

[0068] On the other hand, this embodiment does not exclude the following situations:

[0069] 1. The local image blocks are obtained by segmenting the face image, but the local image blocks cannot be spliced ​​to obtain a complete face image. For example, the face image is segmented to obtain 9 local image blocks, and 8 of the local image blocks are taken to form an image sequence.

[0070] 2. The sizes of the image local blocks are different, and a complete face image can be obtained by splicing the image local blocks. For example, a 3*3 face image is divided into 1*1 image local blocks, 2*2 image local blocks, 1*2 image local blocks, and 2*1 image local blocks.

[0071] 3. There are repeated parts in the image partial blocks. For example, a 2*3 face image is divided into two 2*2 image partial blocks, and there are repeated parts between the two obtained image partial blocks.

[0072] The above three situations are only used to illustrate the implementation methods that are easy to popularize and are not used to limit the present application.

[0073] Step 204: Generate a target live feature vector based on the image sequence, where the target live feature vector is generated based on the interaction of live feature information between the face image and the local blocks of the image and the interaction of live feature information between the local blocks of the image.

[0074] The target live feature vector is generated based on the interaction of live feature information between the face image and the local blocks of the image and the interaction of live feature information between the local blocks of the image. Specifically, the face image mainly carries the overall live feature information, while the local blocks of the image mainly carry the local live feature information. In the process of generating the target live feature vector, there is an interaction between the overall live feature information and the local live feature information, as well as an interaction between the local live feature information, so that the output target live feature vector includes the complete live feature information in the face image, which can better complete liveness detection.

[0075] Optionally, there is one target living body feature vector generated, and the target living body feature vector is taken to perform living body detection.

[0076] Optionally, there are multiple target live feature vectors generated, and one of the live feature vectors is selected for liveness detection.

[0077] Optionally, there are multiple target live feature vectors generated, and all live feature vectors are taken to perform liveness detection.

[0078] Optionally, the interactive relationship between the overall live feature information and the local live feature information in the face image and the interactive relationship between the local live feature information are extracted by a model with image sequence processing capability. Exemplarily, the model is one of a transformer model, an ELMo (Embeddings from Language Models) model, a Text CNN (Text Convolutional Neural Networks) model, a Text RNN (Text Recurrent Neural Network), a Text R-CNN (Text Recurrent-Convolutional Neural Networks) text recurrent convolutional neural network, a bidirectional long short-term memory network, and a bidirectional long short-term memory text relationship dual recurrent neural network.

[0079] Optionally, the image sequence is input into a natural language processing (NLP) model to generate a target living body feature vector.

[0080] Step 206: Classify the target living body feature vector to obtain a prediction score.

[0081] According to the live feature information carried in the live feature vector, live feature classification is performed on the live feature, and the classification result obtained is the prediction score. Optionally, the prediction score has a value range. Exemplarily, the output prediction score belongs to the interval (0, 1).

[0082] Optionally, the live feature vector is input into a classification network, and a prediction score is outputted, and the classification network is used to classify the live feature vector according to the live information in the live feature vector. Exemplarily, the live feature vector is input into the classification network to obtain a prediction score, and the prediction score belongs to the interval (0, 1).

[0083] Step 208: In response to the prediction score being higher than the judgment threshold, determining that the face in the face image is alive.

[0084] On the other hand, in response to the prediction score being not higher than the judgment threshold, it is determined that the face in the face image is not a living body.

[0085] The judgment threshold is used to judge whether the face in the face image corresponding to the current prediction score is alive. Optionally, the technician can adjust the judgment threshold according to actual needs. Exemplarily, the judgment threshold is set to 0.5, and in the ath liveness detection, the prediction score obtained is 0.6. According to the prediction score, the face in the face image is determined to be alive. However, the face in the face image is not actually alive, and in the next b liveness detections, the prediction results continue to be inconsistent with the actual results. At this time, the technician can increase or decrease the judgment threshold according to the actual situation, for example, increase the judgment threshold from 0.5 to 0.6.

[0086] Exemplarily, the judgment threshold is set to 0.5. When the prediction score is greater than 0.5, it is determined that the face in the face image is alive; when the prediction score is less than 0.5, it is determined that the face in the face image is not alive.

[0087] In summary, this embodiment divides the input face image into several parts, and extracts live feature vectors from the original face image and the local blocks of the divided image, so as to determine whether the face in the face image is alive. This method obtains the overall live feature information through the face image, obtains the local live feature information through the local blocks of the image, introduces the relationship between the whole and the local, and the local and the local, and can perform liveness detection from a more comprehensive perspective. The obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection.

[0088] In the following embodiments, on the one hand, a single face image is segmented to obtain local image blocks to obtain local liveness feature information in the face image. On the other hand, different weight values ​​are given to different liveness feature information in the face image through the attention mechanism to highlight more important liveness feature information. On the other hand, different judgment methods are provided for the final liveness detection, and they are respectively applicable to different scenarios, which can achieve faster liveness detection and obtain more accurate liveness detection results. The specific embodiments are as follows:

[0089] Figure 4 FIG. 1 is a flow chart showing a method for detecting a living face according to an exemplary embodiment of the present invention. Figure 1 The terminal 120 shown executes the method, which includes the following steps:

[0090] Step 401: Divide the face image into n equal-sized local image blocks through a grid structure.

[0091] Optionally, the obtained n equal-sized local image blocks can be spliced ​​to obtain a complete face image.

[0092] Optionally, the obtained n equal-sized image partial blocks have repeated parts, so a complete face image cannot be obtained by splicing. Exemplarily, a face image of size 3*3 is segmented and 4 2*2 image partial blocks are obtained, wherein the 4 2*2 face images have repeated parts and a complete face image cannot be obtained by splicing.

[0093] Optionally, the obtained n equal-sized image local blocks do not have repeated parts, and a complete face image cannot be obtained by splicing. Exemplarily, a face image of size 2*2 is segmented and 4 1*1 image local blocks are obtained, wherein the 4 image local blocks do not have repeated areas, and the 4 image local blocks can be spliced ​​to obtain a complete face image, and 3 of the image local blocks are taken as the image local blocks to be used in the next step.

[0094] For example, Figure 5 As shown, a face image 501 is divided into 9 image partial blocks 502 of equal size through a 3*3 grid, and these 9 image partial blocks can be spliced ​​to obtain a complete face image 503, and there is no repeated part between the 9 image partial blocks.

[0095] Step 402: Adjust the size of the face image to be the same as the local image block.

[0096] Optionally, the size of the face image is adjusted to be the same as the local image block by an image interpolation method.

[0097] Optionally, the size of the face image is adjusted to be the same as the local block of the image by downsampling.

[0098] For example, the size of the face image is 126*126, and the size of the local image block obtained by 3*3 grid segmentation is 42*42, then the size of the face image is reduced to 42*42 by downsampling.

[0099] Optionally, the size of the local block of the image is adjusted to be the same as that of the face image. For example, the size of the local block of the image is adjusted to be the same as that of the face image by upsampling, or the size of the local block of the image is adjusted to be the same as that of the face image by image interpolation. Exemplarily, the size of the face image is 126*126, and the size of the local block of the image obtained by 3*3 grid segmentation is 42*42, then the size of the face image is enlarged to 126*126 by down-up sampling.

[0100] For example, Figure 5 As shown, the size of the face image 501 is reduced to the same size as the image segmentation block 502 to obtain a face image 503.

[0101] Step 403: Arrange n image local blocks with the resized face image as the first column to generate an image sequence.

[0102] Optionally, the order between the face image and the n image partial blocks is randomly arranged. Exemplarily, assuming that there are image partial block 1, image partial block 2, image partial block 3, and image partial block 4, the image partial block 3, image partial block 1, face image, image partial block 4, image partial block 1, and image partial block 2 are arranged in the order of random arrangement to generate an image sequence.

[0103] Optionally, the n image partial blocks are first arranged from left to right and from top to bottom according to the order in the face image, and then the face image is inserted into the image sequence as the column tail.

[0104] Optionally, the n image partial blocks are first arranged from left to right and from top to bottom according to the order in the face image, and then the face image is randomly inserted into the image sequence.

[0105] For example, Figure 5 As shown, the face local blocks 502 are arranged from left to right and from top to bottom, and the face image 501 is inserted into the image sequence 504 as the first column.

[0106] Step 404: Map the image sequence into an input vector group, where the input vector group includes n+1 one-dimensional vectors.

[0107] Optionally, the image sequence is mapped into an input vector group through a linear mapping network.

[0108] Optionally, the image sequence is mapped into a set of input vectors by an encoder.

[0109] For example, Figure 6 As shown, the length of the image sequence 601 is 10. Assuming that the input shape of the image sequence 601 is 10*3*16*16 (number of images*number of channels*length*width), the two-dimensional image in the image sequence 601 is mapped to x0, x1, ..., x9 in the 10*768 input vector group 602 through linear mapping, where the one-dimensional vector corresponding to the face image is x0.

[0110] Step 405: Based on the input vector group, generate a key-value vector group, a query vector group and a value vector group.

[0111] Based on the product of the input vector group and the key value matrix (Key matrix, referred to as K matrix for short), the key value vector group is calculated.

[0112] Based on the product of the input vector group and the query matrix (Query matrix, referred to as Q matrix for short), the query vector group is calculated.

[0113] Based on the product of the input vector group and the value matrix (Value matrix, referred to as V matrix for short), the value vector group is calculated.

[0114] For example, Figure 6 As shown, the Query matrix is ​​multiplied with x0, x1, ..., x9 in the input vector group 602 in sequence to obtain q0, q1, ..., q9 in the key-value vector group 603, where the size of the Query matrix is ​​512*768. The Key matrix is ​​multiplied with x0, x1, ..., x9 in the input vector group 502 in sequence to obtain k0, k1, ..., k9 in the query vector group 604. Wherein, the size of the Key matrix is ​​512*768. The Value matrix is ​​multiplied with x0, x1, ..., x9 in the input vector group 602 in sequence to obtain v0, v1, ..., v9 in the query vector group 605. Wherein, the size of the Value matrix is ​​512*768.

[0115] Step 406: Generate a liveness score array based on the product of the key-value vector group and the query vector group.

[0116] The liveness score array is obtained based on the key-value vector group and the query vector group, which includes the overall liveness feature information and local liveness feature information in the face image.

[0117] Optionally, the target key vector in the key vector group and all the query vectors in the query vector group are multiplied in pairs to generate a liveness score array, where the target key vector is the key vector corresponding to the face image. Exemplarily, the target key vector k1 in the key vector group is multiplied in pairs with q1, ..., q5 in the query vector group to generate a liveness score array s0, ..., s5.

[0118] For example, Figure 6 As shown, the vectors in the query vector group 603 and the vectors in the key value vector group 604 are multiplied in pairs to obtain a liveness score array (Score) 606. For example, q0*k0=112, q0*k0=96.

[0119] Step 407: Calculate the weight value of the numerical vector group based on the living body score array.

[0120] The process of calculating weight values ​​includes the following two sub-steps:

[0121] 1. Normalize the liveness score array according to the dimension of the one-dimensional vector in the input vector group.

[0122] The one-dimensional vector is a vector in the above input vector group.

[0123] This normalization is used to stabilize the gradient and prevent the results from over-converging or under-converging.

[0124] For example, if a one-dimensional vector in the input vector group is represented as 10*768, the dimension of the one-dimensional vector is 768. When normalizing, the liveness score in the liveness score array is divided by the square root of the dimension. For example, if the liveness score in the liveness score array is 98, the normalization of the liveness score is 98 / √768.

[0125] Exemplary, normalized liveness score array d k is the dimension of a one-dimensional vector, score is the array of liveness scores, Figure 6 In the corresponding example, d k =768.

[0126] 2. Perform exponential normalization on the normalized liveness score array and calculate the weight value.

[0127] Optionally, exponential normalization is performed using a softmax function.

[0128] For example, Figure 6 As shown, the softmax activation function is used on the live score array 606, and the obtained weight value 607 is recorded as score i ,but

[0129] Step 408: perform dot multiplication on the weight value and the numerical vector group and sum them up to generate a living feature vector group of the face image.

[0130] Optionally, the weight values ​​are dot-multiplied with corresponding numerical vectors in the numerical vector group one by one, and all products after the dot multiplication are summed to generate a living feature vector group of the face image.

[0131] For example, Figure 6 As shown, the living body feature vector group 608 is recorded as z, then Among them, Value i =Value·score i , Value is a numerical vector group, score i is the weight value mentioned above.

[0132] Step 409: taking the liveness feature vector corresponding to the face image in the liveness feature vector group as the target liveness feature vector.

[0133] Optionally, the living body feature vector corresponding to the local image block in the living body feature vector group is used as the target living body feature vector.

[0134] Optionally, the liveness feature vectors corresponding to the local image blocks of the face image in the liveness feature vector group are collectively used as the target liveness feature vectors.

[0135] For example, Figure 6 As shown, in the living body feature vector group 608, the living body feature vector corresponding to the face image is taken as the target living body feature vector z0.

[0136] Step 410: Input the target living body feature vector into the classification network and output the prediction score.

[0137] The classification network is used to classify the target living body feature vector according to the living body information in the target living body feature vector. Exemplarily, the obtained target living body feature vector is z1, and after passing through the classification network, a prediction score of 0.6 is output.

[0138] Optionally, the prediction score has a value range. Exemplarily, the output prediction score belongs to the interval (0, 1).

[0139] Step 411: Determine whether the prediction score is higher than a determination threshold.

[0140] If the prediction score is higher than the judgment threshold, execute step 414;

[0141] If the prediction score is not higher than the judgment threshold, step 415 is executed.

[0142] Step 412: Input the target living body feature vector into the image depth network and output the depth map of the face image.

[0143] The image depth network is used to generate a depth map according to the input target living body feature vector. Exemplarily, the target living body feature vector is input into the image depth network, and the image depth network extracts the depth information therein and outputs the depth map of the face image.

[0144] Step 413: Determine whether the depth map is an all-zero map.

[0145] If the depth map is an all-zero map, execute step 415;

[0146] If the depth map is not an all-zero map, step 411 is executed.

[0147] Since the face in the face image does not belong to a living person, the face may be obtained through at least one of the common attack methods including photos, face swapping, masks, occlusion and screen reshoots, and the image depth in the face image obtained by the above attack methods is zero. Therefore, when the output depth map is an all-zero map, the face in the corresponding face image is not a living person.

[0148] Step 414: Determine whether the face in the face image is alive.

[0149] Optionally, the result is obtained through the above steps 410 and 411, or the result can be obtained through the above steps 410, 412, 413, and 411. This application does not make any specific limitation on this.

[0150] Step 415: Determine that the face in the face image is not a living person.

[0151] Optionally, the result is obtained through the above steps 410 and 411, or through the above steps 410, 412, and 413, or through the above steps 410, 412, 413, and 411. This application does not make specific limitations on this.

[0152] Optionally, when there are multiple faces in the face image, it is determined in sequence whether the faces in the face image are alive. Exemplarily, there are face 1, face 2, and face 3 in the face image, and it is determined in sequence whether face 1 is alive, face 2 is alive, and face 3 is not alive.

[0153] In summary, this embodiment divides the input face image into several parts, and extracts live feature vectors from the original face image and the local blocks of the divided image, so as to determine whether the face in the face image is alive. This method obtains the overall live feature information through the face image, obtains the local live feature information through the local blocks of the image, introduces the relationship between the whole and the local, and the local and the local, and can perform liveness detection from a more comprehensive perspective. The obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection.

[0154] Since the input face image includes enough original live feature information, it provides a good feature basis for liveness detection, which is conducive to combining the method with other models to obtain more detailed detection results. In addition, this embodiment introduces the common framework transformer model in natural language processing to extract the relationship between different local blocks of the image in the image sequence. These relationships include both the relationship between local images and local images and the relationship between local images and the overall image. Finally, the output of the corresponding position of the scaled face image is used as the live feature vector of the face image, and then it is input into the subsequent classification network for judgment. At the same time, the constraints of the depth map are imposed on the features corresponding to the original face image to ensure that the feature can contain enough original live feature information, rather than just collapsing to certain fixed values. The transformer model input requires not only local blocks of the image, but also complete face images. Therefore, the model can extract live features from different dimensions and provide a better feature basis for subsequent tasks. At the same time, based on the entire process, the input can be further adjusted to increase the reliability of the model.

[0155] Figure 7 FIG. 1 is a flow chart showing a method for detecting a living face according to an exemplary embodiment of the present invention. Figure 1 The method is executed by the server 140 or other computer device shown, and includes the following steps:

[0156] Step 701: construct a training data set, where the training data set includes target training samples and corresponding pseudo depth maps.

[0157] The training data set includes multiple face images, and the target training sample is any face image in the training data. Exemplarily, face image 1 in the training data set is used as the target training sample.

[0158] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0159] A pseudo depth map is an image that uses the distance (depth) from the image collector to each point in the scene as a pixel value and can reflect the geometric shape of the object surface.

[0160] Step 702: Input the target training samples in the training data set into the encoder model, and output the training living body feature vector.

[0161] The specific process may refer to the above-mentioned steps 404 to 409.

[0162] Optionally, when inputting an image sequence, the order of the local image blocks in the image sequence is randomly adjusted to generate an adjusted image sequence. This method can expand the training data of the model and ensure that the trained model can adapt to more different scenes. Moreover, for the feature vectors generated by different image sequences, the consistency of model training is ensured by constraining the features of the same local blocks to be sufficiently similar.

[0163] Step 703: Input the training living body feature vector into the image depth network, and output the training image depth of the target training sample.

[0164] The image depth network can output the image depth of the input image. For example, a face image 1 is input, and a depth map of the face image 1 is output through the image depth network.

[0165] Optionally, the pseudo depth map corresponding to the target training sample is used as the correct label, and the all-zero map is input into the image depth network as the wrong label. When the face in the face image does not belong to a living body, the face may be obtained through at least one common attack method of photo, face swap, mask, occlusion and screen reshoot, and the image depth in the face image obtained by the above attack methods is zero. Therefore, when the output depth map is an all-zero map, the face in the corresponding face image is not a living body.

[0166] Step 704: Train the encoder model based on the error between the training image depth and the pseudo depth map.

[0167] Optionally, a mini-batch (a gradient descent calculation method) is used to perform forward calculation to obtain the depth loss, and the BGD (batch gradient descent) optimization algorithm is used to update the encoder model.

[0168] Exemplarily, the pseudo depth map is used as the depth label of the living body, and the all-zero map is used as the depth label of the non-living body to train the encoder model. The all-zero map means that all pixel values ​​in the image are 0, in other words, the all-zero map appears as a black image.

[0169] Optionally, the encoder model is trained by calculating the error between the training image depth and the pseudo depth map through a loss function.

[0170] For example, the loss function is denoted as l depth , then:

[0171] l depth =E x～(img) ||Dep(transformer(x))-dep x ||1;

[0172] Among them, img is the target training sample, dep xis a pseudo depth map, Dep(transformer(x)) represents the output depth, E x～(img) Represents iterative multiplication.

[0173] In summary, the encoder model can simultaneously learn the discriminative parts from the features of image blocks of different categories and domains, thereby ensuring that the encoder model can learn sufficiently robust and transferable features. Moreover, the method provided in this embodiment can better model the relationship between the whole and the local and the different local parts, which can not only improve the accuracy of classification but also increase the generalization performance. At the same time, for new attack types, the network can also extract the distribution differences between the attack type and the real sample to a certain extent, and has a certain cross-domain generalization performance. In addition, when the obtained model is applied, no user interaction is required, and the time consumption is short, which improves the user experience. Moreover, the model can simultaneously learn the discriminative parts from the features of local blocks of images of different categories and domains, which greatly increases the learning difficulty of the model, thereby ensuring that the model can learn sufficiently robust features.

[0174] Figure 8 FIG. 1 is a flow chart showing a method for detecting a living face according to an exemplary embodiment of the present invention. Figure 1 The method is executed by the server 140 or other computer device shown, and includes the following steps:

[0175] Step 801: construct a training data set, where the training data set includes target training samples and corresponding living body real annotations.

[0176] The training data set includes multiple face images, and the target training sample is any face image in the training data. Exemplarily, face image 1 in the training data set is used as the target training sample.

[0177] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0178] Step 802: Input the target training samples in the training data set into the encoder model, and output the training living body feature vector.

[0179] The specific process may refer to the above-mentioned steps 404 to 409, which will not be described in detail here.

[0180] Optionally, when inputting an image sequence, the order of the local image blocks in the image sequence is randomly adjusted to generate an adjusted image sequence. This method can expand the training data of the model and ensure that the trained model can adapt to more different scenes. Moreover, for the feature vectors generated by different image sequences, the consistency of model training is ensured by constraining the features of the same local blocks to be sufficiently similar.

[0181] Step 803: Input the training live feature vector into the classification network and output the pseudo-label of the target training sample.

[0182] Pseudo-labeling refers to the labeling of the liveness detection results output by the encoder model through the target training samples.

[0183] Exemplarily, after the target training sample passes through the above model and network, the output is "yes", and the generated pseudo-annotation indicates that the face in the face image is alive.

[0184] Step 804: Train the encoder model based on the error between the pseudo-labeling and the real living body labeling.

[0185] Optionally, the classification loss is obtained by forward calculation through mini-batch (a gradient descent calculation method), the BGD (batch gradient descent) optimization algorithm is used, and the encoder model is updated.

[0186] Optionally, the encoder model is trained by calculating the error between the pseudo-labeling and the real-live labeling through a loss function.

[0187] For example, the loss function is denoted as l depth , then:

[0188]

[0189] Among them, x i is the target training sample, y i is the target training sample x i The corresponding true annotation, softmax() is the exponential normalization function, transformer() represents the encoder model, Classifier() represents the image classification network, and log() represents the logarithmic operation.

[0190] In summary, the encoder model can simultaneously learn the discriminative parts from the features of images of different categories and domains, thereby ensuring that the encoder model can learn sufficiently robust features with transferability. Moreover, the method provided in this embodiment can better model the relationship between the whole and the local and between different local parts, which can not only improve the classification accuracy but also increase the generalization performance.

[0191] At the same time, for new attack types, the model can also extract the distribution difference between the attack type and the real sample to a certain extent, and has a certain cross-domain generalization performance. In addition, when the obtained model is applied, it does not require user interaction and takes a short time, which improves the user experience. Moreover, the model can simultaneously learn the discriminative parts from the features of local blocks of images of different categories and domains, which greatly increases the learning difficulty of the model, thereby ensuring that the model can learn sufficiently robust features.

[0192] On the other hand, the transformer model is used to extract the relationships between different local blocks of images and facial images in the image sequence. These relationships include both the relationship between local images and the relationship between local images and the overall image. Finally, we use the output of the corresponding position of the scaled original facial image as the live feature vector of the facial image, and then input it into the subsequent classification network for judgment. At the same time, the depth map constraints are imposed on the features corresponding to the original facial image to ensure that the feature can contain enough original information rather than just collapsing to certain fixed values.

[0193] Fig. 9 A schematic diagram of an encoder model provided by an exemplary embodiment of the present application is shown. The encoder model is mainly composed of a Transformer encoder 901, a classification network 902 and an image depth network 903.

[0194] The input of the Transformer encoder 901 is an input vector group, wherein the input vector group is obtained by linear mapping of the picture sequence 904. The output of the Transformer encoder 901 is a living body feature vector 905.

[0195] The live feature vector 905 will be sent to the classification network 902 to output a prediction result. Based on the relationship between the prediction result and the judgment threshold, it is determined whether the face in the face image is live.

[0196] Among them, the image depth network 903 is only used in the training process of the encoder model. The input of the image depth network 903 is also the living feature vector 905, and the output is the image depth.

[0197] In summary, the Transformer encoder can simultaneously learn the discriminative parts from the features of image blocks of different categories and domains, thereby ensuring that the encoder model can learn sufficiently robust and transferable features. Moreover, the method provided in this embodiment can better model the relationship between the whole and the local and between different local parts, which can not only improve the accuracy of classification but also increase the generalization performance. At the same time, for new attack types, the network can also extract the distribution differences between the attack type and the real sample to a certain extent, and has a certain cross-domain generalization performance. In addition, when the obtained model is applied, no user interaction is required, and the time consumption is short, which improves the user experience. Moreover, the model can simultaneously learn the discriminative parts from the features of local blocks of images of different categories and domains, which greatly increases the learning difficulty of the model, thereby ensuring that the model can learn sufficiently robust features.

[0198] Face liveness detection has a wide range of applications in modern society, and face liveness detection is used in combination with other technologies in actual use, such as face authentication, bank remote authentication, face payment, online driver remote authentication, community access control system, etc. Moreover, face liveness detection, as the first line of defense, controls an important link in authentication security, so a brief introduction to the application scenarios of liveness detection is given below.

[0199] By way of example, an exemplary implementation scenario of the present application is provided. Fig.10 A flowchart of an exemplary implementation scenario provided by an exemplary embodiment of the present application is shown. The scenario is a face payment scenario, including the following steps:

[0200] Step 1001: Acquire a user's facial image through a camera.

[0201] The terminal obtains the user's facial image through the camera.

[0202] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0203] Optionally, the user's facial image can also be obtained by accessing a local storage space, or sent from other terminals, or downloaded from the Internet.

[0204] Step 1002: Perform liveness detection on the face image.

[0205] The face liveness detection method of the present application is used to perform liveness detection on face images.

[0206] Specifically, this step includes the following sub-steps:

[0207] 1. Generate image sequences based on face images.

[0208] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0209] 2. Extract the live feature vector of the face image based on the image sequence.

[0210] The target live feature vector is generated based on the interaction of live feature information between the face image and the local blocks of the image and the interaction of live feature information between the local blocks of the image. Specifically, the face image mainly carries the overall live feature information, while the local blocks of the image mainly carry the local live feature information. In the process of generating the target live feature vector, there is an interaction between the overall live feature information and the local live feature information, as well as an interaction between the local live feature information, so that the output target live feature vector includes the complete live feature information in the face image, which can better complete liveness detection.

[0211] Optionally, there is one target living body feature vector generated, and the target living body feature vector is taken to perform living body detection.

[0212] Optionally, there are multiple target live feature vectors generated, and one of the live feature vectors is selected for liveness detection.

[0213] Optionally, there are multiple target live feature vectors generated, and all live feature vectors are taken to perform liveness detection.

[0214] 3. Classify the live feature vector and obtain the prediction score.

[0215] According to the live feature information carried in the live feature vector, live feature classification is performed on the live feature, and the classification result obtained is the prediction score. Optionally, the prediction score has a value range. Exemplarily, the output prediction score belongs to the interval (0, 1).

[0216] Optionally, the live feature vector is input into a classification network, and a prediction score is outputted, and the classification network is used to classify the live feature vector according to the live information in the live feature vector. Exemplarily, the live feature vector is input into the classification network to obtain a prediction score, and the prediction score belongs to the interval (0, 1).

[0217] 4. In response to the prediction score being higher than the judgment threshold, determining that the face in the face image is alive.

[0218] On the other hand, in response to the prediction score being not higher than the judgment threshold, it is determined that the face in the face image is not a living body.

[0219] The judgment threshold is used to judge whether the face in the face image corresponding to the current prediction score is alive. Optionally, the technician can adjust the judgment threshold according to actual needs.

[0220] Step 1003: In response to the face image passing the liveness detection, the user's identity information is identified.

[0221] In response to the face image, the user's identity information is identified through liveness detection.

[0222] In response to the facial image failing the liveness detection, the facial image is confirmed as a malicious attack. The server may issue a corresponding instruction according to the confirmation result, for example, at least one of issuing an alarm, reacquiring the facial image, and performing liveness detection on the next face.

[0223] Step 1004: Complete facial payment based on identity information.

[0224] The terminal completes facial payment based on the identity information.

[0225] Optionally, in response to failure to identify the identity information of the user, an account is created for the user.

[0226] In summary, the method of the present application can be used in face payment, and face liveness detection is an important part of controlling payment security. The face liveness detection method of the present application can reject some transactions attempted by illegal attacks, ensure the security of transactions, and ensure that the interests of companies and individuals are not harmed. At the same time, liveness detection can be performed from a more comprehensive perspective, and the obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection.

[0227] By way of example, an exemplary implementation scenario of the present application is provided. Fig.11 A flowchart of an exemplary implementation scenario provided by an exemplary embodiment of the present application is shown. The scenario is a scenario of remote bank account opening, including the following steps:

[0228] Step 1101: The terminal obtains a user's facial image through a camera.

[0229] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0230] Optionally, the user's facial image can also be obtained by accessing a local storage space, or sent from other terminals, or downloaded from the Internet.

[0231] Step 1102: The terminal sends the facial image to the server.

[0232] The terminal sends the face image to the server.

[0233] Optionally, the terminal sends the facial image to the server via at least one of 2G (2-Generation, second generation mobile communication technology), 3G (3rd-Generation, third generation mobile communication technology), 4G (4th-Generation, fourth generation mobile communication technology) or other communication technologies.

[0234] Step 1103: The server performs face liveness detection on the face image.

[0235] The server performs liveness detection on the face image through the face liveness detection method of the present application.

[0236] Specifically, this step includes the following sub-steps:

[0237] 1. Generate image sequences based on face images.

[0238] A face image refers to an image containing a human face. Optionally, the face image includes a whole face, or includes a part of a face. Optionally, the face image includes multiple faces.

[0239] 2. Extract the live feature vector of the face image based on the image sequence.

[0240] The target live feature vector is generated based on the interaction of live feature information between the face image and the local blocks of the image and the interaction of live feature information between the local blocks of the image. Specifically, the face image mainly carries the overall live feature information, while the local blocks of the image mainly carry the local live feature information. In the process of generating the target live feature vector, there is an interaction between the overall live feature information and the local live feature information, as well as an interaction between the local live feature information, so that the output target live feature vector includes the complete live feature information in the face image, which can better complete liveness detection.

[0241] Optionally, there is one target living body feature vector generated, and the target living body feature vector is taken to perform living body detection.

[0242] Optionally, there are multiple target live feature vectors generated, and one of the live feature vectors is selected for liveness detection.

[0243] Optionally, there are multiple target live feature vectors generated, and all live feature vectors are taken to perform liveness detection.

[0244] 3. Classify the live feature vector and obtain the prediction score.

[0245] According to the live feature information carried in the live feature vector, live feature classification is performed on the live feature, and the classification result obtained is the prediction score. Optionally, the prediction score has a value range. Exemplarily, the output prediction score belongs to the interval (0, 1).

[0246] Optionally, the live feature vector is input into a classification network, and a prediction score is outputted, and the classification network is used to classify the live feature vector according to the live information in the live feature vector. Exemplarily, the live feature vector is input into the classification network to obtain a prediction score, and the prediction score belongs to the interval (0, 1).

[0247] 4. In response to the prediction score being higher than the judgment threshold, determining that the face in the face image is alive.

[0248] On the other hand, in response to the prediction score being not higher than the judgment threshold, it is determined that the face in the face image is not a living body.

[0249] The judgment threshold is used to judge whether the face in the face image corresponding to the current prediction score is alive. Optionally, the technician can adjust the judgment threshold according to actual needs.

[0250] Step 1104: In response to the face image passing the liveness detection, identify the user's identity information and open a settlement account for the user.

[0251] In response to the face image passing the liveness detection, the bank server identifies the user's identity information and opens a settlement account for the user.

[0252] In response to the facial image failing the liveness detection, the facial image is confirmed as a malicious attack. The bank server may issue a corresponding instruction according to the confirmation result, for example, at least one of issuing an alarm, reacquiring the facial image, and performing liveness detection on the next face.

[0253] Optionally, in response to failure to identify the identity information of the user, an account is created for the user.

[0254] In summary, the method of the present application can be used in remote bank account opening, and face liveness detection is an important part of controlling property safety. The face liveness detection method of the present application can reject some transactions attempted by illegal attacks, ensure the security of transactions, and ensure that the interests of companies and individuals are not harmed. At the same time, liveness detection can be performed from a more comprehensive perspective, and the obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection.

[0255] The following is an embodiment of the device of the present application. For details not described in detail in the embodiment of the device, reference can be made to the corresponding records in the above method embodiment, and they will not be repeated herein.

[0256] Fig.12 A schematic diagram of the structure of a face segmentation device provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 1200 includes:

[0257] A sequence generation module 121, configured to generate an image sequence according to a face image, wherein the image sequence includes the face image and n image local blocks, wherein the image local blocks are obtained by segmenting the face image, and n is a positive integer;

[0258] A feature extraction module 122, configured to generate a target live feature vector based on the image sequence, wherein the target live feature vector is generated based on the interaction of live feature information between the face image and the local block of the image and the interaction of live feature information between the local blocks of the image;

[0259] A classification module 123, used to classify the living body feature vector to obtain a prediction score;

[0260] The judgment module 124 is used to determine that the face in the face image is alive in response to the prediction score being higher than a judgment threshold.

[0261] In an optional design of the present application, the feature extraction module 122 is also used to linearly map the image sequence into an input vector group, the input vector group includes n+1 one-dimensional vectors; input the input vector group into the encoder model, output the liveness feature vector group of the face image, the liveness feature vector group includes n+1 liveness feature vectors; and use the liveness feature vector corresponding to the face image in the liveness feature vector group as the target liveness feature vector.

[0262] In an optional design of the present application, the feature extraction module 122 is also used to input the input vector group into the self-attention layer, and generate a key-value vector group, a query vector group and a numerical vector group based on the input vector group; generate a liveness score array based on the product of the key-value vector group and the query vector group; calculate the weight value of the numerical vector group based on the liveness score array; perform dot multiplication of the weight value and the numerical vector group and sum them to generate the liveness feature vector group of the face image.

[0263] In an optional design of the present application, the feature extraction module 122 is also used to calculate the key-value vector group based on the product of the input vector group and the key-value matrix; calculate the query vector group based on the product of the input vector group and the query matrix; calculate the numerical vector group based on the product of the input vector group and the numerical matrix.

[0264] In an optional design of the present application, the feature extraction module 122 is further used to normalize the liveness score array according to the dimension of the one-dimensional vector in the input vector group; perform exponential normalization on the normalized liveness score array to calculate the weight value.

[0265] In an optional design of the present application, the sequence generation module 121 is also used to divide the facial image into n equal-sized image local blocks through a grid structure; adjust the size of the facial image to be the same as the image local blocks; arrange the n image local blocks with the resized facial image as the first column to generate the image sequence.

[0266] In an optional design of the present application, the classification module 123 is further used to input the live feature vector into a classification network and output the prediction score, and the classification network is used to classify the live feature vector according to the live information in the live feature vector.

[0267] In an optional design of the present application, the judgment module 123 is also used to input the target living feature vector into an image depth network and output a depth map of the face image; in response to the prediction score being higher than a judgment threshold and the depth map being not an all-zero map, it is determined that the face in the face image is alive, and the all-zero map indicates that the pixel values ​​of all pixels in the image are zero.

[0268] The device further includes: a training module 125 .

[0269] In an optional design of the present application, the training module 125 is used to construct a training data set, which includes target training samples and corresponding pseudo depth maps; input the target training samples in the training data set into the encoder model, and output a training live feature vector; input the training live feature vector into an image depth network, and output a training image depth of the target training sample; and train the encoder model based on the error between the training image depth and the pseudo depth map.

[0270] In an optional design of the present application, the training module 125 is also used to construct a training data set, which includes target training samples and corresponding living body real annotations; input the target training samples in the training data set into the encoder model, and output a training living body feature vector; input the training living body feature vector into the classification network, and output a pseudo-annotation of the target training sample; and train the encoder model based on the error between the pseudo-annotation and the living body real annotation.

[0271] In summary, this embodiment divides the input face image into several parts, and extracts live feature vectors from the original face image and the local blocks of the divided image, so as to determine whether the face in the face image is alive. This method obtains the overall live feature information through the face image, obtains the local live feature information through the local blocks of the image, introduces the relationship between the whole and the local, and the local and the local, and can perform liveness detection from a more comprehensive perspective. The obtained liveness detection results can better reflect the actual situation, which can not only improve the accuracy of liveness detection, but also increase the performance of liveness detection.

[0272] Since the input face image includes enough original live feature information, it provides a good feature basis for liveness detection, which is conducive to combining the method with other models to obtain more detailed detection results. In addition, this embodiment introduces the common framework transformer model in natural language processing to extract the relationship between different local blocks of the image in the image sequence. Of course, these relationships include both the relationship between local images and local images and the relationship between local images and the overall image. Finally, the output of the corresponding position of the scaled face image is used as the live feature vector of the face image, and then it is input into the subsequent classification network for judgment. At the same time, the depth map constraint is imposed on the feature corresponding to the original face image to ensure that the feature can contain enough original information, rather than just collapsed to some fixed values. The transformer model input requires not only the local block of the image, but also the complete face image. Therefore, the model can extract live features from different dimensions and provide a better feature basis for subsequent tasks. At the same time, based on the entire process, the input can be further adjusted to increase the reliability of the model.

[0273] Fig.13 1 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 for helping to transmit information between various devices in the computer device, and a large-capacity storage device 1307 for storing an operating system 1313, an application program 1314 and other program modules 1315.

[0274] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 such as a mouse and a keyboard for user inputting information. The display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include an input / output controller 1310 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, a printer, or other types of output devices.

[0275] The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer device readable medium provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include a computer device readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0276] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer device storage medium is not limited to the above. The above-mentioned system memory 1304 and mass storage device 1307 can be collectively referred to as memory.

[0277] According to various embodiments of the present disclosure, the computer device 1300 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 1300 can be connected to the network 1311 through the network interface unit 1312 connected to the system bus 1305, or the network interface unit 1312 can be used to connect to other types of networks or remote computer device systems (not shown).

[0278] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1301 implements all or part of the steps of the above-mentioned face liveness detection method by executing the one or more programs.

[0279] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the face liveness detection method provided by the above-mentioned method embodiments.

[0280] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the face liveness detection method provided in the above method embodiment.

[0281] Optionally, the present application also provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the human face liveness detection method described in the above aspects.

[0282] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0283] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0284] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting liveness of a face, characterized in that: The method comprises: Generate an image sequence according to a face image, wherein the image sequence includes the face image and n image local blocks, wherein the image local blocks are obtained by segmenting the face image, and n is a positive integer; Linearly mapping the image sequence into an input vector group, wherein the input vector group includes n+1 one-dimensional vectors; Inputting the input vector group into an encoder model, and outputting a live feature vector group of the face image, wherein the live feature vector group includes n+1 live feature vectors, and the n+1 live feature vectors are generated based on the live feature information interaction between the face image and the local blocks of the image and the live feature information interaction between the local blocks of the image; Taking the liveness feature vector corresponding to the face image in the liveness feature vector group as a target liveness feature vector, wherein the target liveness feature vector is a liveness feature vector corresponding to the one-dimensional vector corresponding to the face image in the n+1 liveness feature vectors; Classifying the target living body feature vector to obtain a prediction score; In response to the prediction score being higher than a judgment threshold, it is determined that the human face in the facial image is a living body.

2. The method according to claim 1, characterized in that The encoder model includes a self-attention layer; The step of inputting the input vector group into an encoder model and outputting a living feature vector group of the face image comprises: Inputting the input vector group into the self-attention layer, and generating a key-value vector group, a query vector group, and a value vector group based on the input vector group; Generate a liveness score array based on the product of the key-value vector group and the query vector group; Based on the living body score array, calculating the weight value of the numerical vector group; The weight value is dot-multiplied with the numerical vector group and the sum is calculated to generate the living feature vector group of the face image.

3. The method according to claim 2, characterized in that The step of generating a key-value vector group, a query vector group, and a value vector group based on the input vector group includes: The key-value vector group is calculated based on the product of the input vector group and the key-value matrix; The query vector group is calculated based on the product of the input vector group and the query matrix; The numerical vector group is calculated based on the product of the input vector group and the numerical matrix.

4. The method according to claim 2, characterized in that: The step of calculating the weight value based on the living body score array includes: Normalizing the living body score array according to the dimension of the one-dimensional vector in the input vector group; the normalization comprises: dividing the living body score of the one-dimensional vector in the living body score array by the square root of the dimension; The normalized living body score array is exponentially normalized to calculate the weight value.

5. The method according to any one of claims 1 to 4, characterized in that: The step of generating an image sequence according to a face image comprises: Dividing the face image into n equal-sized local image blocks through a grid structure; Adjusting the size of the face image to be the same as the local block of the image; The image sequence is generated by arranging n partial image blocks with the resized face image as the first column.

6. The method according to any one of claims 1 to 4, characterized in that: The classifying the living body feature vector to obtain a prediction score includes: The living body feature vector is input into a classification network, and the prediction score is output. The classification network is used to classify the target living body feature vector according to the living body information in the target living body feature vector.

7. The method according to any one of claims 1 to 4, characterized in that: In response to the prediction score being higher than a judgment threshold, determining that the face in the face image is a living body includes: Inputting the target living body feature vector into an image depth network, and outputting a depth map of the face image; In response to the prediction score being higher than a judgment threshold and the depth map being not an all-zero map, it is determined that the face in the face image is a living body, and the all-zero map indicates that the pixel values ​​of all pixels in the image are zero.

8. The method according to any one of claims 1 to 4, characterized in that: The encoder model is trained by the following steps: Constructing a training data set, wherein the training data set includes target training samples and corresponding pseudo depth maps; Inputting the target training sample in the training data set into the encoder model, and outputting a training living body feature vector; Inputting the training living body feature vector into an image depth network, and outputting the training image depth of the target training sample; The encoder model is trained based on the error between the training image depth and the pseudo depth map.

9. The method according to any one of claims 1 to 4, characterized in that: The encoder model is trained by the following steps: Constructing a training data set, wherein the training data set includes target training samples and corresponding living body real annotations; Inputting the target training sample in the training data set into the encoder model, and outputting a training living body feature vector; Inputting the training living body feature vector into a classification network, and outputting a pseudo label of the target training sample; The encoder model is trained based on the error between the pseudo-labeling and the real living body labeling.

10. A face liveness detection device, characterized in that: The device comprises: A sequence generation module, used to generate an image sequence according to a face image, wherein the image sequence includes the face image and n image local blocks, wherein the image local blocks are obtained by segmenting the face image, and n is a positive integer; A feature extraction module, configured to linearly map the image sequence into an input vector group, wherein the input vector group includes n+1 one-dimensional vectors; input the input vector group into an encoder model, and output a live feature vector group of the face image, wherein the live feature vector group includes n+1 live feature vectors, wherein the n+1 live feature vectors are generated based on the interaction of live feature information between the face image and the local blocks of the image and the interaction of live feature information between the local blocks of the image; and use the live feature vector corresponding to the face image in the live feature vector group as a target live feature vector, wherein the target live feature vector is a live feature vector corresponding to the one-dimensional vector corresponding to the face image in the n+1 live feature vectors; A classification module, used to classify the target living body feature vector to obtain a prediction score; A judgment module is used to determine that the face in the face image is alive in response to the prediction score being higher than a judgment threshold.

11. The device according to claim 10, characterized in that The device also includes: A training module is used to construct a training data set, wherein the training data set includes a target training sample and a corresponding pseudo depth map; input the target training sample in the training data set into the encoder model, and output a training live feature vector; input the training live feature vector into an image depth network, and output a training image depth of the target training sample; and train the encoder model based on an error between the training image depth and the pseudo depth map.

12. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the face liveness detection method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by the processor to implement the face liveness detection method according to any one of claims 1 to 9.

Images