Live detection method, live detection model training method, and device and system thereof

Abstract

The application provides a live body detection method, a training method and device of a live body detection model, and relates to the field of artificial intelligence. The live body detection method comprises the following steps: acquiring a face image, inputting the face image into a target live body detection model, and obtaining a live body detection result. The live body detection result is used to indicate whether a person in the face image is a live body. The target live body detection model comprises a first convolutional neural network. The first convolutional neural network comprises a second convolutional neural network and a full connection layer. The second convolutional neural network is used to obtain a category feature vector of a face according to the face image. The full connection layer is used to perform live body discrimination according to the category feature vector, and obtain the live body detection result. In the scheme, the target live body detection model has high face feature extraction capability and live body discrimination capability, thereby effectively improving the accuracy of the live body detection result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a liveness detection method, a training method for a liveness detection model, and an apparatus and system thereof. Background Technology

[0002] With the rapid development of facial recognition (FR) technology, facial recognition systems have surpassed human accuracy in terms of recognition precision, and many authentication systems are now based on facial recognition. However, facial recognition systems are also vulnerable to attacks from unauthorized users, such as printed photos, replayed videos, or the use of masks to breach the system. To address these issues, liveness detection technology has emerged.

[0003] Existing liveness detection methods are suitable for open scenes, but these methods are easily compromised by non-real face data (such as a pre-recorded video, 2D masks, 3D masks, etc.).

[0004] Therefore, improving the accuracy of liveness detection is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a liveness detection method, a liveness detection model training method, and a device and system thereof, which can improve the accuracy of liveness detection.

[0006] In a first aspect, a liveness detection method is provided. The method includes: acquiring a face image, and inputting the face image into a target liveness detection model to obtain a liveness detection result. The liveness detection result is used to indicate whether a person in the face image is a live person. The target liveness detection model includes a first convolutional neural network, which includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain a class feature vector of the face based on the face image, and the fully connected layer is used to perform liveness determination based on the class feature vector to obtain the liveness detection result.

[0007] In the technical solution of this application, the target liveness detection model used for liveness detection includes a neural network capable of obtaining category feature vectors and a fully connected layer capable of liveness discrimination. Therefore, it possesses both high capabilities in extracting facial features and liveness discrimination, thereby effectively improving the accuracy of liveness detection results. It should be understood that the accuracy of facial feature extraction directly affects the accuracy of subsequent liveness discrimination. The technical solution of this application fully considers both capabilities, effectively improving the accuracy of liveness detection.

[0008] Face images can include real face images and non-real face images. They can be images or video frames of real and non-real faces, or feature vectors extracted from real and non-real face images.

[0009] Non-realistic human faces are divided into two-dimensional (2D) and three-dimensional (3D) categories. The 2D category mainly includes printed photos of faces, photos and video playback on tablet screens, etc., while the 3D category mainly includes 3D masks and 3D head models.

[0010] The first convolutional neural network (hereinafter referred to as the liveness detection neural network) consists of two parts: the second convolutional neural network (hereinafter referred to as the basic neural network) and a fully connected layer. The first convolutional neural network can be understood as a modification of the second convolutional neural network.

[0011] The training datasets for the first and second convolutional neural networks can be different, and the training process can be completed in stages. In conjunction with the first aspect, in some implementations of the first aspect, the target liveness detection model is obtained by updating the parameters of the first convolutional neural network using the first training data, and the second convolutional neural network is pre-trained using the second training data. The first training data includes data of real faces and data of non-real faces, while the second training data includes data of real faces.

[0012] This implementation approach can further improve model accuracy, thereby enhancing the accuracy of liveness detection. It also simplifies the training process of the target liveness detection model and reduces its training cost. Firstly, the second convolutional neural network (CNN) can utilize publicly available datasets of real faces with abundant data (i.e., the second training data can be data from publicly available real face datasets). This allows the second CNN to be trained more thoroughly and has a better ability to extract facial features. Of course, some non-real face data can also be mixed in without affecting the overall performance. It's even possible to directly select a publicly available CNN for face recognition as the second CNN. This effectively reduces training costs; that is, when training the first CNN (updating its parameters), the requirements for training data and equipment are relatively low because there is no need for extensive training of the second CNN. Secondly, the amount of the first training data (i.e. the training data used to improve the liveness detection capability) does not need to be too large. That is, the amount of data for non-real faces is relatively small, and a good effect can be achieved without a large number of training times. This is because the training stage of the first convolutional neural network is equivalent to the process of fine-tuning the parameters to enable it to have liveness detection capability.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the target liveness detection model is obtained by updating the parameters of the shallow layers of the second convolutional neural network, the parameters of the fully connected layers of the second convolutional neural network, and the parameters of the fully connected layers of the first convolutional neural network using the first training data. This significantly reduces the computational load of the training process without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the parameters of the intermediate layers of the first convolutional neural network (which are also the intermediate layers of the second convolutional neural network) remain unchanged. This is equivalent to freezing the parameters of the intermediate layers during the process of updating the parameters of the first convolutional neural network, or it can be understood as not updating the parameters of the intermediate layers. This utilizes the domain independence characteristic of the intermediate layers. This can significantly reduce the computational load of the training process without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the second convolutional neural network is a lightweight neural network used for face recognition. This can further reduce computational and storage pressure. That is, it uses a neural network model with a simple structure, fewer parameters, and smaller storage requirements, which is beneficial for deployment in application scenarios with limited computing and storage capabilities, such as in-vehicle scenarios.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned face image includes images under multiple lighting scenarios. For example, these lighting scenarios could be: sunny outdoor day, cloudy or overcast outdoor day, dimly lit indoor day, and brightly lit indoor day. Since the appearance of a face differs under different lighting environments, and the target liveness detection model of this application performs well for face images under various lighting scenarios, this can be achieved by enriching the training data of the target detection model. That is, the target detection model is trained using training data from these different lighting scenarios, thereby possessing the ability to detect face images under different lighting scenarios.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned facial images are captured using one or more cameras installed inside the vehicle. For some simple application scenarios, the acquired facial images are clear enough and less affected by the background. However, for the special scenario of a vehicle, the image is inevitably affected by factors such as interior decoration, glass reflections, and the closed nature of the environment. The target liveness detection model of this application still performs well for facial images in in-vehicle scenes. This can be achieved by enriching the training data of the target detection model; that is, the target detection model is trained using training data from such in-vehicle scenes, thereby possessing the ability to detect facial images in in-vehicle scenes.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, when there are multiple cameras, these cameras are positioned at different locations within the vehicle to obtain facial images from different angles and / or distances. The target liveness detection model of this application performs well for facial images from different angles and / or distances.

[0019] The advantage of this is that the person being tested does not need to move to accommodate a camera in a specific location to complete the facial image capture for liveness detection. For example, if there is only a camera at the control panel, the driver needs to turn right and look down for the camera to capture their face, which is not a user-friendly process and may also interfere with the person being tested.

[0020] In addition, by deploying multiple cameras inside the vehicle, liveness detection can be performed on all occupants of the vehicle.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned camera is a near-infrared camera. In the embodiments of this application, facial images can be acquired using cameras, webcams, etc. Common cameras can be used to acquire facial images, such as RGB cameras and near-infrared cameras. Among them, RGB cameras are greatly affected by light, and facial images acquired using RGB need to be processed into grayscale images before subsequent liveness detection is performed. Near-infrared cameras, on the other hand, are less affected by light, have a wider range of applications, and screen-based images (including photos displayed on a screen or videos played on a screen) cannot be imaged on a near-infrared camera because screen-based images cannot be imaged within the wavelength band of a near-infrared camera. Therefore, using a near-infrared camera is equivalent to filtering out non-realistic facial images displayed on a screen. In other words, near-infrared cameras have the advantages of being less affected by light interference and shielding non-realistic facial images displayed on a screen. Therefore, if a near-infrared camera is used, screen-based attacks are automatically filtered out.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: sending a face image, which is used to train the target liveness detection model. That is, the face image is used to train the target detection model, i.e., to update the parameters of the target liveness detection model. This achieves online updating, further improving the accuracy of the target liveness detection model.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: when the liveness detection result indicates that the person in the face image is alive, executing a decision on a target task, the target task including at least one of the following: unlocking, account login, permission granting, or payment confirmation. For example, for the unlocking task, when the liveness detection result indicates that the person is alive, it is further determined whether the person has unlocking permission; if the person has unlocking permission, unlocking is performed; otherwise, unlocking is not performed.

[0024] In other words, before executing the target task, a liveness detection is performed. If the person is determined to be non-live, there is no need to execute the subsequent task decision, thus improving the security of the decision. Alternatively, the task decision can be executed first, followed by liveness detection. That is, first determine whether the person in the face image has the necessary permissions, and if so, then further determine whether the face image is a live person.

[0025] Secondly, a training method for a liveness detection model is provided. The training method includes: acquiring first training data, which includes data of real faces and data of non-real faces; updating the parameters of a first convolutional neural network based on the first training data to obtain a target liveness detection model. The first convolutional neural network includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain the class feature vector of the face based on the training data, and the fully connected layer is used to perform liveness detection based on the class feature vector.

[0026] The training method of the technical solution in this application has the advantages of relatively simple training, relatively small training data requirements, and relatively higher accuracy of the trained model.

[0027] First, real face data is relatively easy to obtain in liveness detection, and it is diverse and plentiful. However, non-real face data, such as 2D / 3D mask data, is difficult to obtain, and its variety and quantity are limited. This leads to an imbalanced distribution of training data (which is detrimental to the training of binary classifiers) and insufficient quantity, which also prevents existing liveness detection from achieving ideal accuracy. Second, as mentioned above, existing liveness detection technologies only focus on the ability to detect liveness but neglect the ability to recognize faces (i.e., the ability to extract facial features). In fact, the accuracy of facial feature extraction directly affects the accuracy of subsequent liveness detection. Therefore, existing technologies cannot achieve high accuracy in liveness detection. The solution in this application fully considers both capabilities, effectively improving the accuracy of liveness detection.

[0028] Furthermore, during the training phase, the device used to train the model (referred to as the training device) needs to store a large amount of training data, model parameters, and perform calculations during the training process. During the execution phase, the device used to deploy the model (i.e., the device that uses the model to perform liveness detection tasks, which can be called the inference device) needs to store the model, process data and intermediate data, and perform calculations during the inference process. Therefore, both types of devices require sufficient computing and storage capabilities. However, existing liveness detection models are large in scale, complex to train, and computationally intensive, placing high demands on the storage and computing capabilities of both the training and inference devices. This makes them unsuitable for scenarios with limited computing and / or storage capabilities, such as in automotive environments. In-vehicle devices struggle to handle complex calculations and large model storage, thus rendering existing liveness detection models unsuitable for automotive scenarios.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the second convolutional neural network is pre-trained using second training data, which includes real face data. This can further improve model accuracy, thereby improving the accuracy of liveness detection, while also simplifying the training process of the target liveness detection model and reducing the training cost of the target detection model.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, when updating the parameters of the first convolutional neural network, the parameters of the shallow layers and the fully connected layers of the first convolutional neural network are also updated. This can significantly reduce the computational load of the training process without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the parameters of the intermediate layers of the first convolutional neural network (which are also the intermediate layers of the second convolutional neural network) remain unchanged. This is equivalent to freezing the parameters of the intermediate layers while updating the parameters of the first convolutional neural network, or in other words, not updating the parameters of the intermediate layers. This leverages the domain independence of the intermediate layers. This significantly reduces the computational load of the training process without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the second convolutional neural network is a lightweight neural network used for face recognition. This can further reduce computational and storage pressure. That is, using a neural network model with a simple structure, fewer parameters, and smaller storage requirements is beneficial for deployment in application scenarios with limited computing and storage capabilities, such as in-vehicle scenarios.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the first training data includes data from multiple lighting scenarios. For example, these could be the following lighting scenarios: sunny outdoor day, cloudy or overcast outdoor day, dimly lit indoor day, and brightly lit indoor day. This increases the richness of the training data, thereby improving the training effect and enabling the development of a liveness detection model applicable to more complex lighting scenarios.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the first training data is obtained using one or more cameras installed inside the vehicle. The resulting liveness detection model can perform well in in-vehicle scenarios.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, when there are multiple cameras, these cameras are positioned at different locations on the vehicle to obtain first training data from different angles and / or different distances. This increases the richness of the training data, thereby improving the training effect and enabling the training of a liveness detection model applicable to more complex lighting scenarios.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned camera is a near-infrared camera. This is applicable to scenarios that do not require screen data as training data, thus eliminating the need for collecting and training non-realistic face data from the screen. In other words, in applications using near-infrared cameras, the object detection model does not need to detect non-realistic face data from the screen. In this case, it is sufficient to train the object detection model using other training data, eliminating the need for collecting, processing, and training screen data, thereby effectively reducing training costs.

[0037] Thirdly, a liveness detection device is provided, the device including a unit for performing the method of any of the implementations of the first aspect described above.

[0038] Fourthly, a training apparatus for a liveness detection model is provided, the training apparatus including a unit for performing a training method according to any of the implementations of the second aspect described above.

[0039] Fifthly, a liveness detection device is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor executes the method in any implementation of the first aspect. This device can be installed in various devices or systems requiring liveness detection, such as vehicle terminals, smart screens, and access control systems. The device can also be a chip.

[0040] Sixthly, a training device for a liveness detection model is provided. The training device includes: a memory for storing a program; and a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor executes the training method in any implementation of the second aspect. This training device can be a host computer, server, cloud device, or other device capable of model training. The training device can also be a chip.

[0041] A seventh aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0042] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0043] Ninth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the method in any one of the implementations of the first or second aspect described above.

[0044] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any of the implementations of the first aspect. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to an embodiment of this application.

[0046] Figure 2 This is a schematic diagram illustrating the application scenarios of a liveness detection solution.

[0047] Figure 3 This is a schematic diagram of a system architecture according to an embodiment of this application.

[0048] Figure 4 This is a schematic diagram of the structure of a convolutional neural network.

[0049] Figure 5 This is a schematic diagram of the structure of a convolutional neural network.

[0050] Figure 6 This is a schematic diagram of the hardware structure of a chip according to an embodiment of this application.

[0051] Figure 7 This is a schematic structural diagram of a basic neural network according to an embodiment of this application.

[0052] Figure 8 This is a schematic structural diagram of the liveness detection neural network according to an embodiment of this application.

[0053] Figure 9 This is a schematic flowchart of a liveness detection method according to an embodiment of this application.

[0054] Figure 10 This is a schematic diagram of the layout of an in-vehicle camera according to an embodiment of this application.

[0055] Figure 11 This is a schematic diagram of another in-vehicle camera layout according to an embodiment of this application.

[0056] Figure 12 This is a schematic flowchart of the training method of the liveness detection model according to an embodiment of this application.

[0057] Figure 13 This is a schematic block diagram of a liveness detection device according to an embodiment of this application.

[0058] Figure 14 This is a schematic diagram of the hardware structure of the liveness detection device provided in the embodiments of this application.

[0059] Figure 15 This is a schematic block diagram of a training device for a liveness detection network according to an embodiment of this application.

[0060] Figure 16 This is a schematic diagram of the hardware structure of the training device for the liveness detection network provided in the embodiments of this application. Detailed Implementation

[0061] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0062] This application relates to neural networks. In order to better understand the methods of this application, the relevant terms and concepts of neural networks will be introduced below.

[0063] (1) Neural network (NN)

[0064] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit takes an intercept of 1 as input, and its output can be, for example, as shown in formula (1):

[0065]

[0066] Where s = 1, 2, ..., n, n is a natural number greater than 1, representing the number of layers in the neural network, W s For x sThe weights, also known as the parameters or coefficients of a neural network, x s Let 'b' be the input to the neural network, and 'f' be the bias of the neural unit. 'f' represents the activation function of the neural unit, which performs a non-linear transformation on the features in the neural network, converting the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from that local receptive field, which can be a region composed of several neural units.

[0067] (2) Deep neural network (DNN)

[0068] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of different layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that each neuron in the i-th layer is connected to a neuron in the (i+1)-th layer.

[0069] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. Here, W is the offset vector, W is the weight, also called coefficients or parameters; the weights can be in the form of a weight matrix, and α() is the activation function. Each layer is simply an adjustment of the input vector... The output vector is obtained after simple operations. Because DNNs have many layers, the weights W and offset vectors... The number is also relatively large. The meaning of the weights in each layer of a DNN (for ease of description, we'll call them coefficients) is as follows: Taking the coefficient W as an example, assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0070] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0071] The input layer has no weights W. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more weights a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning weights, and its ultimate goal is to obtain the weights of all layers in the trained deep neural network (e.g., a weight matrix formed by the coefficients W of multiple layers).

[0072] (3) Convolutional Neural Network (CNN)

[0073] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0074] (4) Classifier

[0075] In this embodiment, based on the pre-trained basic neural network (i.e., the second convolutional neural network), a fully connected layer is added after its last fully connected layer (usually the output layer) to form a binary classifier. This classifier is used to distinguish whether a face in an image is a real person or not, i.e., to determine whether the image is a live image. This classifier is used to classify objects in an image. The classifier may include a fully connected layer and a softmax function (which can be called a normalized exponential function), which can output the probability of different categories based on the input.

[0076] (5) Loss function

[0077] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weights of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where weights are pre-configured for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, pre-defining "how to compare the difference between the predicted and target values" is the core of the process—the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0078] (6) Back propagation (BP)

[0079] Neural networks can employ backpropagation algorithms to correct the initial weight values ​​during training, thereby minimizing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then used to update the weights in the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process designed to obtain the optimal weights, such as the weight matrix, of the neural network model.

[0080] In traditional solutions, liveness detection models are often trained using liveness detection data (including data of real and non-real faces). However, traditional solutions only focus on distinguishing whether a person is alive or not, while neglecting to improve the face recognition capability. Therefore, the liveness detection model has a poor ability to extract facial features, and it is difficult to achieve high accuracy in liveness detection based on such poor facial feature extraction capability.

[0081] To address the aforementioned issues, this application proposes a liveness detection scheme. In this scheme, the target liveness detection model includes a neural network capable of obtaining category feature vectors and a fully connected layer capable of liveness discrimination. Therefore, it possesses both high capabilities in extracting facial features and liveness discrimination, thereby effectively improving the accuracy of liveness detection results.

[0082] The solutions in this application can be applied to various liveness detection scenarios such as screen unlocking, device unlocking, account login, permission granting (e.g., access permission), and secure payment.

[0083] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to an embodiment of this application. The main framework describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0084] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0085] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0086] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0087] (1) Infrastructure:

[0088] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0089] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0090] The intelligent chip here can be a hardware acceleration chip such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0091] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0092] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0093] (2) Data:

[0094] The data at the layer above the infrastructure is used to represent data sources in the field of artificial intelligence. This data involves at least one type of information, such as graphics, images, voice, and text. This data varies across different application areas and can have different forms of representation. For example, in the context of the Internet of Things (IoT), the content of the data is related to specific IoT connected terminals and may include sensor data such as force, displacement, liquid level, temperature, or humidity.

[0095] In the embodiments of this application, the data is, for example, liveness data, which includes data of real faces and data of non-real faces. This data can be in the form of images or graphics, or in the form of feature vectors or matrices.

[0096] (3) Data processing:

[0097] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0098] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0099] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0100] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0101] (4) General abilities:

[0102] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0103] (5) Smart products and industry applications:

[0104] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, or intelligent terminals, etc.

[0105] The embodiments of this application can be applied to many fields of artificial intelligence, such as intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, or safe cities. Specifically, they are applied to the branches of these artificial intelligence fields that require liveness detection. For example, in the field of intelligent security, access is only granted to authorized personnel after liveness detection confirms the person's identity, thus preventing unauthorized access to the security system using fingerprint gloves, masks, or other tools. Similarly, in the field of autonomous driving, access is only granted to authorized personnel to log in and activate in-vehicle devices after liveness detection confirms the person's identity, thus preventing unauthorized access to the vehicle using non-facial recognition methods.

[0106] The following is a brief introduction to the two application scenarios of unlocking and permission granting.

[0107] Application Scenario 1: Unlocking

[0108] In the unlocking scenario, the input facial image can be categorized into one of two types (live or inactive). This means determining whether the person in the facial image is a real person (i.e., the live category). Unlocking is allowed if the live category is confirmed; otherwise, it is not. Figure 2 As shown. In other words, when facial images are input into the liveness detection model, the input image can be classified into one of the two categories mentioned above (i.e., the detection result is either live or not live), and then the unlock decision module determines whether to allow unlocking. The unlocking can be screen unlocking, access control unlocking, device unlocking, or vehicle unlocking, etc. Figure 2 The image shows four faces: A, B, C, and D. A and B are images of real faces; C is an image of a 2D printed photograph (not a real face); and D is an image of a 3D head model (also not a real face). When these four images are input into a liveness detection model, corresponding detection results are obtained. A and B are detected as live, while C and D are detected as inactive. These results are then input into the unlocking decision module. For A and B, the unlocking decision module further determines whether unlocking permission is granted. If permission is granted, unlocking is performed; otherwise, unlocking is not performed. For C and D, the unlocking decision module directly determines that unlocking permission is not granted, and unlocking is not performed. In other words, Figure 2It utilizes a liveness detection process to enhance the security of the unlocking task, effectively preventing unauthorized access from being stolen by using non-real facial data of individuals with unlocking privileges.

[0109] Application Scenario 2: Permissions

[0110] In permission-related scenarios, it's possible to determine whether permission has been granted, which can be done directly by... Figure 2 The unlock decision module has been replaced with an access control decision module. This means that after determining whether the person in the facial image is a live subject (i.e., whether the person in the facial image is a real person), the access control decision module can decide whether to grant permission. This utilizes a liveness detection step to improve the security of access control tasks, effectively preventing unauthorized individuals from obtaining permission by using non-real facial data of authorized individuals. For example, in an in-car payment scenario, when a user initiates a payment request, the vehicle's system can use a camera to capture an image of the user's facial region, performing liveness detection and face detection to determine whether the current user has payment authorization.

[0111] Figure 3 This is a schematic diagram of a system architecture according to an embodiment of this application, which can be used to train neural network models, such as face recognition models and liveness detection models. Figure 3 As shown, the data acquisition device 160 is used to acquire training data. For the method of this embodiment, the training data may include training images and corresponding classification results, wherein the results of the training images may be manually pre-annotated. For training the first convolutional neural network, the training images include images of real faces and images of non-real faces. For training the second convolutional neural network, the training images include images of real faces.

[0112] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130. "A / B" describes the relationship between related objects, indicating that there can be three kinds of relationships. For example, A / B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone.

[0113] The following describes how the training device 120 obtains the target model / rule 101 based on training data. In one scenario, the training device 120 processes the input original image and compares the output image with the original image until the difference between the output image and the original image is less than a certain threshold, thus completing the training of the target model / rule 101. In this case, the face recognition model of this embodiment can be trained, that is, a trained basic neural network (i.e., a second convolutional neural network) can be obtained, which can then be used to further train a liveness detection model. In another scenario, the training device 120 processes the category feature vector of the input face image and compares the output category with the label category until the accuracy of the category output by the training device 120 is greater than or equal to a certain threshold, thus completing the training of the target model / rule 101. In this case, the liveness detection model of this embodiment can be trained, that is, a liveness detection model is further trained based on the aforementioned basic neural network.

[0114] The target model / rule 101 described above can be used to implement the method of the embodiments of this application. Specifically, the target model / rule 101 in the embodiments of this application can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0115] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 3 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include a face image input by the client device.

[0116] Preprocessing module 113 and preprocessing module 114 are used to preprocess the input data (such as a face image) received by I / O interface 112. In this embodiment, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of them may be used), and the calculation module 111 may be used directly to process the input data.

[0117] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0118] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0119] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0120] exist Figure 3 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If prior authorization from the user is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in the form of display, sound, or action. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0121] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0122] like Figure 3 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be a neural network obtained by using the method of the embodiment of this application. Specifically, the neural network of the embodiment of this application can be a CNN that can be used for liveness detection, or a deep convolutional neural network (DCNN), etc.

[0123] Since CNN is a very common type of neural network, and is the neural network that this application focuses on in its embodiments, the following will be combined with... Figure 4 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to the input image.

[0124] In one implementation, the structure of the underlying neural network in the liveness detection method of this application embodiment can be as follows: Figure 4 As shown.

[0125] Figure 4 This is a schematic diagram of the structure of a convolutional neural network. Figure 4 In this example, the Convolutional Neural Network (CNN) 200 may include an input layer 210, a layer 220 (layer 220 may include convolutional layers and pooling layers, or layer 220 may include convolutional layers but not pooling layers), and a fully connected layer 230. The input layer 210 acquires the face image to be processed and passes it to layer 220 and the subsequent fully connected layer 230 for processing to obtain the image processing result. The following section... Figure 4 This section provides a detailed introduction to the internal layer structure of CNN 200.

[0126] Floor 220:

[0127] Convolutional layers:

[0128] by Figure 4 For example, Figure 4Layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation. The number and position of convolutional and pooling layers shown here are merely examples; there may be more or fewer convolutional and pooling layers, or no pooling layers at all.

[0129] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0130] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple dimensions" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the resulting convolutional feature maps are also of the same size. These extracted convolutional feature maps are then merged to form the output of the convolution operation.

[0131] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0132] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0133] Pooling layer:

[0134] Since it is often necessary to reduce the number of training parameters, pooling layers can be periodically introduced after convolutional layers, such as... Figure 4 Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0135] Fully connected layer 230:

[0136] After processing by layer 220, the convolutional neural network 200 is still insufficient to output the required output information. To generate the final output information (the required class information or other relevant information), the convolutional neural network 200 further utilizes fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 may include multiple hidden layers (such as...). Figure 4 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0137] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 4 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 4 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0138] The specific structure of the neural network used in the liveness detection method of this application embodiment can be as follows: Figure 5 As shown. In Figure 5 In this example, the convolutional neural network (CNN) 300 may include an input layer 310, a layer 320 (layer 320 may include convolutional layers and pooling layers, where pooling layers are optional), and a fully connected layer 330. Figure 4 compared to, Figure 5 Multiple convolutional or pooling layers in layer 320 are processed in parallel, and the extracted features are all input into the fully connected layer 330 for processing.

[0139] It should be noted that, Figure 4 and Figure 5 The convolutional neural networks shown are merely examples of two possible convolutional neural networks that form the basis of the liveness detection method in this application. In specific applications, the convolutional neural network used in the basis of the liveness detection method in this application may also exist in the form of other network models.

[0140] It should be noted that, in the embodiments of this application, the liveness detection model can be a CNN (a first convolutional neural network after training). This CNN is simply a basic neural network (which is also a CNN structure) that can be used for face recognition, with one or more fully connected layers added for binary classification. Therefore, it can be regarded as... Figure 4 or Figure 5 Based on the structure shown, one or more binary classification fully connected layers are added after the output layer. Therefore... Figure 4 and Figure 5 The processing result of the face image to be processed output by the output layer can be called the category feature vector, that is, the feature vector of the face that can be used for classification.

[0141] It should also be understood that the liveness detection neural network (i.e., the first convolutional neural network) obtained above based on the basic neural network is still a CNN structure. Therefore, the liveness detection model (CNN for liveness detection) in the embodiments of this application can also be used. Figure 4 and Figure 5 The architecture shown indicates that, at this point, the processing result of the face image to be processed is the classification result of whether it is a live object.

[0142] Figure 6 This is a schematic diagram of the hardware structure of a chip according to an embodiment of this application. The chip includes a neural network processor (NPU600 shown). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 4 , Figure 5 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 6 This is achieved in the chip shown.

[0143] The NPU600 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 60, and the controller 604 controls the arithmetic circuit 603 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0144] In some implementations, the arithmetic circuit 603 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 603 is a two-dimensional pulsating array. The arithmetic circuit 603 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 603 is a general-purpose matrix processor.

[0145] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 602 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 601 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 608.

[0146] The vector computation unit 607 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 607 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0147] In some implementations, the vector computation unit 607 can store the processed output vector into a unified buffer 606. For example, the vector computation unit 607 can apply a nonlinear function to the output of the arithmetic circuit 603, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 607 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 603, for example, for use in subsequent layers of a neural network.

[0148] The unified memory 606 is used to store input data and output data.

[0149] The weight data is directly transferred from the external memory to the input memory 601 and / or the unified memory 606 through the direct memory access controller 605 (DMAC), the weight data in the external memory is stored in the weight memory 602, and the data in the unified memory 606 is stored in the external memory.

[0150] The bus interface unit (BIU) 610 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 609 via a bus.

[0151] The instruction fetch buffer 609, connected to the controller 604, is used to store the instructions used by the controller 604.

[0152] The controller 604 is used to call the instructions cached in the instruction memory 609 to control the operation of the computing accelerator.

[0153] Entry point: The data here is for illustrative purposes, such as captured facial images.

[0154] Optionally, the unified memory 606, input memory 601, weighted memory 602, and instruction fetch memory 609 are all on-chip memories, while the external memory is memory outside the NPU. The external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0155] in, Figure 4 and Figure 5 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 603 or the vector calculation unit 607.

[0156] The above-mentioned Figure 3 The execution device 110 in the embodiment is capable of executing each step of the liveness detection method or the liveness detection model training method of the present application. Figure 4 and Figure 5 The CNN model shown and Figure 6 The chip shown can also be used to perform various steps of the liveness detection method according to the embodiments of this application.

[0157] In this embodiment, the liveness detection model can be further derived from a basic neural network, as described below. Figure 7 and Figure 8 Let me introduce the two neural networks separately.

[0158] Figure 7 This is a schematic structural diagram of a basic neural network according to an embodiment of this application.

[0159] Optionally, the base neural network (i.e., the second convolutional neural network) can be an existing neural network for face recognition, or a base neural network for face recognition can be trained using a training library. Since this base neural network is used for face recognition rather than liveness detection, it can be trained using a large amount of real face data; alternatively, it can also be trained using non-real face data.

[0160] In some implementations, to further reduce computational and storage pressure, the base neural network can be a lightweight neural network, meaning a neural network with a simple structure, few parameters, and small storage requirements. For example, the base neural network can be an existing lightweight convolutional neural network (LightCNN) face recognition model (also called a face recognition neural network), such as a 9-layer version of the LightCNN FR model (hereinafter referred to as LightCNN-9). LightCNN-9 is one of the most accurate publicly available FR CNNs, achieving excellent performance with a smaller parameter set compared to other FR CNNs. For current liveness detection, such a small network size is particularly suitable for scenarios with limited computational and storage capabilities, such as those involving vehicles.

[0161] It should be understood that the LightCNN-9 described above is merely an example of a lightweight convolutional neural network for face recognition. Other LightCNNs, such as LightCNN-4 and LightCNN-29, can also be used. As mentioned above, any neural network used for face recognition can be used in the scheme of this application. For example, DeepFace, Webface, FaceNet, or visual geometry group (VGG) networks can also be used. For the sake of brevity, they will not be listed one by one.

[0162] Figure 7 The LightCNN-9 diagram shows a convolutional layer (conv1), a max-feature-map (MFM) layer (MFM1), pooling layers (pool1-pool4), a combination layer of convolutional layers (group2-group5), and a fully connected layer (MFM_fc1). Figure 7As shown, when a 128x128 input image is fed into LightCNN-9, the output layer (here, the fully connected layer MFM_fc1) can output a 256-dimensional feature vector (or a feature vector of other dimensions such as 128-dimensional). These 256-dimensional feature vectors are vectors of discriminative features that can distinguish features of different faces, such as the shape and position of the nose, the shape and position of the eyes, etc., and are called category feature vectors. LightCNN-9 can be trained using existing large datasets. During training, the images in the dataset can be enhanced by random flipping, random cropping, etc., and then converted to grayscale before training. Loss functions commonly used in the field of face recognition, such as arcface, cosineface, or sphereface, can be used for training, and this application does not limit the specific loss function used.

[0163] It should be noted that the size of the input image and the dimension of the output feature vector are determined by LightCNN-9. If other lightweight neural networks are used, the input image size and output dimension may also be different, and there is no limitation.

[0164] Based on the basic neural network, we can obtain Figure 8 The liveness detection neural network shown. Figure 8 This is a schematic structural diagram of the liveness detection neural network according to an embodiment of this application. In this embodiment, the liveness detection neural network is also referred to as a liveness detection model. After training it (i.e., updating its parameters), the target liveness detection model can be obtained. However, it should be understood that... Figure 8 The underlying neural network in this context can be any neural network used for face recognition; it is not limited to... Figure 7 The neural network shown is only used for ease of understanding the scheme and employs a different approach. Figure 7 The same structure is shown, but in practice, there are no restrictions on the structure of the basic neural network; any CNN used for face recognition can be used.

[0165] Furthermore, the parameters of a neural network can also be called coefficients or weights, and they can be in matrix form.

[0166] A liveness detection neural network (i.e., a first-order convolutional neural network) is used for liveness detection. After training, this neural network can serve as a target liveness detection model for liveness detection. For example... Figure 8 As shown, this liveness detection neural network can be divided into three parts: a shallow layer, an intermediate layer, and a fully connected layer. The shallow layer can be the shallowest layer of the neural network, but there is no specific limitation on which first few layers are considered the shallow layer. The intermediate layer can be understood as all the non-fully connected layers other than the shallow layer. Figure 8The fully connected layer portion of the neural network includes the fully connected layers of the basic neural network and additional fully connected layers. For ease of distinction, in Figure 8 In this diagram, FC1 and FC2 represent the fully connected layer of the basic neural network and a new fully connected layer that is not part of the basic neural network, respectively. The new fully connected layer can be a single layer or multiple layers.

[0167] Figure 8 This can be understood as adding one or more fully connected layers for binary classification after the last layer of the basic neural network (i.e., the fully connected layer that serves as the output layer). This added fully connected layer is used for liveness detection, that is, determining whether someone is a live person (i.e., determining whether they are a real person or not). Figure 8 The output of the liveness detection neural network shown is the detection result, which can be understood as the determination of whether a person is alive or not, or as classifying the input image into one of two categories: "live" or "non-live".

[0168] Research shows that the information learned by the shallow layers of CNNs is task- and dataset-specific. Therefore, these shallow, domain-specific features in CNNs can be named domain-specific units (DSUs). DSUs can correspond to... Figure 8 The shallow layer shown. Layers higher than the shallow layer in a CNN (corresponding to...) Figure 8 The intermediate layers (the parts of the image processing system) can share features across different imaging domains. The learned features are domain-independent and exhibit robust performance across different tasks and datasets. In other words, the intermediate layers can share parameters across different datasets. Fully connected layers, on the other hand, are highly task- and dataset-specific. Figure 8 A regression-based classifier (FC2) was added to the basic neural network. This classifier is specific to binary classification tasks, namely liveness detection tasks.

[0169] Assumption Figure 8 The basic neural network is Figure 7 As shown in the LightCNN-9 diagram, the shallow layers can include the layers up to group 2, the intermediate layers can include pool 2 to pool 4, and the fully connected layers can include MFM_fc1 (which is equivalent to FC1 shown in Figure 8) and FC2. The FC2 layer takes the 256-dimensional feature vector output from MFM_fc1 as input and outputs a binary classification result indicating whether the person is alive. In other words, the newly added FC2 learns a 256-dimensional category feature vector with rich discriminative facial features to distinguish whether a person in an image is alive.

[0170] When training a liveness detection neural network, the binary cross-entropy (BCE) loss function can be used. The BCE loss function can be expressed by the formula L. BCE =-(ylog(p)+(1-y)log(p)) means, where L BCE The value of the loss function is represented by y, which indicates whether the face is alive and takes the value of 1 or 0, and p represents the predicted probability of a real face.

[0171] During training, stochastic gradient descent (SGD) can be used, with the learning rate set to 1e-2, weight decay set to 1e-4, and momentum set to 0.90.

[0172] As analyzed above, intermediate layers possess domain independence, meaning they can be shared across different datasets. Therefore, this can be used to further reduce the computational cost of training. The parameters (weights) of the intermediate layers can be frozen. Freezing can also be understood as maintaining, not updating, and not training. During training, only the parameters of the shallow network and fully connected layers are updated, while the parameters of the intermediate layers remain unchanged. This significantly reduces the computational cost of the training process without compromising training performance. It's equivalent to... Figure 8 When training the liveness detection neural network shown, the parameters of the shallow and fully connected layers can be updated, while the parameters of the intermediate layers remain unchanged.

[0173] It should be understood that "remaining unchanged" means that the base neural network has already been trained, so it uses the parameters of the intermediate layers, not that the parameters of the intermediate layers have never been trained. In other words, the parameters of the intermediate layers are obtained during the training phase of the base neural network (which can be called the first training phase for ease of understanding), while during the training phase of the liveness detection neural network (which can be called the second training phase for ease of understanding), the parameters of the intermediate layers remain unchanged (i.e., frozen). That is to say, the second training phase only updates the parameters of the shallow and fully connected layers of the liveness detection neural network.

[0174] It should also be understood that a basic neural network can also be divided into shallow layers, intermediate layers, and fully connected layers. Shallow layers are the first few layers of the basic neural network, but there is no specific limit to the number of these layers. Intermediate layers are all non-fully connected layers outside the shallow layers. The intermediate layers of a basic neural network and the intermediate layers of a liveness detection neural network are the same part, as are the shallow layers of a basic neural network and the shallow layers of a liveness detection neural network. However, the fully connected layers of a basic neural network and the fully connected layers of a liveness detection neural network are different because the fully connected layers of a liveness detection neural network include additional fully connected layers for binary classification (such as...). Figure 8 FC2), excluding fully connected layers of the underlying neural network (such as FC2). Figure 8 (FC1).

[0175] Figure 9 This is a schematic flowchart of the liveness detection method according to an embodiment of this application. The following is a description of... Figure 9 The steps involved will be described below. This liveness detection method can be executed using a device or system that has deployed a liveness detection model, such as a mobile terminal, vehicle terminal, computer, smart screen, or intelligent control system, etc.

[0176] 901. Obtain a face image.

[0177] Face images can include real face images and non-real face images. They can be images or video frames of real and non-real faces, or feature vectors extracted from real and non-real face images.

[0178] Non-realistic human faces are divided into one or both of 2D and 3D categories. 2D mainly includes printed photos of faces, photos and video playback on tablets and other screens, while 3D mainly includes 3D masks and 3D head models.

[0179] The facial image can be acquired using the acquisition unit of a liveness detection device. This acquisition unit can be an image acquisition device, a communication interface, an interface circuit, etc. When the acquisition unit is an image acquisition device, it is equivalent to integrating the image acquisition device into the liveness detection device. For example, a smartphone with a camera can be considered a liveness detection device, and the acquisition unit can be the smartphone's camera. When the smartphone executes the liveness detection method of this application embodiment, the camera acquires the aforementioned facial image and transmits it to the phone processor, which then executes subsequent steps. When the acquisition unit is a device with transceiver functions, such as a communication interface or interface circuit, it is equivalent to acquiring the facial image from an external image acquisition device through the acquisition unit. The specific connection and communication methods used can be any method, such as circuit connection, wired communication, or wireless communication, without any limitation. For example, in a vehicle scenario, the vehicle's control system can be used to execute the liveness detection method of this application embodiment. When executing the liveness detection method, the image acquisition device can acquire the facial image and send / transmit the acquired facial image to the control system. The acquisition unit in the control system executes step 901 to acquire the facial image.

[0180] Image acquisition devices can include cameras, webcams, etc.

[0181] Common cameras, such as RGB cameras and near-infrared (NIR) cameras, can be used to capture facial images. RGB cameras are significantly affected by lighting conditions, and facial images acquired using RGB need to be converted to grayscale before subsequent liveness detection is performed. Near-infrared cameras, on the other hand, are less affected by lighting conditions, have a wider range of applications, and cannot image from screens (including photos or videos displayed on a screen) because their wavelengths are not within the range of near-infrared cameras. Therefore, using a near-infrared camera effectively filters out non-realistic facial images displayed on screens. In other words, near-infrared cameras have the advantages of being less affected by lighting conditions and effectively filtering out non-realistic facial images displayed on screens.

[0182] Since faces appear differently under different lighting conditions, and the target liveness detection model of this application can perform well for face images under different lighting scenarios, that is, face images can include images from multiple lighting scenarios. For example, these scenarios can include: sunny outdoor day, cloudy or overcast outdoor day, dimly lit indoor day, and brightly lit indoor day.

[0183] For some simple application scenarios, the acquired facial images are clear enough and less affected by the background. However, for the special scenario of a vehicle, the effects of interior decoration, glass reflection, and the closed environment are unavoidable. However, the target liveness detection model of this application can still perform well for facial images in the in-vehicle scene.

[0184] In vehicle scenarios, one or more cameras can be installed inside the vehicle to capture the aforementioned facial images.

[0185] Furthermore, facial images vary depending on the position and angle of the camera. The target liveness detection model in this application performs well for facial images from different angles and / or distances. In other words, facial images can include images from different angles and / or distances. For example, a camera can be placed on each side of the vehicle's windshield, and another camera can be positioned at the dashboard; all of these cameras can capture facial images of the driver. Figure 10 This is a schematic diagram of the layout of an in-vehicle camera according to an embodiment of this application. Figure 10 The main focus is on filming the driver's seat, such as... Figure 10 As shown, cameras facing the driver's seat can be installed at any of the following locations: left front (A), right front (B), steering wheel (C), control panel (D), etc., thus capturing images from different distances and angles from the driver's seat. It can be seen that... Figure 10 The shape of the camera at point A differs from the other three cameras; this is to demonstrate that different types of cameras can be used in a layout simultaneously. However, it should be understood that... Figure 10 This is just an example; in practice, there are many possible layouts. The advantage of this is that the driver doesn't need to move to accommodate a camera in a specific location, and facial image capture for liveness detection can still be completed even while focused on driving. For example, if there's only a camera at control panel D, the driver needs to turn right and look down to allow that camera to capture their face, making the interaction less user-friendly and potentially distracting. However, it should be understood that... Figure 10 The purpose is to explain that multiple cameras can be used to capture facial images at the same location, but this location is not limited to the driver's seat; it can also be the front passenger seat, the rear passenger seat, etc.

[0186] Alternatively, multiple cameras can be placed inside the vehicle to enable liveness detection for all occupants. Figure 11 This is a schematic diagram illustrating another layout of an in-vehicle camera according to an embodiment of this application. For example... Figure 11As shown, cameras can be installed at points A (in front of the driver's seat), B (in front of the passenger seat), C (behind the driver's seat), and D (behind the passenger seat). Camera A captures the driver's face, camera B captures the passenger's face, camera C captures the face of the person in the left rear position, and camera D captures the face of the person in the right rear position. This allows for liveness detection of people in any location within the vehicle.

[0187] Figure 11 (a) in the image is a top view of the vehicle's cockpit. Figure 11 (b) is a rear view inside the vehicle. Figure 11 The layout of the cameras in different views is marked in (a) and (b).

[0188] The camera in this application can be a standalone camera or a camera from a device with shooting capabilities, such as... Figure 11 In (b) of the diagram, C and D are the cameras of the display devices installed on the front seats.

[0189] It should be understood that Figure 11 This is just one example of a camera layout. In practice, many other layout options can be provided according to specific needs, which will not be listed here.

[0190] To facilitate understanding, let's combine a... Figure 11 This section introduces a practical application scenario for the camera layout. Imagine a passenger in the front passenger seat is shopping through the vehicle's intelligent system. When it's time to pay, the passenger lacks payment authorization, while the passenger in the rear left seat does. In this case, camera C captures the rear left passenger's face, completing a liveness detection. Payment is then successful if the liveness detection is successful. However, if the passenger wears a 3D mask of the rear left passenger, camera B captures the passenger's face. But during liveness detection, the passenger's face is deemed not to be alive, and payment fails.

[0191] 902. Input the face image into the target liveness detection model to obtain the liveness detection result. The liveness detection result is used to indicate whether the person in the face image is a live person.

[0192] The target liveness detection model includes a first convolutional neural network, which includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain the class feature vector of the face based on the face image, and the fully connected layer is used to perform liveness detection based on the class feature vector to obtain the liveness detection result.

[0193] The first convolutional neural network (i.e., the liveness detection neural network mentioned above) consists of a second convolutional neural network (i.e., the basic neural network mentioned above) and fully connected layers. The first convolutional neural network (i.e., the liveness detection neural network mentioned above) can be understood as a modification of the second convolutional neural network (i.e., the basic neural network mentioned above). The explanations of the first and second convolutional neural networks can refer to the introductions of the liveness detection neural network and the basic neural network above. The explanations of the categorical feature vectors mentioned above can refer to the explanations of the discriminative feature vectors above; for simplicity, they will not be repeated.

[0194] The training datasets for the first and second convolutional neural networks can be different, and the training process can be completed in stages.

[0195] In some implementations, the aforementioned target liveness detection model is obtained by updating the parameters of the first convolutional neural network using the first training data, while the second convolutional neural network is pre-trained using the second training data. The first training data includes data from real faces and data from non-real faces, while the second training data includes data from real faces. This implementation can further improve model accuracy, thereby improving the accuracy of liveness detection, while also simplifying the training process of the target liveness detection model and reducing the training cost of the target detection model. First, the second convolutional neural network can utilize publicly available datasets of various types of real faces with abundant data (i.e., the second training data can be data from publicly available datasets of real faces), which allows the second convolutional neural network to be trained more thoroughly and has a better ability to extract facial features. Of course, some non-real face data can also be mixed in without affecting the overall effect. It is even possible to directly select one from publicly available CNNs used for face recognition as the second convolutional neural network, such as LightCNN-9 mentioned above. This effectively reduces training costs. When training the first convolutional neural network (updating its parameters), the requirements for training data and equipment are relatively low, as there's no need for extensive training of the second convolutional neural network. Secondly, the initial training data (i.e., the training data used to improve liveness detection) doesn't need to be excessive; the demand for non-real face data is relatively small, and a large number of training iterations are sufficient to achieve good results. This is because the training phase of the first convolutional neural network is essentially a process of fine-tuning the parameters to give it liveness detection capabilities.

[0196] In some implementations, only the parameters of the shallow layers and fully connected layers of the first convolutional neural network (CNN) can be updated. In other words, the target liveness detection model can be obtained by updating the parameters of the shallow layers and fully connected layers of the second CNN and the fully connected layers of the first CNN using the first training data. This significantly reduces the computational load of the training process without compromising training effectiveness, further simplifying the training of the first CNN and reducing training costs.

[0197] In some implementations, the parameters of the intermediate layers of the first convolutional neural network (which are also the intermediate layers of the second convolutional neural network) can remain unchanged. This is equivalent to freezing the parameters of the intermediate layers while updating the parameters of the first convolutional neural network, or in other words, not updating the parameters of the intermediate layers. This leverages the domain independence of the intermediate layers; for details, please refer to the relevant content above. This significantly reduces the computational load during training without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0198] In some implementations, to further reduce computational and storage pressure, the second convolutional neural network can be a lightweight neural network used for face recognition. This means employing a neural network model with a simple structure, fewer parameters, and smaller storage requirements, making it suitable for deployment in application scenarios with limited computational and storage capabilities, such as in-vehicle environments.

[0199] Alternatively, the aforementioned face images can also be used to train the object detection model. These face images can be sent to train the object liveness detection model, i.e., to update the parameters of the model. This achieves online updates, further improving the accuracy of the object liveness detection model.

[0200] It should be noted that the above sending can be sent to a local device or a cloud device, that is, to a device that can update the object detection model parameters; there is no limitation.

[0201] Optionally, for different application scenarios, different execution actions can be generated based on the liveness detection results. That is, the above liveness detection method can also include: when the liveness detection result indicates that the person in the face image is alive, executing a decision for a target task, where the target task includes at least one of the following: unlocking, account login, permission granting, or payment confirmation. Specifically, it decides whether to unlock, whether to log in, whether to grant permission, or whether to confirm payment. For example, for the unlocking task, when the liveness detection result indicates that the person is alive, it further determines whether the person has unlocking permissions. If they have unlocking permissions, unlocking is executed; otherwise, unlocking is not performed.

[0202] In other words, before executing the target task, a liveness detection is performed. If the person is determined to be non-live, there is no need to execute the subsequent task decision, thus improving the security of the decision. Alternatively, the task decision can be executed first, followed by liveness detection. That is, first determine whether the person in the face image has the necessary permissions, and if so, then further determine whether the face image is a live person.

[0203] Figure 12 This is a schematic flowchart of the training method for the liveness detection model according to an embodiment of this application. The following is a description of... Figure 12 The process will be described step by step.

[0204] 1201. Obtain the first training data.

[0205] In this embodiment, the training data can be referred to as liveness detection data, including data of real faces and data of non-real faces. This data can be images or video frames of real and non-real faces, or feature vectors extracted from real and non-real faces. The reception of real and non-real faces is described in the relevant content above and will not be repeated here.

[0206] Training data can be collected using cameras, webcams, or read from storage devices.

[0207] The first training data refers to the training data used to update the parameters of the first convolutional neural network. The first training data includes data of real faces and data of non-real faces. In other words, the training data that enables the first convolutional neural network to have the ability to detect liveness needs to include data of real faces and data of non-real faces.

[0208] Since faces appear differently under different lighting conditions, the first training data can include data from multiple lighting scenarios to enrich the training data and improve training effectiveness. For example, it could include the following lighting scenarios: sunny outdoor day, cloudy or overcast outdoor day, dimly lit indoor day, and brightly lit indoor day. This allows for the training of a liveness detection model that can be applied to more complex lighting scenarios.

[0209] For simple application scenarios, background effects are not a concern, so the acquired liveness detection data can be shared; that is, the dataset doesn't heavily consider the differences in the acquisition scene. However, for the specific scenario of a vehicle, the influence of interior decoration, glass reflections, and the enclosed nature of the environment is unavoidable. Therefore, directly using liveness detection data from other scenarios for training in a vehicle interior can easily lead to poor training results. To adapt to the vehicle interior scenario, one or more cameras installed inside the vehicle can be used to capture the initial training data. The liveness detection model trained in this way can perform well in vehicle interior scenarios.

[0210] For vehicle scenarios, the method for obtaining real human face data is simply by taking a picture, so it will not be discussed further. The main focus will be on how to obtain non-real human face data inside the vehicle.

[0211] As mentioned above, since screen-based liveness data cannot be imaged on near-infrared cameras, using a near-infrared camera eliminates the need to acquire screen-based liveness data. Printed photos, 3D masks, and 3D head models, on the other hand, can be created by having a person wear these tools while seated in a vehicle, and then the image can be captured by a camera.

[0212] Since faces appear differently when captured from different angles and distances, the liveness data to be trained can include liveness data from multiple angles to improve training effectiveness and enrich the training data. For example, in a car interior scenario: when there are multiple cameras, they are positioned at different locations within the vehicle to obtain initial training data from different angles and / or distances. For instance, a camera can be placed on each side of the windshield, and another can be positioned at the dashboard; all of these cameras can capture data from the driver's seat. The method for arranging cameras inside the car to obtain training data from different angles and / or distances can be referenced from the methods for acquiring face images described above, and will not be repeated here.

[0213] Optionally, the aforementioned camera can be a near-infrared camera, which is suitable for scenarios that do not require screen data as training data, thus eliminating the need for collecting and training non-real face data from the screen. In other words, in applications using near-infrared cameras, the object detection model does not need to have the ability to detect non-real face data from the screen. In this case, it is only necessary to train the object detection model using other training data, which can save the collection, processing, and training of screen data, thereby effectively reducing training costs.

[0214] A liveness detection dataset can be created, from which liveness detection data can be selected as training data or used to test the performance of the liveness detection model. For example, an in-vehicle liveness detection dataset can be created. This dataset can include images of real faces and images of non-real faces. Real face images can include images of real faces with different accessories (whether they have hats or glasses, and the type of glasses), at different angles (looking up, looking down, looking straight ahead), and under different lighting conditions. Non-real face images can include 2D printed photos, 3D head molds, 3D masks, etc. For convenience, the images in the dataset can also be numbered, for example, by row or column.

[0215] 1202. Based on the first training data mentioned above, update the parameters of the first convolutional neural network to obtain the target liveness detection model.

[0216] The target liveness detection model is the updated first convolutional neural network.

[0217] The first convolutional neural network includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain the class feature vector of the face based on the training data, and the fully connected layer is used to perform liveness detection based on the class feature vector.

[0218] Optionally, the first convolutional neural network can be any of the liveness detection neural networks described above, for example, it can be... Figure 8 The liveness detection neural network shown is illustrated. The second convolutional neural network can be a real-world network used for face recognition, i.e., the basic neural network mentioned above.

[0219] In some implementations, the second convolutional neural network (CNN) is pre-trained using a second training dataset, which includes real-world face data. That is, the first and second CNNs can be trained using different training datasets (first and second training datasets). This approach can further improve model accuracy, thereby enhancing liveness detection accuracy, while also simplifying the training process and reducing the training cost of the target liveness detection model. Firstly, the second CNN can utilize publicly available datasets rich in real-world face data (i.e., the second training data can be data from publicly available real-world face datasets), allowing for more thorough training and better facial feature extraction capabilities. Of course, some non-real-world face data can also be incorporated without affecting the overall performance. Alternatively, a publicly available CNN for face recognition can be directly selected as the second CNN, such as LightCNN-9 mentioned above. This effectively reduces training costs. When training the first convolutional neural network (updating its parameters), the requirements for training data and equipment are relatively low, as there's no need for extensive training of the second convolutional neural network. Secondly, the initial training data (i.e., the training data used to improve liveness detection) doesn't need to be excessive; the demand for non-real face data is relatively small, and a large number of training iterations are sufficient to achieve good results. This is because the training phase of the first convolutional neural network is essentially a process of fine-tuning the parameters to give it liveness detection capabilities.

[0220] In some cases, it can be understood that the first convolutional neural network is obtained by adding one or more fully connected layers after the output layer of the second convolutional neural network. The added fully connected layers are used for liveness detection. The second convolutional neural network is pre-trained and is used to obtain the class feature vector of the face.

[0221] In some implementations, updating the parameters of the first convolutional neural network can involve updating the parameters of its shallow layers and fully connected layers. This significantly reduces the computational load during training without compromising training effectiveness, further simplifying the training of the first convolutional neural network and lowering training costs.

[0222] In some implementations, the parameters of the intermediate layers of the first convolutional neural network (which are also the intermediate layers of the second convolutional neural network) can remain unchanged. This is equivalent to freezing the parameters of the intermediate layers while updating the parameters of the first convolutional neural network, or in other words, not updating the parameters of the intermediate layers. This leverages the domain independence of the intermediate layers; for details, please refer to the relevant content above. This significantly reduces the computational load during training without compromising training effectiveness, further simplifying the training of the first convolutional neural network and reducing training costs.

[0223] In some implementations, to further reduce computational and storage pressure, the second convolutional neural network can be a lightweight neural network used for face recognition. This means employing a neural network model with a simple structure, fewer parameters, and smaller storage requirements, making it suitable for deployment in application scenarios with limited computational and storage capabilities, such as in-vehicle environments.

[0224] Figure 12 The training method shown has the advantages of being relatively simple to train, requiring relatively little training data, and producing a more accurate model compared to existing training methods.

[0225] First, real face data is relatively easy to obtain in liveness detection, and it is diverse and plentiful. However, non-real face data, such as 2D / 3D mask data, is difficult to obtain, and its variety and quantity are limited. This leads to an imbalanced distribution of training data (which is detrimental to the training of binary classifiers) and insufficient quantity, which also prevents existing liveness detection from achieving ideal accuracy. Second, as mentioned above, existing liveness detection technologies only focus on the ability to detect liveness but neglect the ability to recognize faces (i.e., the ability to extract facial features). In fact, the accuracy of facial feature extraction directly affects the accuracy of subsequent liveness detection. Therefore, existing technologies cannot achieve high accuracy in liveness detection. The solution in this application fully considers both capabilities, effectively improving the accuracy of liveness detection.

[0226] Furthermore, during the training phase, the device used to train the model (referred to as the training device) needs to store a large amount of training data, model parameters, and perform calculations during the training process. During the execution phase, the device used to deploy the model (i.e., the device that uses the model to perform liveness detection tasks, which can be called the inference device) needs to store the model, process data and intermediate data, and perform calculations during the inference process. Therefore, both types of devices require sufficient computing and storage capabilities. However, existing liveness detection models are large in scale, complex to train, and computationally intensive, placing high demands on the storage and computing capabilities of both the training and inference devices. This makes them unsuitable for scenarios with limited computing and / or storage capabilities, such as in automotive environments. In-vehicle devices struggle to handle complex calculations and large model storage, thus rendering existing liveness detection models unsuitable for automotive scenarios.

[0227] Figure 13 This is a schematic block diagram of a liveness detection device according to an embodiment of this application. Figure 13 The liveness detection device 2000 shown includes an acquisition unit 2001 and a processing unit 2002.

[0228] The acquisition unit 2001 and the processing unit 2002 can be used to execute the liveness detection method of the present application embodiment. Specifically, the acquisition unit 2001 can execute the above step 901, and the processing unit 2002 can execute the above step 902.

[0229] Processing unit 2002 can achieve Figure 8 The function of the liveness detection neural network shown.

[0230] It should be understood that the processing unit 2002 in the above-described device 2000 may be equivalent to the processor 3002 in the device 3000 described below.

[0231] Figure 14 This is a schematic diagram of the hardware structure of the liveness detection device provided in the embodiments of this application. Figure 14 The liveness detection device 3000 shown (specifically, this device 3000 can be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, processor 3002, and communication interface 3003 are interconnected via the bus 3004.

[0232] The memory 3001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are used to execute the various steps of the liveness detection method of the embodiments of this application.

[0233] The processor 3002 may be a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the liveness detection device of this application embodiment, or to execute the liveness detection method of this application method embodiment.

[0234] The processor 3002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the liveness detection method of this application can be completed by the integrated logic circuitry in the hardware of the processor 3002 or by instructions in software form. The aforementioned processor 3002 can also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 3001. The processor 3002 reads the information in the memory 3001 and, in conjunction with its hardware, performs the functions required by the units included in the liveness detection device of this application embodiment, or executes the liveness detection method of this application method embodiment.

[0235] The communication interface 3003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 3000 and other devices or communication networks. For example, the aforementioned face image can be acquired through the communication interface 3003.

[0236] Bus 3004 may include a pathway for transmitting information between various components of device 3000 (e.g., memory 3001, processor 3002, communication interface 3003).

[0237] Figure 15 This is a schematic block diagram of a training device for a liveness detection network according to an embodiment of this application. Figure 15 The training device 4000 for the liveness detection network shown includes an acquisition unit 4001 and a training unit 4002.

[0238] The acquisition unit 4001 and the training unit 4002 can be used to execute the training method of the liveness detection model of the present application embodiment. Specifically, the acquisition unit 4001 can execute the above step 1201, and the training unit 4002 can execute the above step 1202.

[0239] It should be understood that the training unit 4002 in the above-described device 4000 may be equivalent to the processor 5002 in the device 5000 described below.

[0240] Figure 16 This is a schematic diagram of the hardware structure of the training device for the liveness detection network provided in the embodiments of this application. Figure 16 The training device 5000 for the liveness detection network shown (specifically, the device 5000 can be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0241] The memory 5001 can be a ROM, a static storage device, a dynamic storage device, or RAM. The memory 5001 can store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to execute the various steps of the training method of the liveness detection network of the embodiments of this application.

[0242] The processor 5002 may be a CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to implement the functions required by the units in the training device of the liveness detection network in the embodiments of this application, or to execute the training method of the liveness detection network in the method embodiments of this application.

[0243] The processor 5002 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the training method for the liveness detection network of this application can be completed by the integrated logic circuits in the hardware of the processor 5002 or by instructions in software form. The aforementioned processor 5002 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 5001. The processor 5002 reads the information in the memory 5001 and, in conjunction with its hardware, performs the functions required by the units included in the training device of the liveness detection network of the present application embodiment, or executes the training method of the liveness detection network of the present application method embodiment.

[0244] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, the first training data mentioned above can be obtained through the communication interface 5003.

[0245] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0246] It should be noted that, although Figure 14 The device 3000 shown Figure 16 The illustrated device 5000 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 3000 and 5000 also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 3000 and 5000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 3000 and 5000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 14 , Figure 16 All the devices shown.

[0247] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different devices to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0248] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0250] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0251] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0252] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a Universal Serial Bus flash disk (USB flash disk, UFD), which can also be simply referred to as a USB flash drive, portable hard drive, ROM, RAM, magnetic disk, or optical disk, and other media capable of storing program code.

[0253] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting liveness, characterized in that, include: Acquire facial images; The face image is input into a target liveness detection model to obtain a liveness detection result. The liveness detection result is used to indicate whether the person in the face image is a live person. The target liveness detection model includes a first convolutional neural network, which includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain a class feature vector of the face based on the face image. The fully connected layer is used to perform liveness determination based on the class feature vector to obtain the liveness detection result. The target liveness detection model is obtained by updating the parameters of the shallow network of the second convolutional neural network, the parameters of the fully connected layer of the second convolutional neural network, and the parameters of the fully connected layer of the first convolutional neural network using the first training data. The first training data includes images of real faces and images of non-real faces.

2. The method as described in claim 1, characterized in that, The second convolutional neural network is pre-trained using second training data, which includes images of real human faces.

3. The method as described in claim 1 or 2, characterized in that, The parameters of the intermediate layers of the second convolutional neural network remain unchanged.

4. The method as described in claim 1 or 2, characterized in that, The second convolutional neural network is a lightweight neural network used for face recognition.

5. The method as described in claim 1 or 2, characterized in that, The face images include images under multiple lighting conditions.

6. The method as described in claim 1 or 2, characterized in that, The facial images were captured using one or more cameras installed inside the vehicle.

7. The method as described in claim 6, characterized in that, When there are multiple cameras, the multiple cameras are set at different positions on the vehicle to obtain facial images from different angles and / or different distances.

8. The method as described in claim 6, characterized in that, The camera is a near-infrared camera.

9. The method as described in claim 1 or 2, characterized in that, The method further includes: The face image is sent, and the face image is used to train the target liveness detection model.

10. The method as described in claim 1 or 2, characterized in that, The method further includes: When the liveness detection result indicates that a human is alive in the face image, a decision is made to perform a target task, which includes at least one of the following: unlocking, account login, permission granting, or payment confirmation.

11. A training method for a liveness detection model, characterized in that, include: Acquire first training data, which includes images of real faces and images of non-real faces; Based on the first training data, the parameters of the first convolutional neural network are updated to obtain a target liveness detection model. The first convolutional neural network includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain the category feature vector of the face based on the training data, and the fully connected layer is used to perform liveness detection based on the category feature vector. The step of updating the parameters of the first convolutional neural network includes updating the parameters of the shallow layers of the second convolutional neural network, the parameters of the fully connected layers of the second convolutional neural network, and the parameters of the fully connected layers of the first convolutional neural network, while the parameters of the intermediate layers of the second convolutional neural network remain unchanged.

12. The training method as described in claim 11, characterized in that, The second convolutional neural network is pre-trained using second training data, which includes images of real human faces.

13. The training method as described in claim 11 or 12, characterized in that, The second convolutional neural network is a lightweight neural network used for face recognition.

14. The training method as described in claim 11 or 12, characterized in that, The first training data includes data from multiple lighting scenarios.

15. The training method as described in claim 11 or 12, characterized in that, The first training data was obtained using one or more cameras installed inside the vehicle.

16. The training method as described in claim 15, characterized in that, When there are multiple cameras, the multiple cameras are set at different positions on the vehicle to obtain the first training data at different angles and / or different distances.

17. The training method as described in claim 15, characterized in that, The camera is a near-infrared camera.

18. A liveness detection device, characterized in that, include: The acquisition unit is used to acquire face images; The processing unit is configured to input the face image into a target liveness detection model to obtain a liveness detection result. The liveness detection result is used to indicate whether the person in the face image is a live person. The target liveness detection model includes a first convolutional neural network, which includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain a class feature vector of the face based on the face image. The fully connected layer is used to perform liveness detection based on the class feature vector to obtain the liveness detection result. The target liveness detection model is obtained by updating the parameters of the shallow network of the second convolutional neural network, the parameters of the fully connected layer of the second convolutional neural network, and the parameters of the fully connected layer of the first convolutional neural network using the first training data. The first training data includes images of real faces and images of non-real faces.

19. The apparatus as claimed in claim 18, characterized in that, The second convolutional neural network is pre-trained using second training data, which includes images of real human faces.

20. The apparatus as claimed in claim 18 or 19, characterized in that, The parameters of the intermediate layers of the second convolutional neural network remain unchanged.

21. The apparatus as claimed in claim 18 or 19, characterized in that, The second convolutional neural network is a lightweight neural network used for face recognition.

22. The apparatus as claimed in claim 18 or 19, characterized in that, The face images include images under multiple lighting conditions.

23. The apparatus as claimed in claim 18 or 19, characterized in that, The facial images were captured using one or more cameras installed inside the vehicle.

24. The apparatus as claimed in claim 23, characterized in that, When there are multiple cameras, the multiple cameras are set at different positions on the vehicle to obtain facial images from different angles and / or different distances.

25. The apparatus as claimed in claim 23, characterized in that, The camera is a near-infrared camera.

26. The apparatus as claimed in claim 18 or 19, characterized in that, The device further includes: A sending unit is used to send the face image, which is used to train the target liveness detection model.

27. The apparatus as claimed in claim 18 or 19, characterized in that, The processing unit is also used for: When the liveness detection result indicates that a human is alive in the face image, a decision is made to perform a target task, which includes at least one of the following: unlocking, account login, permission granting, or payment confirmation.

28. A training device for a liveness detection model, characterized in that, include: An acquisition unit is used to acquire first training data, the first training data including images of real faces and images of non-real faces; The training unit is used to update the parameters of the first convolutional neural network based on the first training data to obtain a target liveness detection model. The first convolutional neural network includes a second convolutional neural network and a fully connected layer. The second convolutional neural network is used to obtain the category feature vector of the face based on the training data, and the fully connected layer is used to perform liveness detection based on the category feature vector. The step of updating the parameters of the first convolutional neural network includes updating the parameters of the shallow layers of the second convolutional neural network, the parameters of the fully connected layers of the second convolutional neural network, and the parameters of the fully connected layers of the first convolutional neural network, while the parameters of the intermediate layers of the second convolutional neural network remain unchanged.

29. The training device as described in claim 28, characterized in that, The second convolutional neural network is pre-trained using second training data, which includes images of real human faces.

30. The training device as described in claim 28 or 29, characterized in that, The second convolutional neural network is a lightweight neural network used for face recognition.

31. The training device as described in claim 28 or 29, characterized in that, The first training data includes data from multiple lighting scenarios.

32. The training device as described in claim 28 or 29, characterized in that, The first training data was obtained using one or more cameras installed inside the vehicle.

33. The training device as described in claim 32, characterized in that, When there are multiple cameras, the multiple cameras are set at different positions on the vehicle to obtain the first training data at different angles and / or different distances.

34. The training device as described in claim 32, characterized in that, The camera is a near-infrared camera.

35. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code for execution by the device, the program code including instructions for performing the method as described in any one of claims 1 to 10 or any one of claims 11 to 17.

36. A liveness detection device, characterized in that, The device includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 1 to 10.

37. A training device for a liveness detection model, characterized in that, The training device includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the training method as described in any one of claims 11 to 17.

38. A computer program product, characterized in that, When the computer program is executed on a computer, it causes the computer to perform the method as claimed in any one of claims 1 to 10 or any one of claims 11 to 17.

Images