Liveness detection model training methods, liveness detection methods and systems

Abstract

The liveness detection model training method, liveness detection method, and system provided in this specification, after acquiring a set of facial image samples, use an image modality conversion network to convert each facial image sample in the set into an infrared image sample and a depth image sample. This image modality conversion network is trained by introducing random variables. Based on each facial image sample, infrared image sample, and depth image sample, a preset liveness detection model is trained to obtain the liveness detection model. This scheme can improve the accuracy of liveness detection of the liveness detection model.

Description

Technical Field

[0001] This specification relates to the field of image processing, and in particular to a method for training a liveness detection model, a liveness detection method, and a system. Background Technology

[0002] In recent years, with the rapid development of internet technology, liveness detection has become an indispensable part of facial recognition systems, effectively intercepting non-liveness attack samples. To improve the accuracy of liveness detection, multimodal data is often used to train the liveness detection model, allowing it to adapt to multiple scenarios and domains. Existing liveness detection model training methods typically involve extracting features from multiple modalities using a multi-head structure, then fusing these features and performing end-to-end optimization through loss function fusion.

[0003] In the process of researching and practicing existing technologies, the inventors of this invention have found that existing training processes often produce weight shift problems, which may lead to excessive reliance on a certain modality. In addition, due to the limitation of the amount of training data, the diversity of feature distribution is also relatively limited, making it difficult to cover multiple scenarios and domains in actual deployment. Therefore, the accuracy of the trained liveness detection model is low.

[0004] Therefore, there is a need to provide a more accurate liveness detection model. Summary of the Invention

[0005] The liveness detection model training method, liveness detection method, and system provided in this manual can improve the liveness detection accuracy of the liveness detection model.

[0006] Firstly, this specification provides a method for training a liveness detection model, comprising: acquiring a set of facial image samples; using an image modality conversion network to convert each facial image sample in the set of facial image samples into an infrared image sample and a depth image sample, wherein the image modality conversion network is trained by introducing random variables corresponding to the facial images; and training a preset liveness detection model based on each facial image sample, the infrared image sample, and the depth image sample to obtain a liveness detection model.

[0007] In some embodiments, the image modality conversion network includes an infrared image generation subnetwork and a depth estimation subnetwork. The step of using the image modality conversion network to convert each facial image sample in the facial image sample set into an infrared image sample and a depth image sample includes: using the infrared image generation subnetwork to map each facial image sample into an infrared image sample; and using the depth estimation subnetwork to perform depth estimation on each facial image sample to obtain a depth image sample corresponding to each facial image sample.

[0008] In some embodiments, mapping each facial image sample to an infrared image sample using the infrared image generation subnetwork includes: obtaining a mapping relationship matrix corresponding to the infrared image generation subnetwork, the mapping relationship matrix including the mapping relationship between facial images and infrared images; and mapping each facial image sample to an infrared image sample based on the mapping relationship matrix.

[0009] In some embodiments, training the infrared image generation subnetwork includes the following steps: acquiring at least one image sample pair, each image sample pair including a current face image and a current infrared image corresponding to the current face image; randomly generating a first random variable corresponding to the current face image based on the current face image, and using the first random variable as first constraint information; and training a preset infrared image generation subnetwork based on the image sample pair, and perturbing the training of the preset infrared image generation subnetwork using the first constraint information to obtain the infrared image generation subnetwork.

[0010] In some embodiments, training the depth estimation subnetwork includes the following steps: acquiring target image samples; randomly generating a second random variable corresponding to the target image samples based on the target image samples, and using the second random variable as second constraint information; and training a preset depth estimation subnetwork based on the target image samples, and perturbing the training of the preset depth estimation subnetwork using the second constraint information to obtain the depth estimation subnetwork.

[0011] In some embodiments, training a preset liveness detection model based on each facial image sample, the infrared image sample, and the depth image sample to obtain a liveness detection model includes: selecting candidate facial image samples from the set of facial image samples based on each facial image sample, the infrared image sample, and the depth image sample; performing liveness annotation on the candidate facial image samples to obtain target facial image samples, the target facial image samples including facial image samples that are positively optimized for the preset liveness detection model; and training the preset liveness detection model based on the target facial image samples to obtain a liveness detection model.

[0012] In some embodiments, selecting candidate facial image samples from the set of facial image samples based on each facial image sample, the infrared image sample, and the depth image sample includes: performing liveness detection on each facial image sample to select a first facial image sample from the set of facial image samples; selecting a second facial image sample from the current set of facial image samples based on the infrared image sample and the depth image sample, the current set of facial image samples including facial image samples other than the first facial image sample from the set of facial image samples; and using the first facial image sample and the second facial image sample as the candidate facial image samples.

[0013] In some embodiments, performing liveness detection on the facial image samples to select a first facial image sample from the set of facial image samples includes: performing liveness detection on each facial image sample to obtain a first liveness prediction value for each facial image sample; obtaining the difference between the first liveness prediction value and a preset liveness threshold; and selecting facial image samples from the set of facial image samples whose difference is within a preset difference range to obtain a first facial image sample.

[0014] In some embodiments, selecting a second facial image sample from the current facial image sample set based on the infrared image sample and the depth image sample includes: selecting a candidate infrared image sample corresponding to each facial image sample in the current facial image sample set from the infrared image sample; selecting a candidate depth image sample corresponding to each facial image sample in the current facial image sample set from the depth image sample; performing liveness detection on the candidate infrared image sample and the candidate depth image sample respectively to obtain a second liveness prediction value for each facial image sample in the current facial image sample set; and selecting a second facial image sample from the current facial image sample set based on the second liveness prediction value.

[0015] In some embodiments, training the preset liveness detection model based on the target facial image sample to obtain a liveness detection model includes: selecting a target infrared image sample corresponding to the target facial image sample from the infrared image samples; selecting a target depth image sample corresponding to the target facial image sample from the depth image samples; and using the target facial image sample, the target infrared image sample, and the target depth image sample as a multimodal image group, and training the preset liveness detection model based on the multimodal image group to obtain a liveness detection model.

[0016] In some embodiments, training the preset liveness detection model based on the multimodal image set to obtain a liveness detection model includes: using the preset liveness detection model to extract multimodal features from the multimodal image set to obtain liveness features corresponding to the target facial image sample; determining a third liveness prediction value for the target facial image sample based on the liveness features; and converging the preset liveness detection model based on the third liveness prediction value to obtain a liveness detection model.

[0017] In some embodiments, the step of using the preset liveness detection model to extract multimodal features from the multimodal image group to obtain the liveness features corresponding to the target facial image sample includes: using the feature extraction network of the preset liveness detection model to extract multimodal features from the multimodal image group to obtain high-level features corresponding to each modality; and fusing the high-level features to obtain the liveness features corresponding to the target facial image sample.

[0018] In some embodiments, the feature extraction network includes an image extraction subnetwork, an infrared extraction subnetwork, and a depth extraction subnetwork. The feature extraction network employing a preset liveness detection model performs multimodal feature extraction on the multimodal image group to obtain high-level features corresponding to each modality. This includes: using the image extraction subnetwork to perform multidimensional feature extraction on the target facial image sample to obtain facial image features; using the infrared extraction subnetwork to perform multidimensional feature extraction on the target infrared image sample to obtain facial infrared features; using the depth extraction subnetwork to perform multidimensional feature extraction on the target depth image sample to obtain facial depth features; and using the facial image features, the facial infrared features, and the facial depth features as the high-level features.

[0019] In some embodiments, the image extraction subnetwork, the infrared extraction subnetwork, and the depth extraction subnetwork adopt a weight-sharing network structure.

[0020] In some embodiments, the step of using the image extraction subnetwork to perform multidimensional feature extraction on the target facial image sample to obtain facial image features includes: when the target facial image sample is a target video frame, acquiring the time information of the target facial image sample; using the image extraction subnetwork to perform multidimensional feature extraction on the target facial image sample to obtain initial facial image features; and using the image extraction subnetwork to extract features from the time information to obtain time features, and fusing the initial facial features and the time features to obtain the facial image features.

[0021] In some embodiments, the step of converging the preset liveness detection model based on the third liveness prediction value to obtain a liveness detection model includes: updating the network parameters of the preset liveness detection model based on the third liveness prediction value to obtain an initial liveness detection model; updating the preset liveness detection model to the initial liveness detection model; and returning to the step of obtaining the facial image sample set until the preset liveness detection model converges to obtain the liveness detection model.

[0022] Secondly, this specification also provides a liveness detection method, comprising: acquiring a target multimodal image group, the target multimodal image group including facial images of multiple modalities; and inputting the target multimodal image group into a liveness detection model, and outputting a liveness detection result corresponding to the target multimodal image group, wherein the liveness detection model is trained according to the liveness detection model training method described in the first aspect.

[0023] Thirdly, this specification also provides a liveness detection system, comprising: at least one storage medium storing a liveness detection model for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the liveness detection model and implements the liveness detection method as described in the second aspect, wherein the liveness detection model is trained according to the liveness detection model training method described in the first aspect.

[0024] As can be seen from the above technical solutions, the liveness detection model training method, liveness detection method, and system provided in this specification, after obtaining a set of facial image samples, use an image modality conversion network to convert each facial image sample in the set into an infrared image sample and a depth image sample. This image modality conversion network is trained by introducing random variables corresponding to the facial images. Based on each facial image sample, infrared image sample, and depth image sample, a preset liveness detection model is trained to obtain the liveness detection model. Since this solution can convert facial images into image samples of multiple modalities through an image modality conversion network, and since this image modality conversion network is trained by introducing random variables, it can convert diverse multimodal data, expand the diversity of training samples, and thus improve the multi-scenario deployment capability of the liveness detection model. Therefore, it can improve the liveness detection accuracy of the liveness detection model.

[0025] Other functions of the liveness detection model training method, liveness detection method, and system provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the liveness detection model training method, liveness detection method, and system provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0028] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0029] Figure 3 A flowchart of a liveness detection model training method provided according to an embodiment of this specification is shown;

[0030] Figure 4 The training process of a multimodal liveness attack detection model provided according to embodiments of this specification is illustrated; and

[0031] Figure 5 A flowchart of a liveness detection method provided according to an embodiment of this specification is shown. Detailed Implementation

[0032] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0033] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0034] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0035] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0036] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0037] Liveness detection: also known as liveness attack detection, refers to the process of distinguishing between real people and attackers during the facial recognition process. Attack types mainly include techniques such as attacks using photos, mobile phones, screens, and masks.

[0038] Multimodal data: In offline facial recognition systems, various camera modalities are used to collect data to ensure high pass rates and high security. Therefore, data from different modalities can be used for multimodal feature extraction and prediction to achieve better model performance. Image modalities can be diverse, including color images, infrared images, and depth images, etc.

[0039] Recurrent adversarial data generation: An adversarial data generation method based on CycleGAN (a recurrent adversarial network) learns mapping relationships through pattern templates or input pairs of data to generate corresponding data, thereby expanding the diversity of training data.

[0040] Relative depth estimation data generation: This is an image processing technique that estimates relative depth based on monocular RGB data to generate depth maps and corresponding data, thereby expanding the diversity of training data.

[0041] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0042] The liveness detection model provided in this manual can be applied to any liveness detection scenario. For example, in a face payment scenario, the liveness detection model in this manual can be used to perform liveness detection on the collected face image of the user who needs to make a payment operation; in an identity verification scenario, the liveness detection model in this manual can be used to perform liveness detection on the collected identity image of the user; it can also be applied to any face recognition scenario, which will not be elaborated on here.

[0043] Those skilled in the art should understand that the liveness detection model training method, liveness detection method, and system described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0044] Figure 1 This diagram illustrates an application scenario of a liveness detection system 001 provided according to an embodiment of this specification. The liveness detection system 001 (hereinafter referred to as System 001) can be applied to liveness detection in any scenario, such as liveness detection in face payment scenarios, liveness detection in identity verification scenarios, liveness detection in other face recognition scenarios, etc. Figure 1 As shown, system 001 may include target user 100, client 200, server 300 and network 400.

[0045] The target user 100 can be the user who triggers the liveness detection of the target facial image, and the target user 100 can perform the liveness detection operation on the client 200.

[0046] Client 200 can be a device for performing liveness detection on a target facial image in response to a liveness detection operation by target user 100. In some embodiments, the liveness detection model training method and the liveness detection method can be executed on client 200. In this case, client 200 may store data or instructions for executing the liveness detection model training method and the liveness detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 200 may include a hardware device with data processing capabilities and the necessary programs required to drive the hardware device. Figure 1 As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 via network 400 to receive or send messages, etc. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 200 may include an image acquisition device for acquiring a target facial image or other modal facial images. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (such as an RGB camera), or a two-dimensional image acquisition device (such as an RGB camera) and a depth image acquisition device (such as a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0047] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APPs provide the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the client 200 may have a target APP installed. The target APP can acquire at least one modality of facial image for the client 200, thereby obtaining a facial image sample set or a target facial image. In some embodiments, the target object 100 can also trigger a liveness detection request through the target APP. The target APP can respond to the liveness detection request by executing the liveness detection method described in this specification. In some embodiments, the target object 100 can also trigger a liveness detection model training request through the target APP. The target APP can respond to the liveness detection model training request by executing the liveness detection model training method described in this specification. The liveness detection model training method and the liveness detection method will be described in detail later.

[0048] Server 300 may be a server providing various services, such as a backend server supporting target facial image samples and target facial images acquired on client 200. In some embodiments, the liveness detection model training method and the liveness detection method may be executed on server 300. In this case, server 300 may store data or instructions for executing the liveness detection model training method and the liveness detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, server 300 may include hardware devices with data processing capabilities and the necessary programs required to drive the hardware devices. Server 300 may communicate with multiple clients 200 and receive data sent by clients 200.

[0049] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or Internet switching points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0050] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0051] It should be noted that the liveness detection model training method and the liveness detection method can be executed entirely on the client 200, entirely on the server 300, or partially on the client 200 and partially on the server 300.

[0052] Figure 2 A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the liveness detection model training method and the liveness detection method described in this specification. The liveness detection model training method and the liveness detection method are described in other parts of this specification. When the liveness detection model training method and the liveness detection method are executed on a client 200, the computing device 600 can be the client 200. When the liveness detection model training method and the liveness detection method are executed on a server 300, the computing device 600 can be the server 300. When the liveness detection model training method and the liveness detection method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0053] like Figure 2As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0054] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0055] I / O component 660 supports input / output between computing device 600 and other components.

[0056] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0057] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the liveness detection model training method and liveness detection method provided in this specification.

[0058] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute the at least one instruction set described above. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the liveness detection model training method and the liveness detection method provided in this specification. Processor 620 can execute all the steps included in the liveness detection model training method and the liveness detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0059] Figure 3 A flowchart of a liveness detection model training method P100 according to an embodiment of this specification is shown. As previously described, computing device 600 can execute the liveness detection model training method P100 of this specification. Specifically, processor 620 can read an instruction set stored in its local storage medium and then execute the liveness detection model training method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0060] S110: Obtain a set of facial image samples.

[0061] The facial image sample set includes multiple facial image samples. A facial image sample can be understood as a facial image used to train a pre-defined liveness detection model. Each facial image can include the face of at least one object; when the object is a person, it can be understood as a human face image. The type of facial image can be various, such as a color facial image, a grayscale facial image, or other types of images. Color facial images can include facial images in various color spaces, such as RGB (a color space), HSV (a color space), YCbCr (a color space), Lab (a color space), and YUV (a color space), etc.

[0062] There are several ways to obtain a set of facial image samples, including the following:

[0063] For example, the processor 620 can receive a set of facial image samples uploaded by the target user 100 through a terminal or client 200, or it can obtain at least one facial image from a network or image database to obtain a set of facial image samples, or it can obtain an original set of images and filter out at least one image containing a face from the original set to obtain a set of facial image samples, or when there are many facial image samples in the set of facial image samples, it can also receive a model training request, which carries the storage address of the set of facial image samples, and obtain the set of facial image samples based on the storage address.

[0064] S120: An image modality conversion network is used to convert each facial image sample in the facial image sample set into an infrared image sample and a depth image sample.

[0065] The image modality conversion network is trained by introducing random variables corresponding to facial images. The image modality conversion network includes a network that performs modality conversion on images. There are various types of modality conversion, such as conversion between RGB images and infrared (IR) images, or between RGB images and depth images, etc. An infrared image can be understood as an image formed by receiving reflected or emitted infrared light. A depth image, also known as a range image, refers to an image where the distance (depth) from the image acquisition device to various points in the scene is used as pixel values; it directly reflects the geometry of the visible surfaces of objects. The image modality conversion network can include an infrared image generation subnetwork and a depth estimation subnetwork. The infrared image generation subnetwork can convert RGB images to infrared images, and the depth estimation subnetwork can convert RGB images to depth images.

[0066] In this context, infrared image samples can be understood as infrared images corresponding to facial image samples, and depth image samples can be understood as depth images corresponding to facial image samples.

[0067] There are several ways to use an image modality conversion network to convert each facial image sample in the facial image sample set into infrared image samples and depth image samples, as follows:

[0068] For example, the processor 620 can use an infrared image subnetwork to map each facial image sample to an infrared image sample, and use a depth estimation subnetwork to perform depth estimation on each facial image sample to obtain a depth image sample corresponding to each facial image sample, as follows:

[0069] S121: An infrared image subnetwork is used to map each facial image sample to an infrared image sample.

[0070] For example, the processor 620 can obtain the mapping matrix corresponding to the infrared image generation sub-network, and based on the mapping matrix, map each facial image sample to an infrared image sample.

[0071] The mapping matrix includes the mapping relationship between facial images and infrared images.

[0072] The infrared image generation subnetwork can have various network structures, such as recurrent adversarial networks (CycleGAN) or other adversarial networks. Taking a recurrent adversarial network as an example, the training method for this infrared image generation subnetwork can be varied. For instance, the processor 620 can acquire at least one image sample pair, where each image sample pair includes a current facial image and a current infrared image corresponding to the current facial image. Based on the current facial image, a first random variable corresponding to the current facial image is randomly generated, and this first random variable is used as first constraint information. Furthermore, the preset infrared image generation subnetwork is trained based on the image sample pair, and the training of the preset infrared image generation subnetwork is perturbed by the first constraint information to obtain the infrared image generation subnetwork.

[0073] The image sample pair includes the current facial image and the corresponding current infrared image. There are multiple ways to obtain at least one image sample pair. For example, taking the current facial image as an RGB image, the processor 620 can use an RGB camera and an infrared camera to capture facial images of the same user, thus obtaining an image sample pair. Similarly, capturing facial images of at least one user will yield at least one image sample pair. Alternatively, a camera from the same source can simultaneously capture RGB and infrared images of at least one user, thus obtaining at least one image sample pair.

[0074] After acquiring at least one image sample pair, a first random variable corresponding to the current face can be randomly generated based on the current face image in the image sample pair. The first random variable can be understood as the image obtained after randomly processing the current face image. There are multiple ways to randomly generate the first random variable corresponding to the current face image. For example, the processor 620 can use a random generator to randomly process the current face image to obtain the first random variable corresponding to the current face image.

[0075] The random processing can take many forms. For example, it can include adding a mask to any location in the current facial image, adjusting one or more pixel values ​​in the current facial image, randomly splitting the current facial image into multiple facial sub-images, randomly adding image styles to any location in the current facial image, or randomly adjusting the image size or resolution of the current facial image, and so on. After generating the first random variable, it can be used as the first constraint information.

[0076] After obtaining the first constraint information, the preset infrared image generation subnetwork can be trained based on the image sample pairs. The infrared image generation subnetwork can be obtained by perturbing the preset infrared generation subnetwork during training using the first constraint information.

[0077] By training a pre-defined infrared image generation sub-network based on image samples, the sub-network can learn the point-to-point mapping relationship between the current facial image and the current infrared image. Since the number of facial image samples is limited, the infrared images output by the sub-network cannot easily cover multiple scenes and domains. However, by introducing a first random variable as a constraint during training, the number of facial image sample categories can be greatly increased. This allows the sub-network to learn a generalized mapping relationship between facial images and infrared images. Therefore, the infrared image samples generated by the trained sub-network have a high degree of diversity, thus enabling the generation of diverse infrared image samples from facial image samples (RGB images).

[0078] S122: Use a depth estimation subnetwork to perform depth estimation on each face image to obtain a depth image sample corresponding to each face image.

[0079] For example, the processor 620 can use a depth estimation subnetwork to perform monocular relative depth estimation on each face image sample, and generate a depth image sample corresponding to each face image sample based on the relative depth estimation results.

[0080] The depth estimation subnetwork can have various network structures, such as MiDaS (a monocular relative depth estimation network) or other monocular / binocular depth estimation networks. Taking MiDaS as an example, the training method for the depth estimation subnetwork can be varied. For instance, the processor 620 can acquire target image samples, randomly generate a second random variable corresponding to the target image samples based on the target image samples, use the second random variable as the second constraint information, and train a preset depth estimation subnetwork based on the target image samples. The training of the preset depth estimation subnetwork can be perturbed by the second constraint information to obtain the depth estimation subnetwork.

[0081] The training process of the depth estimation subnetwork is similar to that of the infrared image generation subnetwork, both involving the introduction of random variables to perturb the training process. The specific process is detailed above and will not be repeated here. By introducing a second random variable, monocular relative depth estimation is performed using MiDaS, thereby enabling the generation of diverse depth image samples from facial image samples (RGB images).

[0082] S130: Based on each facial image sample, infrared image sample, and depth image sample, train the preset liveness detection model to obtain the liveness detection model.

[0083] For example, the processor 620 can select candidate facial image samples from the set of facial image samples based on each facial image sample, infrared image sample, and depth image sample, send the candidate facial image samples to the annotation server for annotation, and receive the annotated target facial image samples returned by the annotation server. Based on the target facial image samples, the processor can train a preset liveness detection model to obtain the liveness detection model, as follows:

[0084] S131: Based on each facial image sample, infrared image sample, and depth image sample, select candidate facial image samples from the set of facial image samples.

[0085] Candidate facial image samples can be understood as facial image samples that are likely to be difficult samples (positive optimization samples) selected from facial image samples through an active learning framework. Difficult samples or positive optimization samples can be understood as generated samples that positively optimize the preset liveness detection model, or as facial image samples that are more difficult to detect. These difficult samples can improve the training accuracy and efficiency of the preset liveness detection model.

[0086] There are several ways to select candidate facial image samples from the set of facial image samples based on each facial image sample, infrared image sample, and depth image sample. These methods can be as follows:

[0087] For example, the processor 620 can perform liveness detection on each facial image sample to select a first facial image sample from the set of facial image samples, and select a second facial image sample from the current set of facial image samples based on infrared image samples and depth image samples. The current set of facial image samples includes facial image samples other than the first facial image sample from the set of facial image samples, and uses the first facial image sample and the second facial image sample as candidate facial image samples.

[0088] There are several ways to perform liveness detection on each facial image sample to select the first facial image sample from the set of facial image samples. For example, the processor 620 can perform liveness detection on each facial image sample to obtain the first liveness prediction value of each facial image sample, obtain the difference between the first liveness prediction value and the preset liveness threshold, and select facial image samples from the set of facial image samples whose difference is within the preset difference range to obtain the first facial image sample.

[0089] There are several ways to perform liveness detection on each facial image sample. For example, the processor 620 can perform liveness detection on the facial image sample through a trained liveness detection network to obtain the first liveness prediction value for each facial image sample. Alternatively, each facial image sample can be sent to a liveness detection server so that the liveness detection server can perform liveness detection on the facial image sample and receive the first liveness prediction value for each facial image sample returned by the liveness detection server.

[0090] The liveness prediction value can be understood as a prediction value that indicates that the facial image sample belongs to a live person. There can be various types of liveness prediction values, such as liveness probability, liveness score, liveness fraction, or any value that can characterize the liveness of the facial image sample, etc.

[0091] The preset liveness threshold can be understood as a pre-set threshold used to determine whether a user's face in a facial image sample is alive. After obtaining the difference between the first liveness prediction value and the preset liveness threshold, facial image samples with differences within the preset difference range can be selected from the facial image sample set to obtain the first facial image sample. For example, taking the first liveness prediction value as the liveness probability and the preset difference range as 5%, facial image samples with a probability difference of less than 5% between the liveness probability and the preset liveness threshold can be selected from the facial image sample set to obtain the first facial image sample. For instance, taking a preset liveness threshold of 70% as an example, this is equivalent to selecting facial image samples with a liveness probability between 65% and 75% from the facial image sample set to obtain the first facial image sample. By limiting the range of this preset difference, facial image samples close to the preset liveness threshold can be selected from the facial image samples. Because these facial image samples are relatively difficult to detect, they can be used as difficult samples for training.

[0092] The current facial image sample set may include facial image samples other than the first facial image sample. A second facial image sample is selected from the current facial image sample set based on infrared image samples and depth image samples. There are several ways to select the second facial image sample. For example, the processor 620 can select candidate infrared image samples corresponding to each facial image sample in the current facial image sample set from the infrared image samples, and candidate depth image samples corresponding to each facial image sample in the current facial image sample set from the depth image samples. Liveness detection is then performed on the candidate infrared image samples and the candidate depth image samples respectively to obtain a second liveness value for each facial image sample in the current facial image sample set. Based on the second liveness prediction value, a second facial image sample is selected from the current facial image sample set.

[0093] There are multiple ways to perform liveness detection on candidate infrared image samples and candidate depth image samples respectively to obtain the second liveness value of each face image sample in the current face image sample set. For example, the processor 620 can perform liveness detection on candidate infrared image samples to obtain the first candidate liveness prediction value of the candidate infrared image samples, perform depth detection on candidate depth image samples to obtain the second candidate liveness prediction value of the depth image samples, and determine the second liveness prediction value of each face image sample in the current face image sample set based on the first candidate liveness prediction value and the second candidate liveness prediction value.

[0094] There are several ways to determine the second liveness prediction value of each facial image sample in the current facial image sample set based on the first candidate liveness prediction value and the second candidate liveness prediction value. For example, the processor 620 can directly use the first candidate liveness prediction value or the second candidate liveness prediction value as the second liveness prediction value of each facial image sample in the current facial image sample set. Alternatively, the processor 620 can compare the first candidate liveness prediction value with the second candidate liveness prediction value and select the second liveness prediction value of each facial image sample in the current facial image sample set from the first candidate liveness prediction value and the second candidate liveness prediction value based on the comparison result. Alternatively, the processor 620 can obtain a preset liveness weighting coefficient and weight the first candidate liveness prediction value and the second candidate liveness prediction value based on the liveness weighting coefficient. The weighted first candidate liveness prediction value and the weighted second candidate liveness prediction value are then added together to obtain the second liveness prediction value of each facial image sample in the current facial image sample set.

[0095] After determining the second liveness prediction value for each facial image sample in the current facial image sample set, a second facial image sample can be selected from the current facial image sample set based on the second liveness prediction value. The method for selecting the second facial image sample is similar to the method for selecting the first facial image sample from the facial image sample set, as detailed above, and will not be repeated here.

[0096] After selecting the first facial image sample and the second facial image sample, the second facial image sample and the second facial image sample can be used as candidate facial image samples.

[0097] It should be noted that when selecting the first and second facial image samples, the liveness detection model trained in the previous iteration (the initial liveness detection model) can be used to continuously improve the model's coverage of diverse scenarios, thereby achieving better cross-scenario and cross-domain deployment.

[0098] S132: Perform liveness annotation on the candidate facial image samples to obtain the target facial image sample.

[0099] The target facial image samples include facial image samples that have been positively optimized using the preset liveness detection model; these can also be understood as difficult samples. Compared to candidate facial image samples, target facial image samples can be understood as further difficult samples obtained after annotating and reviewing the candidate facial image samples. The accuracy of target facial image samples as difficult samples is higher than that of candidate facial image samples as difficult samples.

[0100] Liveness labeling can include adding labels to candidate facial image samples to indicate whether they belong to a live person, and can also include operations such as reviewing and updating the original liveness labels in candidate facial image samples.

[0101] There are several ways to perform liveness annotation on candidate facial image samples to obtain target facial image samples, as follows:

[0102] For example, the processor 620 can send candidate facial image samples to a liveness annotation server so that the liveness annotation server can perform liveness annotation on the candidate facial images and receive the annotated target facial image samples returned by the annotation server. Alternatively, it can perform liveness annotation on the candidate facial image samples based on the liveness prediction values ​​(first liveness prediction value and second liveness prediction value) of the candidate facial image samples to obtain the target facial image samples.

[0103] After sending candidate facial image samples to the annotation server for liveness annotation, there are several ways to perform liveness annotation on the candidate facial image samples on the annotation server side. For example, the annotation server can perform liveness recognition on the candidate facial image samples manually, and based on the liveness recognition results, clean the candidate facial image samples and add liveness tags to the cleaned candidate facial image samples to obtain the target facial image samples. Alternatively, the annotation server can also perform liveness annotation on the candidate facial image samples through a liveness annotation network to obtain the target image samples.

[0104] Among them, manually reviewing and labeling candidate facial image samples can greatly increase the accuracy of target facial image samples. This allows for the extraction of facial image samples that are positive for optimization of the preset liveness model from the facial image sample set, and further improves the accuracy of the trained liveness detection model.

[0105] S133: Train the preset liveness detection model based on the target facial image samples to obtain the liveness detection model.

[0106] For example, the processor 620 can select the target infrared image sample corresponding to the target face image sample from the infrared image sample, select the target depth image sample corresponding to the target face image sample from the depth image sample, and take the target face image sample, the target infrared image sample and the target depth image sample as a multimodal image group, and train the preset liveness detection model based on the multimodal image group to obtain the liveness detection model.

[0107] The multimodal image set can be understood as an image set composed of facial images of different modalities corresponding to the same face. This multimodal image set can include three image modalities, such as two-dimensional images (RGB images), infrared images (IR images), and depth images. There are various ways to train a preset liveness detection model based on the multimodal image set. For example, the processor 620 can use the preset liveness detection model to extract multimodal features from the multimodal image set to obtain liveness features corresponding to the target facial image sample. Based on the liveness features, a third liveness prediction value for the target facial image sample is determined. The preset liveness detection model is then converged based on the third liveness prediction value to obtain an initial liveness detection model. Finally, the initial liveness detection model is iteratively updated based on the third liveness prediction value to obtain the final liveness detection model. Specifically, this can be done as follows:

[0108] (1) A preset liveness detection model is used to extract multimodal features from the multimodal image group to obtain the liveness features corresponding to the target facial image sample.

[0109] Liveness features can be understood as feature information characterizing the degree of liveness of the face in a target facial image sample. There are several ways to extract multimodal features from a multimodal image set using a pre-defined liveness detection model to obtain the liveness features corresponding to the target facial image sample. These methods include:

[0110] For example, the processor 620 can use the feature extraction network in the preset liveness detection model to extract multimodal features from the multimodal image group, obtain the high-level features corresponding to each modality, and fuse the high-level features to obtain the liveness features corresponding to the target facial image sample.

[0111] High-level features can be understood as the high-level semantic features of each modality of facial image. The feature values ​​of high-level semantic features can represent the content that the user can see. For example, taking a face image as an example, the low-level features extracted from a face image can include texture information such as the contour of the face, nose, and eyes, while the extracted high-level features can be displayed as a face. High-level features have richer semantic information, but the target location is relatively coarse.

[0112] The feature extraction network of the preset liveness detection model can include an image extraction subnetwork, an infrared extraction subnetwork, and a depth extraction subnetwork. The image extraction subnetwork is mainly used to extract high-level features from the target facial image samples, the infrared extraction subnetwork is mainly used to extract high-level features from the target infrared image samples, and the depth extraction subnetwork is mainly used to extract high-level features from the target depth image samples. Therefore, there are multiple ways to use the feature extraction network of the preset liveness detection model to perform multimodal feature extraction on the multimodal image group. For example, the processor 620 can use the image extraction subnetwork to perform multi-dimensional feature extraction on the target facial image samples to obtain facial image features, use the infrared extraction subnetwork to perform multi-dimensional feature extraction on the target infrared image samples to obtain facial infrared features, and use the depth extraction subnetwork to perform multi-dimensional feature extraction on the target depth image samples to obtain facial depth features, and also use the facial image features, facial infrared features, and facial depth features as high-level features respectively.

[0113] Multidimensional feature extraction can be understood as using at least one network layer for feature extraction, with the output of the previous network layer serving as the input of the next. Each network layer corresponds to a different feature dimension. Image extraction subnetworks, infrared extraction subnetworks, and depth extraction subnetworks can each include at least one network layer. The structure of the network layers in different extraction subnetworks can be the same or different. Furthermore, the number of network layers in different extraction subnetworks can also be the same or different. Taking a 3resblock network structure as an example, the extraction subnetwork can include three network layers: res1, res2, and res3. When the structure and number of network layers in the image extraction subnetwork, infrared extraction subnetwork, and depth extraction subnetwork are the same, it can be understood that the image extraction subnetwork, infrared extraction subnetwork, and depth extraction subnetwork use a weight-sharing network structure. A weight-sharing network structure can be understood as sharing weights within the network layers corresponding to each dimension. For example, consider an image extraction sub-network consisting of three layers (res1, res2, and res3), an infrared extraction sub-network also consisting of three layers (res1, res2, and res3), and a depth extraction sub-network also consisting of three layers (res1, res2, and res3). In this case, feature interconnection can be performed in the res1 layer of the three different extraction sub-networks to achieve weight sharing; feature interconnection can be performed in the res2 layer of the three different extraction sub-networks; and feature interconnection can be performed in the res3 layer of the three different extraction sub-networks. However, it should be noted that feature interconnection cannot be performed between res1 and res2.

[0114] There are several ways to extract multi-dimensional features from the target facial image sample using an image extraction sub-network to obtain facial image features. For example, when the target facial image sample is a single-frame image, the processor 620 can use the first layer of the image extraction sub-network to extract features from the target facial image sample, and then input the extracted features into the next network layer for feature extraction, and so on, until the last network layer outputs facial image features, and use the facial image features as the high-level features of the modality corresponding to the target facial image sample. When the target facial image sample is a target video frame, the temporal information of the target facial image sample can also be obtained. The image feature extraction sub-network can be used to extract multi-dimensional features from the target facial image sample to obtain initial facial image features, and the image extraction sub-network can be used to extract features from the temporal information to obtain temporal features. The initial facial features and temporal features are then fused to obtain facial image features, and the facial image features are used as the high-level features of the modality corresponding to the target facial image sample.

[0115] The method of multidimensional feature extraction using infrared extraction subnetwork and depth extraction subnetwork is similar to the method of multidimensional feature extraction using image extraction subnetwork, as detailed above, and will not be repeated here.

[0116] It should be noted that this scheme not only considers single-frame multimodal images as input, but also incorporates video temporal information when using video frame multimodal images as input. This allows for the construction of a video-based multimodal framework, which can further improve the attack recall capability of the liveness detection model.

[0117] After extracting the high-level features corresponding to each modality, these features can be fused to obtain the liveness features corresponding to the target facial image sample. There are several ways to fuse high-level features. For example, the processor 620 can directly concatenate or add the high-level features of each modality to obtain the liveness features corresponding to the target facial image sample. Alternatively, it can obtain the fusion weights corresponding to each modality separately, and based on these weights, weight the high-level features corresponding to that modality separately, then concatenate or add the weighted high-level features to obtain the liveness features corresponding to the target facial image sample.

[0118] (2) Based on the liveness features, determine the third liveness prediction value of the target facial sample.

[0119] For example, the processor 620 can use a liveness detection network in a preset liveness detection model to perform liveness classification prediction on the target facial image sample based on liveness features, thereby obtaining a third liveness prediction value. Alternatively, it can use a liveness detection network in a preset liveness detection model to perform liveness scoring based on liveness features, thereby obtaining a liveness score value for the target facial image sample, and using this liveness score value as the third liveness prediction value. Or, it can also use a liveness detection network in a preset liveness detection model to determine the liveness probability of the target face in the target facial image sample based on liveness features, and use this liveness probability as the third liveness prediction value, and so on.

[0120] (3) Based on the third liveness prediction value, the preset liveness detection model is converged to obtain the liveness detection model.

[0121] For example, the processor 620 can update the network parameters of the preset liveness detection model based on the third liveness prediction value to obtain an initial liveness detection model, update the preset liveness detection model to the initial liveness detection model, and return to the step of obtaining the target facial image sample set until the preset liveness detection model converges, thus obtaining the liveness detection model.

[0122] Here, network parameters can be understood as the hyperparameters of each network or network layer in the preset liveness detection model. Based on the third liveness prediction value, there are several ways to update the network parameters of the preset liveness detection model. For example, the processor 620 can obtain the liveness label of the target facial image sample, and based on the liveness label and the third liveness prediction value, determine the target loss information of the target facial image sample. Based on this target loss information, the network parameters of the preset liveness detection model are updated to obtain the initial liveness detection model.

[0123] There are several ways to update the network parameters of the preset liveness detection model based on the target loss information. For example, the processor 620 can use the gradient descent algorithm to update the network parameters of the preset liveness detection model based on the target loss information to obtain the initial liveness detection model. Alternatively, other network parameter update algorithms can be used to update the network parameters of the preset liveness detection model based on the target loss information to obtain the initial liveness detection model.

[0124] After obtaining the initial liveness detection model, the preset liveness detection model can be updated to the initial liveness detection model. Then, the step of obtaining the facial image sample set is returned until the preset liveness detection model converges, from which the liveness detection model can be obtained.

[0125] It's important to note that training the preset liveness detection model can be done iteratively. Each round of training of the liveness detection model (the preset model) serves as the basis for the next round of the active learning framework to filter new positive data (target facial image samples). In other words, when using the facial image sample set, the active framework can use the previously trained liveness detection model (the preset model) to perform liveness detection on the facial image samples, their corresponding depth images, and infrared images, thus filtering out positive data. This iterative approach continuously improves the model's coverage of diverse scenarios, enabling better cross-scene and cross-domain deployment.

[0126] The liveness detection model trained in this scheme can be understood as a multimodal liveness attack detection model. This multimodal liveness attack detection can perform liveness detection on multimodal facial images, including RGB images, infrared (IR) images, and depth images. This multimodal liveness attack detection model is trained using recurrent adversarial methods and relative depth data generation. Taking image modality transformation networks, including CycleGAN and MiDaS networks, as an example, the training process of the multimodal liveness attack detection model can be as follows: Figure 4 As shown, it can be divided into two stages, as detailed below:

[0127] (1) Data Generation Stage: IR data generation based on recurrent adversarial model. The input consists of pairs of RGB and IR images fed into the generator network. Additionally, random variables generated from the RGB images are incorporated into the generator network's learning process as constraints. This training process establishes the mapping relationship between RGB and IR images, allowing the generation of corresponding IR images from the RGB images using the mapping matrix. Next, depth data generation based on relative depth estimation is performed. Random variables are also introduced, and MiDaS is used for monocular relative depth estimation, enabling the generation of diverse depth images from RGB images. Finally, the augmented IR and depth images are filtered using an active learning framework to extract positively generated samples for model optimization, thus expanding data diversity and yielding the final training data (multimodal image sample set).

[0128] (2) Multimodal liveness detection model training stage: For the three-modal data, the basic model based on 3resblock is used to extract features, and then the extracted high-level features are fused to achieve the final liveness detection prediction. Based on the prediction results, the preset liveness detection model is converged to obtain the liveness detection model.

[0129] It should be noted that the overall data generation and liveness detection training are presented in a cyclical iteration. The liveness detection model trained in each epoch (round) serves as the basis for the next round of active learning framework to select new positive data, thereby continuously improving the model's ability to cover diverse scenarios and achieving better cross-scenario and cross-domain deployment.

[0130] This scheme uses random variables as the basis for generating diverse data. It generates diverse data such as infrared (IR) images and depth images through cyclic adversarial methods and relative depth estimation, respectively. Simultaneously, an active learning framework iteratively filters the generated data to positively expand its diversity. Incorporating this generated diverse data into the training of the multimodal liveness detection model ensures that the model has good multi-scenario and multi-domain adaptability during actual deployment.

[0131] After training the preset liveness detection model to obtain the liveness detection model, the liveness detection model can be used for liveness detection.

[0132] Figure 5 A flowchart of a liveness detection method P200 according to some embodiments of this specification is shown. As previously described, computing device 600 can execute the liveness detection method P200 described herein. Specifically, processor 620 can read an instruction set stored in its local storage medium and then execute the liveness detection method P200 described herein according to the instructions in the instruction set. Figure 5 As shown, method P200 may include:

[0133] S210: Acquire the target multimodal image group.

[0134] The target multimodal image group includes facial images of multiple modalities, such as at least two of the following: RGB modal facial images, infrared modal infrared images, and depth modal depth images.

[0135] There are several ways to obtain the target multimodal image group, as follows:

[0136] For example, the processor 620 can directly receive the target multimodal image set uploaded by the target user 100 through the terminal or client 200; or, it can obtain multiple modal facial images corresponding to the same face from a network or image database to obtain the target multimodal image set; or, it can acquire multimodal images of the user's face through a multimodal image acquisition device to obtain the target multimodal image set; or, it can receive the target facial image uploaded by the target user 100 through the terminal or client 200 and convert the target facial image into a facial infrared image and a facial depth image through an image modality conversion network, and then convert the target facial image and facial depth image into a target facial image. Infrared images and facial depth images can be used as a target multimodal image set. Alternatively, target facial images can be obtained from a network or image database and converted into facial infrared images and facial depth images through an image modality conversion network. Alternatively, the target facial images can be obtained by acquiring the target user's facial images through an RGB image acquisition device and converted into facial infrared images and facial depth images through an image modality conversion network.

[0137] S220: Input the target multimodal pattern group into the liveness detection model and output the liveness detection result corresponding to the target multimodal image group.

[0138] For example, the processor 620 can directly input the target multimodal image group to the... Figure 3 The training method is used to train a liveness detection model, which performs liveness detection on facial images of multiple modalities in a target multimodal image group, thereby obtaining the liveness detection results corresponding to the target multimodal image group.

[0139] It should be noted that when using a liveness detection model to perform liveness detection on a multimodal image set, the feature extraction subnetwork corresponding to each modality in the liveness detection model can be used to extract high-level features of the facial image for that modality. These high-level features are then fused to obtain the liveness features corresponding to the multimodal image set. Based on these liveness features, the liveness detection result for the multimodal image set is determined. The liveness detection model can include feature extraction subnetworks corresponding to three modalities. When the multimodal image set includes RGB images, infrared images, and depth images, the three feature extraction subnetworks can be used respectively to extract high-level features, thereby determining the liveness detection result. When the multimodal image set includes two modalities, the corresponding two feature extraction subnetworks can be used to extract high-level features, and the liveness detection result is determined based on these two high-level features. Alternatively, a preset high-level feature for another modality (excluding the two modalities) can be obtained and fused with the high-level features of the two modalities to determine the liveness detection result.

[0140] In summary, the liveness detection model training method P100, liveness detection method P200, and system 001 provided in this specification, after acquiring a set of facial image samples, use an image modality conversion network to convert each facial image sample in the set into an infrared image sample and a depth image sample. This image modality conversion network is trained by introducing random variables. Based on each facial image sample, infrared image sample, and depth image sample, a preset liveness detection model is trained to obtain the liveness detection model. Since this scheme can convert facial images into image samples of multiple modalities through an image modality conversion network, and since this image modality conversion network is trained by introducing random variables, it can convert diverse multimodal data, expanding the diversity of training samples, thereby improving the multi-scene deployment capability of the liveness detection model. Therefore, it can improve the liveness detection accuracy of the liveness detection model.

[0141] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing data processing. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the liveness detection model training method P100 and the liveness detection method P200 described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the liveness detection model training method P100 and the liveness detection method P200 described in this specification. The program product for implementing the above methods may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages.The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0142] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0143] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0144] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0145] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0146] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0147] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for training a liveness detection model, comprising: Obtain a set of facial image samples; An image modality conversion network is used to convert each facial image sample in the facial image sample set into an infrared image sample and a depth image sample. The image modality conversion network is trained by image sample pairs and random variables. The image sample pairs include the current facial image and the current infrared image or the current depth image corresponding to the current facial image. The random variables are images obtained by randomly processing the current facial image. as well as Based on each facial image sample, the infrared image sample, and the depth image sample, a preset liveness detection model is trained to obtain a liveness detection model.

2. The liveness detection model training method according to claim 1, wherein, The image mode conversion network includes an infrared image generation subnetwork and a depth estimation subnetwork, and The step of using an image modality conversion network to convert each facial image sample in the facial image sample set into an infrared image sample and a depth image sample includes: The infrared image generation subnetwork is used to map each facial image sample to an infrared image sample; and The depth estimation subnetwork is used to perform depth estimation on each facial image sample to obtain a depth image sample corresponding to each facial image sample.

3. The liveness detection model training method according to claim 2, wherein, The step of mapping each facial image sample to an infrared image sample using the infrared image generation sub-network includes: Obtain the mapping matrix corresponding to the infrared image generation sub-network, the mapping matrix including the mapping relationship between the facial image and the infrared image; and Based on the mapping matrix, each facial image sample is mapped to an infrared image sample.

4. The liveness detection model training method according to claim 2, wherein, The training of the infrared image generation subnetwork includes the following steps: Acquire at least one image sample pair, wherein each image sample pair includes a current face image and a current infrared image corresponding to the current face image; Based on the current facial image, a first random variable corresponding to the current facial image is randomly generated, and the first random variable is used as the first constraint information; and The preset infrared image generation subnetwork is trained based on the image sample pairs, and the training of the preset infrared image generation subnetwork is perturbed by the first constraint information to obtain the infrared image generation subnetwork.

5. The liveness detection model training method according to claim 2, wherein, The training of the depth estimation subnetwork includes the following steps: Obtain target image samples; Based on the target image sample, a second random variable corresponding to the target image sample is randomly generated, and the second random variable is used as the second constraint information; and The preset depth estimation subnetwork is trained based on the target image samples, and the training of the preset depth estimation subnetwork is perturbed by the second constraint information to obtain the depth estimation subnetwork.

6. The liveness detection model training method according to claim 1, wherein, The process of training a preset liveness detection model based on each facial image sample, the infrared image sample, and the depth image sample to obtain a liveness detection model includes: Based on each facial image sample, the infrared image sample, and the depth image sample, candidate facial image samples are selected from the set of facial image samples; Liveness annotation is performed on the candidate facial image samples to obtain target facial image samples, wherein the target facial image samples include facial image samples optimized for the preset liveness detection model; and The preset liveness detection model is trained based on the target facial image samples to obtain the liveness detection model.

7. The liveness detection model training method according to claim 6, wherein, The step of selecting candidate facial image samples from the set of facial image samples based on each facial image sample, the infrared image sample, and the depth image sample includes: Liveness detection is performed on each of the facial image samples to select a first facial image sample from the set of facial image samples; Based on the infrared image sample and the depth image sample, a second facial image sample is selected from the current facial image sample set, wherein the current facial image sample set includes facial image samples other than the first facial image sample in the facial image sample set; and The first facial image sample and the second facial image sample are used as the candidate facial image samples.

8. The liveness detection model training method according to claim 7, wherein, The step of performing liveness detection on the facial image samples to select a first facial image sample from the set of facial image samples includes: Perform liveness detection on each facial image sample to obtain a first liveness prediction value for each facial image sample; Obtain the difference between the first predicted liveness value and the preset liveness threshold; and A first facial image sample is obtained by selecting facial image samples whose differences are within a preset difference range from the set of facial image samples.

9. The liveness detection model training method according to claim 7, wherein, The step of selecting a second facial image sample from the current facial image sample set based on the infrared image sample and the depth image sample includes: Select candidate infrared image samples corresponding to each facial image sample in the current facial image sample set from the infrared image samples; Select candidate depth image samples corresponding to each face image sample in the current face image sample set from the depth image samples; Liveness detection is performed on the candidate infrared image samples and the candidate depth image samples respectively to obtain a second liveness prediction value for each face image sample in the current face image sample set; and Based on the second liveness prediction value, a second facial image sample is selected from the current facial image sample set.

10. The liveness detection model training method according to claim 6, wherein, The step of training the preset liveness detection model based on the target facial image samples to obtain the liveness detection model includes: Select the target infrared image sample corresponding to the target face image sample from the infrared image sample; Select the target depth image sample corresponding to the target face image sample from the depth image samples; and The target facial image sample, the target infrared image sample, and the target depth image sample are used as a multimodal image group, and the preset liveness detection model is trained based on the multimodal image group to obtain the liveness detection model.

11. The liveness detection model training method according to claim 10, wherein, The step of training the preset liveness detection model based on the multimodal image set to obtain the liveness detection model includes: The preset liveness detection model is used to extract multimodal features from the multimodal image group to obtain the liveness features corresponding to the target facial image sample; Based on the liveness features, a third liveness prediction value is determined for the target facial image sample; and Based on the third liveness prediction value, the preset liveness detection model is converged to obtain the liveness detection model.

12. The liveness detection model training method according to claim 11, wherein, The step of using the preset liveness detection model to extract multimodal features from the multimodal image group to obtain the liveness features corresponding to the target facial image sample includes: A feature extraction network based on a pre-defined liveness detection model is used to extract multimodal features from the multimodal image set, obtaining high-level features corresponding to each modality; and The high-level features are fused to obtain the liveness features corresponding to the target facial image sample.

13. The liveness detection model training method according to claim 12, wherein, The feature extraction network includes an image extraction subnetwork, an infrared extraction subnetwork, and a depth extraction subnetwork, and The feature extraction network employing a preset liveness detection model performs multimodal feature extraction on the multimodal image group to obtain high-level features corresponding to each modality, including: The image extraction sub-network is used to extract multidimensional features from the target facial image sample to obtain facial image features; The infrared extraction sub-network is used to extract multidimensional features from the target infrared image sample to obtain facial infrared features; The depth extraction sub-network is used to extract multidimensional features from the target depth image samples to obtain facial depth features; and The facial image features, the facial infrared features, and the facial depth features are respectively used as the high-level features.

14. The liveness detection model training method according to claim 13, wherein, The image extraction subnetwork, the infrared extraction subnetwork, and the depth extraction subnetwork adopt a weight-sharing network structure.

15. The liveness detection model training method according to claim 13, wherein, The step of using the image extraction sub-network to extract multidimensional features from the target facial image sample to obtain facial image features includes: When the target facial image sample is a target video frame, the time information of the target facial image sample is obtained; The image extraction sub-network is used to extract multidimensional features from the target facial image samples to obtain initial facial image features; and The image extraction subnetwork is used to extract features from the time information to obtain time features, and the initial facial features and the time features are fused to obtain the facial image features.

16. The liveness detection model training method according to claim 11, wherein, The step of converging the preset liveness detection model based on the third liveness prediction value to obtain a liveness detection model includes: Based on the third liveness prediction value, the network parameters of the preset liveness detection model are updated to obtain the initial liveness detection model. Update the preset liveness detection model to the initial liveness detection model; and Return to the step of obtaining the set of facial image samples until the preset liveness detection model converges, and obtain the liveness detection model.

17. A method for detecting liveness, wherein, include: Acquire a target multimodal image group, wherein the target multimodal image group includes facial images of multiple modalities; as well as The target multimodal image group is input into the liveness detection model, and the liveness detection result corresponding to the target multimodal image group is output. The liveness detection model is trained according to the liveness detection model training method according to any one of claims 1 to 16.

18. A liveness detection system, comprising: At least one storage medium stores a liveness detection model for performing liveness detection; as well as At least one processor is communicatively connected to the at least one storage medium. When the liveness detection system is running, the at least one processor reads the liveness detection model and implements the liveness detection method of claim 17, wherein the liveness detection model is trained according to the liveness detection model training method of any one of claims 1-16.

Images