Liveness detection methods and systems

Abstract

This specification provides a liveness detection method and system. After acquiring a single-modal target image of a target user, the system performs multimodal liveness detection on the target image based on a target liveness detection model, and outputs the obtained target liveness detection results. The target liveness detection model is a single-modal liveness detection model obtained through knowledge distillation of the multimodal liveness detection model. This allows the single-modal liveness detection model to possess the liveness detection performance of the multimodal liveness detection model, thereby improving the accuracy of single-modal liveness detection and saving on the cost of the image acquisition module.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a liveness detection method and system. Background Technology

[0002] Facial recognition has become a primary method of identity authentication and is widely used in various scenarios such as finance and payment. With the widespread application of facial recognition systems, their security has been challenged by liveness detection attacks. Therefore, it is necessary to add a liveness detection step to facial recognition systems. To improve the security of liveness detection algorithms, two approaches exist: one is to add hardware to acquire multimodal images and perform liveness detection based on these images, which is costly. The other approach is a multi-action-based liveness detection method. This approach requires the user to perform certain specified actions and acquires multiple user images over a period of time to increase the amount of information. Both approaches require significant time and computing power and place higher demands on the terminal devices.

[0003] In summary, there is a need to provide a new method and system for liveness detection that can improve liveness detection performance without increasing hardware costs. Summary of the Invention

[0004] This specification provides a liveness detection method and system that can improve liveness detection performance without increasing hardware costs.

[0005] In a first aspect, this specification provides a liveness detection method, comprising: acquiring a target image of a target user, wherein the target image is a single-modal image of the target user acquired under the target modality; performing multimodal liveness detection on the target image based on a target liveness detection model to obtain a target liveness detection result, wherein the target liveness detection model is a single-modal liveness detection model obtained by knowledge distillation based on the multimodal liveness detection model; and outputting the target liveness detection result.

[0006] In some embodiments, the target modality is one of the multiple modalities corresponding to the multimodal liveness detection model.

[0007] In some embodiments, the knowledge distillation includes: performing pre-knowledge distillation on the target liveness detection model based on the multimodal liveness detection model, wherein the training objective of the pre-knowledge distillation includes that the first predicted liveness classification result output by the target liveness detection model is consistent with the predicted multimodal liveness classification result output by the multimodal liveness detection model.

[0008] In some embodiments, the training objective of the pre-knowledge distillation further includes at least one of the following: the first predicted liveness classification result output by the target liveness detection model is consistent with its corresponding real liveness classification result; and the correlation between the first predicted feature of the target modality output by the target liveness detection model and the predicted multimodal fusion features of the multiple modalities output by the multimodal liveness detection model meets a preset requirement.

[0009] In some embodiments, the preset requirement includes at least one of the following: the mutual information between the first predicted feature of the target modality output by the target liveness detection model and the predicted multimodal fusion feature output by the multimodal liveness detection model approaches a first preset value; and when it is determined that the target liveness detection model outputs the first predicted feature of the target modality, the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model approaches a second preset value.

[0010] In some embodiments, the knowledge distillation further includes: performing advanced knowledge distillation on the target liveness detection model based on the multimodal liveness detection model, wherein the training objective of the advanced knowledge distillation includes prediction distribution consistency, wherein the prediction distribution consistency includes: the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model; and the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model.

[0011] In some embodiments, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model, including at least one of the following: the class center of the predicted objects under the liveness category output by the target liveness detection model is consistent with the class center of the predicted objects under the liveness category output by the multimodal liveness detection model; and the distance between the predicted objects under the liveness category output by the target liveness detection model and the class center of their corresponding liveness category is consistent with the distance between the predicted objects under the liveness category output by the multimodal liveness detection model and the class center of their corresponding liveness category.

[0012] In some embodiments, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model, including at least one of the following: the class center of the predicted objects under the attack category output by the target liveness detection model is consistent with the class center of the predicted objects under the attack category output by the multimodal liveness detection model; and the distance between the predicted objects under the attack category output by the target liveness detection model and the class center of their corresponding attack category is consistent with the distance between the predicted objects under the attack category output by the multimodal liveness detection model and the class center of their corresponding attack category.

[0013] In some embodiments, the prediction object includes the predicted features output by the model and / or the predicted liveness classification result.

[0014] In some embodiments, the training objective of the advanced knowledge distillation further includes: the second predicted liveness classification result output by the target liveness detection model is consistent with its corresponding true liveness classification result.

[0015] In some embodiments, the training objective of the multimodal liveness detection model includes: the predicted liveness classification results of multiple modalities output by the multimodal liveness detection model are consistent with their corresponding true liveness classification results.

[0016] In some embodiments, the training objective of the multimodal liveness detection model further includes at least one of the following: the multiple predicted liveness classification results of the multiple modalities output by the multimodal liveness detection model are consistent with the fused predicted multimodal liveness classification results; and the multiple predicted features of the multiple modalities output by the multimodal liveness detection model are consistent.

[0017] Secondly, this specification also provides a liveness detection system, comprising: at least one storage medium storing at least one instruction set for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the method described in any one of the first aspects according to the instructions of the at least one instruction set.

[0018] As can be seen from the above technical solutions, the liveness detection method and system provided in this specification acquire a single-modal target image of the target user, and then perform multimodal liveness detection on the target image based on the single-modal image using a target liveness detection model, thereby outputting the obtained target liveness detection results. In this solution, the single-modal liveness detection model is obtained by knowledge distillation using a multimodal liveness detection model to obtain the target liveness detection model. The target liveness detection model can characterize the mapping relationship between the single-modal image and the multimodal liveness detection results obtained based on the multimodal image, enabling the single-modal liveness detection model to possess the liveness detection performance of the multimodal liveness detection model, thus achieving an effect close to that of multimodal liveness detection when performing liveness detection based on single-modal images. Furthermore, since this solution is based on single-modal image liveness detection, it can save computing power, improve liveness detection efficiency, and reduce the hardware cost of the image acquisition module. Meanwhile, the target liveness detection model is a lightweight model based on single-modal image detection, which has low requirements for computing power. It can be deployed on terminal devices or remote servers. When deployed on terminal devices, image transmission can be avoided, further saving computing time, while avoiding the risk of privacy leakage during data transmission.

[0019] Other functions of the liveness detection methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the liveness detection methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0022] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0023] Figure 3 A flowchart of a liveness detection method provided according to an embodiment of this specification is shown;

[0024] Figure 4A schematic diagram of the network structure of a preset multimodal liveness detection model provided according to embodiments of this specification is shown; and

[0025] Figure 5 A schematic diagram of the network structure of a preset target liveness detection model provided according to an embodiment of this specification is shown. Detailed Implementation

[0026] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0027] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0028] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0029] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0030] For ease of description, the terms used in this specification are explained as follows:

[0031] Liveness attack: This refers to attack methods that attempt to bypass facial recognition systems by using screens, paper, masks, etc.

[0032] Liveness detection: also known as liveness protection, refers to the technology that uses artificial intelligence models to detect and block liveness attacks, such as those on mobile phone screens, printed paper, etc.

[0033] Distillation learning refers to a method that uses a powerful teacher network to guide a lightweight network in learning. Models using distillation learning can achieve better performance than models that do not use this method.

[0034] Cross-modal distillation: refers to a distillation learning method where the modality of the teacher's network input and the modality of the student's network input are inconsistent.

[0035] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0036] The liveness detection method provided in this specification can be applied to any liveness detection scenario in biometric processes. For example, in scenarios such as face payment or face recognition, the liveness detection method can be used to perform liveness detection on the original image of the biometric features of the user to be paid or identified; in identity verification scenarios, the liveness detection method can be used to perform liveness detection on the original image of the user's biometric features; and it can also be applied to any other liveness detection scenario, which will not be elaborated here. The biometric features may include, but are not limited to, one or more of the following: facial image, iris, sclera, fingerprint, palm print, voiceprint, and skeletal projection. For ease of description, this application will use the application of the liveness detection method in a face recognition scenario to perform liveness detection on a face as an example.

[0037] Those skilled in the art should understand that the liveness detection methods and systems described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0038] Figure 1 This diagram illustrates an application scenario of a liveness detection system 001 provided according to an embodiment of this specification. The liveness detection system 001 (hereinafter referred to as System 001) can be applied to liveness detection in any scenario, such as liveness detection in face payment scenarios, liveness detection in identity verification scenarios, liveness detection in other face recognition scenarios, etc. Figure 1 As shown, system 001 may include terminal device 200 and server 300. The application scenarios of system 001 may include target user 100, system 001, and network 400.

[0039] Target user 100 can be a user who needs to undergo biometric identification, or a user who is currently undergoing biometric identification. Target user 100 can be the object detected by system 001.

[0040] Terminal device 200 can be a device for performing liveness detection on target user 100. In some embodiments, the liveness detection method can be executed on terminal device 200. In this case, terminal device 200 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, terminal device 200 may include hardware devices with data information processing capabilities and necessary programs for driving the hardware devices to work. In some embodiments, terminal device 200 may include mobile devices, tablet computers, laptop computers, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, a head-mounted display, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle television, etc. In some embodiments, the terminal device 200 may include an image acquisition device for acquiring a single-modal target image of the target user 100. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (e.g., an RGB camera), or a two-dimensional image acquisition device (e.g., an RGB camera) and a depth image acquisition device (e.g., a 3D structured light camera, a laser detector, etc.). In some embodiments, the terminal device 200 may be a device with positioning technology for locating the position of the terminal device 200.

[0041] In some embodiments, the terminal device 200 may have one or more applications (APPs) installed. The APPs provide the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email terminals, social media platform software, etc. In some embodiments, the terminal device 200 may have a target APP installed. The target APP can collect target images of the target user's biometric features in a target modality for the terminal device 200. In some embodiments, the target user 100 can also trigger a liveness detection request through the target APP. The target APP can respond to the liveness detection request by executing the liveness detection method described in this specification. The liveness detection method will be described in detail later.

[0042] like Figure 1 As shown, terminal device 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple terminal devices 200 and receive data sent by terminal devices 200. In some embodiments, terminal devices 200 can interact with server 300 through network 400 to receive or send messages, etc. Server 300 can be a server that provides various services, such as a backend server that supports a liveness detection method deployed on multiple terminal devices 200. In some embodiments, the liveness detection method can be executed on server 300. In this case, server 300 can store data or instructions for executing the liveness detection method described in this specification, and can execute or be used to execute the data or instructions. In some embodiments, server 300 may include hardware devices with data information processing capabilities and the necessary programs required to drive the hardware devices.

[0043] Network 400 serves as a medium to provide a communication connection between terminal device 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, terminal device 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable network, wired network, fiber optic network, telecommunications network, intranet, Internet, local area network (LAN), wide area network (WAN), wireless local area network (WLAN), metropolitan area network (MAN), public switched telephone network (PSTN), Bluetooth network. TM ZigBee TMA network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of terminal device 200 and server 300 can connect to network 400 to exchange data or information.

[0044] It should be understood that Figure 1 The number of terminal devices 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, any number of terminal devices 200, servers 300, and networks 400 can be included.

[0045] It should be noted that the liveness detection method can be executed entirely on the terminal device 200, entirely on the server 300, or partially on both. Specifically, the liveness detection method uses a multimodal liveness detection model to perform knowledge distillation on a unimodal liveness detection model, enabling the unimodal liveness detection model to possess the liveness detection performance of a multimodal model. This achieves results close to multimodal liveness detection when performing liveness detection based on unimodal images. Therefore, the liveness detection method saves computational power and improves liveness detection efficiency. The unimodal liveness detection model used is a lightweight model with low computational requirements, allowing deployment on the terminal device 200. This further reduces image transmission, saves computation time, and avoids the risk of privacy leaks during data transmission. The following description will use the execution of the liveness detection method on the terminal device 200 as an example.

[0046] Figure 2 A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can execute the liveness detection method described in this specification. The liveness detection method is described in other parts of this specification. When the liveness detection method is executed on a terminal device 200, the computing device 600 can be the terminal device 200. When the liveness detection method is executed on a server 300, the computing device 600 can be the server 300. When the liveness detection method can be executed partly on the terminal device 200 and partly on the server 300, the computing device 600 can be either the terminal device 200 or the server 300.

[0047] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0048] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0049] I / O component 660 supports input / output between computing device 600 and other components.

[0050] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0051] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the liveness detection method provided in this specification.

[0052] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. At least one processor 620 is used to execute the at least one instruction set described above. When the computing device 600 is running, at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the liveness detection method provided in this specification. Processor 620 can execute all the steps included in the liveness detection method. Processor 620 can be in the form of one or more processors. In some embodiments, processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0053] Figure 3 A flowchart of a liveness detection method P100 according to an embodiment of this specification is shown. As previously described, the computing device 600 can execute the liveness detection method P100 of this specification. Specifically, the computing device 600 can read an instruction set stored in its local storage medium and then execute the liveness detection method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0054] S120: Obtain the target image of the target user.

[0055] The target users are those who are about to undergo liveness detection or are currently undergoing liveness detection.

[0056] The target image is a single-modal image captured from the target user in the target modality. The target image includes the target user's biometric features. Biometric features are inherent physiological characteristics of the human body, which may include at least one of the following: face, iris, sclera, fingerprint, palm print, voiceprint, and skeletal projection; they may also include other inherent physiological characteristics of the human body capable of facial recognition. For ease of description, this specification will use the face as an example of biometric features. Those skilled in the art should understand that other biometric features are also within the scope of protection of this specification.

[0057] The target image can be acquired by the terminal device 200. The terminal device 200 can be a device for performing liveness verification or face verification. In some embodiments, the target user can complete the verification on the terminal device 200 by performing liveness verification or face verification on the terminal device 200.

[0058] The terminal device 200 has an image acquisition module corresponding to the target modality. The target modality is a single modality, referring to a single mode. This modality can be an imaging mode. The target modality can be any one of multiple modalities. Multiple modalities can refer to multiple different imaging modes. Different imaging modes can include imaging modes under different visual domains, imaging modes under different dimensions, thermal imaging modes, etc. Correspondingly, different modalities correspond to different image acquisition modules. The visual domain can refer to the spectral range of the image, such as the ultraviolet domain, visible light domain, near-infrared domain, mid-infrared domain, far-infrared domain, etc. Image acquisition modules for different visual domains include, for example, ultraviolet cameras, visible light cameras, near-infrared cameras, mid-infrared cameras, far-infrared cameras, etc. The working principles of image acquisition modules for different visual domains are different. For example, a visible light camera mainly includes a lens, an image sensor, and an image processor. The lens projects the subject onto the image sensor. The image processor calculates appropriate parameters through metering and ranging, and instructs the lens to focus. When a shooting command is detected (e.g., the target user's face is completely within the viewfinder), the image sensor completes an exposure and converts it into an image through the image processor. Near-infrared cameras mainly consist of an infrared emitter and an infrared receiver. Their working principle is that the infrared emitter emits infrared light to illuminate the subject, and the infrared receiver receives the reflected infrared light, thus forming a near-infrared image. Image acquisition modules with different visual domains capture different modal images, such as ultraviolet images, visible light images, near-infrared images, mid-infrared images, and far-infrared images.

[0059] Image acquisition modules in different dimensions can acquire images in different dimensions, such as 2D cameras and 3D cameras (or depth cameras). A 2D camera can acquire a planar image of the target user 100. A 3D camera can acquire a depth image of the target user 100, which includes depth information of the target user 100, such as the distance between the target user 100 and the 3D camera. Examples of 3D cameras include structured light cameras, TOF cameras, binocular stereo cameras, and laser detectors. A thermal imaging camera, also known as a thermal imager, is used to passively receive infrared radiation energy (heat) emitted by the object being measured and convert this heat energy into a visualized image with temperature data. This visualized image is the thermal image, which displays the temperature distribution on the surface of the object being measured.

[0060] Terminal device 200 acquires images of the target user through an image acquisition module, thereby obtaining a target image corresponding to the target modality. For example, the image acquisition module may include one of an RGB camera module, an infrared camera module, a NIR camera module, a 3D camera module, and a thermal imaging camera module. Correspondingly, the target modality can be any of multiple modalities such as RGB, infrared, NIR, 3D, and thermal imaging. Terminal device 200 acquires images of the target user through one of these modules, thereby obtaining a target image.

[0061] S140: Perform multimodal liveness detection on the target image based on the target liveness detection model to obtain the target liveness detection result.

[0062] The target liveness detection model is a single-modal liveness detection model obtained through knowledge distillation from a multimodal liveness detection model. In other words, the target liveness detection model performs liveness detection based on single-modal images, while its liveness detection results or accuracy are comparable to those of a model based on multimodal images. A multimodal liveness detection model performs liveness detection based on images from multiple modalities. For example, terminal device 200 acquires images from multiple modalities and inputs them into a multimodal liveness detection model for multimodal liveness detection, obtaining multimodal liveness detection results. Multimodal liveness detection results can improve the accuracy of liveness detection.

[0063] Knowledge distillation can transfer the knowledge learned by the multimodal liveness detection model to the target liveness detection model, enabling the target liveness detection model to represent the mapping relationship between the single-modal image and the multimodal liveness detection result. This allows for liveness detection using the single-modal target image, achieving the same or similar results as the multimodal liveness detection result.

[0064] Before performing knowledge distillation on the target liveness detection model based on the multimodal liveness detection model, the computing device 600 needs to acquire the multimodal liveness detection model first. The training process of the multimodal liveness detection model will be described below.

[0065] The multimodal liveness detection model can be trained using the following steps: For example, the computing device 600 acquires multimodal training images and their corresponding labels, and trains the preset multimodal liveness detection model based on the multimodal training images, their corresponding labels, and the training target to obtain the multimodal liveness detection model.

[0066] Here, the label represents the real liveness classification result corresponding to the multimodal training image. The real liveness classification result represents whether the multimodal training image is a liveness category or an attack category. The training objective of the multimodal liveness detection model may include a first objective. The first objective may be that the multiple predicted liveness classification results corresponding to the multiple modalities output by the multimodal liveness detection model are consistent with their corresponding real liveness classification results. Consistency can be defined as the difference between the two within a first preset range. During training, the first objective can constrain the multiple predicted liveness classification results output by the multimodal liveness detection model to approach their corresponding real liveness classification results, thereby improving the accuracy of the multimodal liveness detection model.

[0067] In some embodiments, the training objective of the multimodal liveness detection model may further include a second objective. The second objective may be that the multiple predicted liveness classification results output by the multimodal liveness detection model are consistent with the predicted multimodal liveness classification result obtained by fusing the multiple modalities. Consistency may be defined as the difference between the two within a second preset range. During training, the second objective may constrain the multiple predicted liveness classification results output by the multimodal liveness detection model to approach the fused predicted multimodal liveness classification result, thereby improving the accuracy of the multimodal liveness detection model.

[0068] In some embodiments, the training objective of the multimodal liveness detection model may further include a third objective. The third objective may include ensuring consistency among multiple predicted features corresponding to the multiple modalities output by the multimodal liveness detection model. Consistency among multiple predicted features may mean that the differences between the multiple predicted features are within a third preset range.

[0069] The training process of the multimodal liveness detection model is described below with reference to the accompanying diagram. Figure 4 A schematic diagram of the structure of a preset multimodal liveness detection model provided according to an embodiment of this specification is shown. Figure 4 As shown, the preset multimodal liveness detection model may include a preset multimodal feature encoding network, a preset multimodal liveness classification network, and a preset multimodal relationship constraint network.

[0070] The pre-defined multimodal feature encoding network can be a ResNet (residual network) or a classic network such as DenseNet, a convolutional neural network (CNN). The pre-defined multimodal feature encoding network can be configured to extract features from multimodal training images. For example, computing device 600 inputs training images of multiple modalities into the pre-defined multimodal feature encoding network, enabling the network to extract features from multiple modalities and extract multimodal fusion features based on the training images, thereby obtaining predicted features for multiple modalities and predicted multimodal fusion features. For example, the multimodal training images may include RGB (red-green-blue) modal training images, NIR (Near Infrared) modal training images, and Depth modal training images. The computing device 600 can obtain predictive features for the RGB modality, NIR modality, and Depth modality based on training images of the RGB modality, NIR modality, and Depth modality, as well as predictive multimodal fusion features based on RGB modality + NIR modality + Depth modality. Specifically, when extracting predictive multimodal fusion features using a pre-defined multimodal feature encoding network, the computing device 600 can either fuse the extracted predictive features from multiple modalities to obtain the predictive multimodal fusion features, or it can fuse the training images of multiple modalities and extract the predictive multimodal fusion features based on the fused image.

[0071] The pre-defined multimodal liveness classification network can be a multilayer perceptron (MLP), also known as an artificial neural network (ANN), which is configured to perform liveness detection based on the predicted features extracted by the pre-defined multimodal feature encoding network. For example, the computing device 600 inputs the predicted features of multiple modalities and the predicted multimodal fusion features into the pre-defined multimodal liveness classification network, so that the pre-defined multimodal liveness classification network performs liveness detection based on the predicted features of multiple modalities to obtain multiple predicted liveness classification results corresponding to multiple modalities, and performs liveness detection based on the predicted multimodal fusion features to obtain the predicted multimodal liveness classification result. For example, computing device 600 performs liveness detection based on prediction features of RGB mode, prediction features of NIR mode, and prediction features of Depth mode, as well as prediction multimodal fusion features based on RGB mode + NIR mode + Depth mode, respectively. This can yield predicted liveness detection results for RGB mode, predicted liveness detection results for NIR mode, predicted liveness detection results for Depth mode, and multimodal predicted liveness detection results based on RGB mode + NIR mode + Depth mode.

[0072] The pre-defined multimodal relationship constraint network can be a multilayer perceptron (MLP), also known as an artificial neural network (ANN). It is configured to constrain the predicted features of multiple modalities extracted by the pre-defined multimodal feature encoding network, or to constrain the predicted features of multiple modalities extracted by the pre-defined multimodal feature encoding network and the predicted multimodal fusion features. For example, computing device 600 inputs the predicted features of multiple modalities into the pre-defined multimodal relationship constraint network so that the pre-defined multimodal relationship network outputs the pairwise similarity between the predicted features of multiple modalities. Alternatively, computing device 600 inputs the predicted features of multiple modalities and the predicted multimodal fusion features into the pre-defined multimodal relationship constraint network so that the pre-defined multimodal relationship network outputs the pairwise similarity between the predicted features of multiple modalities and the predicted multimodal fusion features.

[0073] After obtaining the outputs of the preset multimodal feature encoding network, the preset multimodal liveness classification network, and the preset multimodal relational constraint network, the computing device 600 can determine the first comprehensive loss based on the outputs of these networks, and then converge the preset multimodal feature encoding network, the preset multimodal liveness classification network, and the preset multimodal relational constraint network based on the first comprehensive loss to obtain the liveness detection model. The first comprehensive loss can be expressed as the following formula (1):

[0074] Loss_total1=Loss_cls1+a×Loss_pred+b×Loss_feat; (1)

[0075] In equation (1), Loss_total1 represents the first comprehensive loss; Loss_cls1 represents the multimodal liveness classification loss, representing the first objective of training the multimodal liveness detection model; Loss_pred represents the multimodal prediction consistency loss, representing the second objective of training the multimodal liveness detection model; and Loss_feat represents the multimodal feature consistency loss, representing the third objective of training the multimodal liveness detection model. a and b represent the weights. Where a = 0 or 1, and b = 0 or 1.

[0076] The multimodal liveness classification loss Loss_cls1 can be determined based on the differences between multiple predicted liveness classification results corresponding to multiple modalities output by the preset multimodal liveness classification network and their corresponding true liveness classification results, as well as the weighted sum of the differences between the predicted multimodal liveness classification results and their corresponding true liveness classification results. For example, the computing device 600 determines the sub-liveness classification loss corresponding to the RGB modality based on the differences between the predicted liveness classification results of the RGB modality and their corresponding true liveness classification results. Similarly, the computing device 600 can obtain the sub-liveness classification loss corresponding to the NIR modality and the sub-liveness classification loss corresponding to the Depth modality. In addition, the computing device 600 can also determine the sub-liveness classification loss corresponding to the multimodality based on the differences between the predicted multimodal liveness classification results and their corresponding true liveness classification results, and obtain the liveness classification loss Loss_cls1 based on the weighted sum of the sub-liveness classification losses corresponding to the RGB modality, the NIR modality, the Depth modality, and the multimodal sub-liveness classification losses. Loss_cls1 is designed to constrain the differences between the predicted liveness classification results of multiple modalities and their corresponding true liveness classification results, as well as the weighted sum of the differences between the predicted multimodal liveness classification results and their corresponding true liveness classification results within a first preset range. The computing device 600 performs backpropagation based on the Loss_cls1 obtained from the current training iteration to update the parameters of the preset multimodal feature encoding network and the preset multimodal liveness classification network until training is complete.

[0077] The multimodal prediction consistency loss (Loss_pred) can be determined based on the differences between multiple predicted liveness classification results from multiple modalities and the fused multimodal liveness classification result. For example, the computing device 600 determines the sub-prediction consistency loss corresponding to the RGB modality based on the differences between the predicted liveness classification result corresponding to the RGB modality and the predicted multimodal liveness classification result. Similarly, the computing device 600 can determine the sub-prediction consistency loss corresponding to the NIR modality and the sub-prediction consistency loss corresponding to the Depth modality. Then, the computing device 600 obtains the multimodal prediction consistency loss by weighted summing the sub-prediction consistency losses corresponding to the RGB modality, the NIR modality, and the Depth modality. The multimodal fusion liveness classification result has high accuracy. By constraining the differences between the predicted liveness classification results from multiple modalities and the multimodal liveness classification result within a second preset range based on the multimodal prediction consistency loss (Loss_pred), the computing device 600 can improve the accuracy of the multimodal liveness classification result. The computing device 600 performs backpropagation based on the Loss_pred obtained from the current training to update the parameters of the preset multimodal feature encoding network and the preset multimodal liveness classification network until the training ends.

[0078] The multimodal feature consistency loss (Loss_feat) can be determined based on the pairwise similarity between predicted features of multiple modalities, or based on the pairwise differences between predicted features of multiple modalities and multimodal predicted features. Taking the determination of the multimodal feature consistency loss (Loss_feat) based on the pairwise similarity between predicted features of multiple modalities as an example, the computing device 600 determines the similarity between the RGB-NIR modal and the NIR modal based on the similarity between the predicted features corresponding to the RGB modal and the predicted features corresponding to the NIR modal. Similarly, the computing device 600 can determine the similarity between the RGB-Depth modal and the NIR-Depth modal. Then, the computing device 600 can obtain the pairwise similarity between the predicted features of multiple modalities based on the weighted sum of the similarities between the RGB-NIR modal, the RGB-Depth modal, and the NIR-Depth modal. The multimodal feature consistency loss (Loss_feat) aims to constrain the differences between the predicted features of multiple modalities within a third preset range. During training, the computing device 600 performs backpropagation based on the Loss_feat obtained from the current training to update the parameters of the preset multimodal feature encoding network and the preset multimodal relation constraint network until training ends.

[0079] During the training process described above, the conditions for ending the training may be that the first comprehensive loss is less than the first loss value, or the number of training iterations reaches the preset number of training iterations, or the training accuracy of the multimodal liveness detection model reaches the preset accuracy, etc. This manual does not impose any restrictions on these conditions.

[0080] After training the multimodal liveness detection model, the computing device 600 can use the multimodal liveness detection model as a teacher model and the unimodal preset target liveness detection model as a student model to guide the training of the unimodal preset target liveness detection model. That is, the computing device 600 can use the multimodal liveness detection model to perform knowledge distillation on the preset target liveness detection model, thereby obtaining a trained target liveness detection model. The trained target liveness detection model can use unimodal target images as input, achieving liveness detection performance close to that of the multimodal liveness detection model.

[0081] The computing device 600 uses a multimodal liveness detection model as the teacher model. There are various ways to implement knowledge distillation of the preset target liveness detection model, such as completing the training in one go, or training in stages and multiple times, etc.

[0082] In some embodiments, the computing device 600 can perform progressive knowledge distillation (staged, multiple training iterations) on the target liveness detection model based on the multimodal liveness detection model. Progressive knowledge distillation can include pre-knowledge distillation and advanced knowledge distillation. Pre-knowledge distillation and advanced knowledge distillation will be described in turn below. The implementation process of progressive knowledge distillation will be described below in conjunction with the network structure of the preset target liveness detection model.

[0083] Figure 5 A schematic diagram of the network structure of a preset target liveness detection model provided according to an embodiment of this specification is shown. Figure 5 As shown, the preset target liveness detection model may include a preset single-modal feature encoding network and a preset single-modal liveness classification network.

[0084] The pre-defined unimodal feature encoding network can be a convolutional neural network such as ResNet or DenseNet, configured to extract features based on the training image of the target modality. The target modality is one of the multiple modalities used in the multimodal liveness detection model. The training image of the target modality is one of the training images of the multiple modalities used in the multimodal liveness detection model, such as the training image corresponding to the RGB modality, NIR modality, or Depth modality. During pre-knowledge distillation, the computing device 600 can input the training image of the target modality into the unimodal feature encoding network, enabling the unimodal feature encoding network to extract features based on the unimodal training image, obtaining the first predicted feature of the target modality.

[0085] The preset unimodal liveness classification network can be an MLP, which is configured to perform liveness detection based on the first predicted features of the target modality extracted by the preset unimodal feature encoding network. During pre-knowledge distillation, the computing device 600 inputs the first predicted features of the target modality into the preset unimodal liveness classification network, enabling the preset unimodal liveness classification network to perform liveness detection based on the first predicted features of the target modality and obtain the first predicted liveness classification result.

[0086] It should be noted that both the preset unimodal feature encoding network and the preset multimodal feature encoding network are convolutional neural networks such as ResNet or DenseNet, and both the preset unimodal liveness classification network and the preset multimodal liveness classification network are MLPs. The teacher model has a higher network structure complexity, thus exhibiting better liveness detection performance and generalization ability. The student model, on the other hand, has a simpler structure, with less parameter computation and lower computational resource consumption. The computing device 600 trains the student model under the guidance of the teacher model, enabling the lightweight student model with a simpler network structure to achieve liveness detection performance close to that of the teacher model. Therefore, when the trained student model is applied to the terminal device 200 for liveness detection, it can significantly save computational resources, improve liveness detection efficiency, and reduce the hardware cost of the image acquisition module.

[0087] After acquiring the outputs of the preset single-modal feature encoding network and the preset single-modal liveness classification network, the computing device 600 can determine the second comprehensive loss based on the outputs of the preset single-modal feature encoding network and the preset single-modal liveness classification network, and converge the preset single-modal feature encoding network and the preset single-modal liveness classification network based on the second comprehensive loss to obtain the intermediate target liveness detection model. The second comprehensive loss can characterize the training objective of the target liveness detection model in the pre-knowledge distillation stage. The second comprehensive loss can be expressed as the following formula (2):

[0088] Loss_total2=Loss_pred'+c×Loss_cls2+d×Loss_h+e×Loss_csi; (2)

[0089] In equation (2), Loss_total2 represents the second comprehensive loss; Loss_pred' represents the prediction fitting loss, which can characterize the first objective of the target liveness detection model during training; Loss_cls2 represents the first unimodal liveness classification loss, which can characterize the second objective of the target liveness detection model during training; Loss_h represents the mutual information loss, which can characterize the third objective of the target liveness detection model during training; and Loss_csi represents the conditional self-information loss, which can characterize the fourth objective of the target liveness detection model during training. c, d, and e are weights, respectively. Wherein, c = 0 or 1, d = 0 or 1, and e = 0 or 1. The training objective of the target liveness detection model during training may include the first objective. The training objective of the target liveness detection model during training may also include at least one of the second, third, and fourth objectives.

[0090] The first objective may include ensuring consistency between the first predicted liveness classification result output by the target liveness detection model and the predicted multimodal liveness classification result output by the multimodal liveness detection model. Consistency can be defined as the difference between the two being within a fourth preset range. Loss_pred' can be determined based on the difference between the first predicted liveness classification result output by the preset unimodal liveness classification network and the predicted multimodal liveness classification result output by the multimodal liveness detection model. Loss_pred' aims to constrain the first predicted liveness classification result output by the target liveness detection model to approach the predicted multimodal liveness classification result output by the multimodal liveness detection model. During training, the computing device 600 can perform backpropagation based on the Loss_pred' obtained in the current training iteration to update the parameters of the preset unimodal feature encoding network and the preset unimodal liveness classification network until training is complete.

[0091] The second objective can include ensuring that the first predicted liveness classification result output by the target liveness detection model is consistent with its corresponding true liveness classification result. Consistency can be defined as the difference between the two being within a fifth preset range. Loss_cls2 can be based on... The difference between the first predicted liveness classification result and its corresponding true liveness classification result is determined. Loss_cls aims to constrain the first predicted liveness classification result to approach its corresponding true liveness classification. During training, the computing device 600 can perform backpropagation based on the Loss_cls2 obtained in the current training to update the parameters of the preset unimodal feature encoding network and the preset unimodal liveness classification network until training ends.

[0092] The third objective can include maximizing the mutual information between the first predicted feature of the target modality output by the target liveness detection model and the predicted multimodal fusion feature output by the multimodal liveness detection model. Maximizing mutual information can mean that the mutual information approaches its maximum value. Loss_h can be determined based on the difference between the mutual information between the first predicted feature of the target modality and the predicted multimodal fusion feature and a first preset value. The first preset value can be the maximum value of the mutual information. It should be noted that the maximum value of the mutual information can be the maximum allowable value after normalization. Loss_h aims to constrain the difference between the mutual information between the first predicted feature of the target modality and the predicted multimodal fusion feature and the first preset value to be within a sixth preset range. This can also be understood as the constraint objective of Loss_h being to make the mutual information between the first predicted feature of the target modality and the predicted multimodal fusion feature approach the first preset value, such as approaching 1, thereby achieving the goal of maximizing mutual information. Loss_h enables the prediction of student model outputs as much as possible based on the teacher model's output, given the teacher model's output. This allows the trained target liveness detection model to achieve detection performance close to that of a multimodal liveness detection model. During training, the computing device 600 can perform backpropagation based on the Loss_h obtained from the current training iteration to update the parameters of the preset single-modal feature encoding network until training is complete.

[0093] The mutual information between the first predicted feature of the target modality and the predicted multimodal fusion feature can be determined as follows: For example, computing device 600 denotes the predicted multimodal fusion feature output by the teacher model as X, and the first predicted feature of the target modality output by the student model as Y, and determines the mutual information between them based on the formula I(X,Y)=H(X)+H(Y)-H(X,Y). In the formula, I(X,Y) represents the correlation between the teacher model and the student model; H(X) represents the entropy of X output by the teacher model; H(Y) represents the entropy of Y output by the student model; and H(X,Y) represents the joint entropy of X output by the teacher model and Y output by the student model. H(X)=-P dos log2P dos +(-P live log2P live P dos The probability of attack type can be determined based on the proportion of the number of attack categories in the predicted multimodal liveness detection results. Similarly, computing device 600 can obtain H(Y). It is worth noting that when computing device 600 calculates H(X,Y), it is based on the joint probability of the teacher model and the student model. That is, H(X,Y) = -P dos log2P dos +(-P livelog2P live )+(-P live log2P dos )+(-P dos log2P live ).

[0094] The fourth objective can include minimizing the conditional self-information of the first predicted features of the target modality output by the multimodal liveness detection model, given a predetermined first predicted feature of the target modality output by the predefined single-modal feature encoding network. Minimizing the conditional self-information can mean that the conditional self-information approaches its minimum value. Loss_csi can be based on determining a predetermined... Given the first predicted feature of the target modality output by the unimodal feature encoding network, the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model is determined by the difference between the conditional self-information and a second preset value. The second preset value can be the minimum value of the conditional self-information. It should be noted that the minimum value of the self-information can be the minimum allowable value after normalization. Loss_csi aims to constrain, given the first predicted feature of the target modality output by the unimodal feature encoding network, the difference between the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model and the second preset value to be within a seventh preset range. Alternatively, the constraint objective of Loss_csi can be understood as ensuring that, given the first predicted feature of the target modality output by the unimodal liveness classification network, the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model approaches the second preset value, such as 0, thereby minimizing the conditional self-information. A larger conditional self-information means a lower probability of inferring the teacher model's output based on the student model's output, given the student model's output. Loss_csi enables the prediction of the teacher model's output with minimal bias, given the student model's output, thus minimizing the dependence of the trained target liveness detection model on the teacher model. Based on Loss_h and Loss_csi, the goal of transferring prior knowledge learned by the teacher model to the student model can be achieved, while simultaneously ensuring the student model's low dependence on the teacher model and its ability to independently produce accurate liveness detection results. During training, the computing device 600 can perform backpropagation based on the Loss_csi obtained from the current training iteration to update the parameters of the preset single-modal feature encoding network until training is complete.

[0095] Conditional self-information can be denoted as I'(X,Y)=log2(P(Y)). dos |X dos ))+log2(P(Y live |X live))+log2(P(Y dos |X live ))+log2(P(Y live |X dos P(Y) dos |X dos P(Y) represents the probability that an attack class appears in X among samples where Y is known to be an attack class. live |X live P(Y) represents the probability of a live sample appearing in X, given that Y is a live sample. dos |X live P(Y) represents the probability of a liveness category appearing in X among samples where Y is known to be an attack category. live |X dos P(Y) represents the probability of the attack class appearing in X among samples where Y is known to be a live sample. live |X dos For example, P(Y) live |X dos The following method can be used to determine this: For example, computing device 600 can determine P(Y) as the proportion of samples classified as attack category in X samples corresponding to Y, where Y is the liveness category. live |X dos ).

[0096] In formula (2) above, the third and fourth terms can be understood as constraining the correlation between the first predicted feature of the target modality output by the target liveness detection model and the predicted multimodal fusion features of multiple modalities output by the multimodal liveness detection model to meet preset requirements. These preset requirements may include: the mutual information between the first predicted feature of the target modality output by the target liveness detection model and the predicted multimodal fusion features output by the multimodal liveness detection model approaches a first preset value; and, given that the first predicted feature of the target modality output by the target liveness detection model is determined, the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model approaches a second preset value.

[0097] It should be noted that when determining the second comprehensive loss, the computing device 600 needs to include single-modality training images in the training images corresponding to the multiple modalities of the predicted multimodal liveness classification result. In other words, the training images corresponding to the first predicted liveness classification result should be included in the training images corresponding to the multiple modalities of the predicted multimodal liveness classification result.

[0098] During the training process of the aforementioned pre-knowledge distillation, the conditions for ending the training may be that the second comprehensive loss is less than the second loss value, or the number of training iterations reaches the preset number of training iterations, or the training accuracy of the preset target liveness detection model reaches the preset accuracy, etc. This specification does not impose any restrictions on these conditions.

[0099] At the end of the pre-knowledge distillation training, the resulting preset target liveness detection model is the trained intermediate target liveness detection model. The intermediate target liveness detection model includes an intermediate unimodal feature encoding network and an intermediate unimodal liveness classification network. The computing device 600 can perform advanced knowledge distillation on the preset target liveness detection model based on the multimodal liveness detection model. Advanced knowledge distillation aims to further improve the liveness detection performance of the student model.

[0100] The training objective of advanced knowledge distillation can include consistency of prediction distribution, and can also include consistency between the second predicted liveness classification result output by the target liveness detection model and its corresponding true liveness classification result. Consistency can mean that the difference between the two is within an eighth preset range, or that the second predicted liveness classification result output by the target liveness detection model approaches its corresponding true liveness classification result.

[0101] The consistency of prediction distribution can be defined as the consistency between the distribution of predicted objects under the target category output by the target liveness detection model and the distribution of predicted objects under the target category output by the multimodal liveness detection model. The target category can be either an attack category or a liveness category. The consistency of prediction distribution can include: the consistency between the distribution of predicted objects under the liveness category output by the target liveness detection model and the distribution of predicted objects under the liveness category output by the multimodal liveness detection model; and the consistency between the distribution of predicted objects under the attack category output by the target liveness detection model and the distribution of predicted objects under the attack category output by the multimodal liveness detection model. Consistency can be defined as the difference between the two being within a ninth preset range. During training, the consistent prediction distribution can constrain the distribution pattern of predicted objects under the liveness category output by the target liveness detection model to approximate the distribution pattern of predicted objects under the liveness category output by the multimodal liveness detection model.

[0102] The prediction objects include the predicted features output by the model and / or the predicted liveness classification results. For a target liveness detection model, the prediction objects can be the second predicted features and / or the second predicted liveness classification results output by the target liveness detection model. For a multimodal liveness detection model, the prediction objects can be the predicted multimodal fusion features and / or the predicted multimodal liveness classification results output by the multimodal liveness detection model.

[0103] In some embodiments, the consistency of the prediction distribution may include: the distribution of the second predicted feature under the liveness category output by the target liveness detection model is consistent with the distribution of the predicted multimodal fusion feature under the liveness category output by the multimodal liveness detection model. This consistency may be such that the difference between the two is within a tenth preset range, constraining the distribution pattern of the second predicted feature under the liveness category output by the target liveness detection model to approach the distribution pattern of the predicted multimodal fusion feature under the liveness category output by the multimodal liveness detection model. Furthermore, the distribution of the second predicted feature under the attack category output by the target liveness detection model is consistent with the distribution of the predicted multimodal fusion feature under the attack category output by the multimodal liveness detection model. This consistency may be such that the difference between the two is within an eleventh preset range. This consistency constrains the distribution pattern of the second predicted feature under the attack category output by the target liveness detection model to approach the distribution pattern of the predicted multimodal fusion feature under the attack category output by the multimodal liveness detection model.

[0104] In some embodiments, the consistency of the predicted distribution may include: the distribution of the second predicted liveness classification result under the liveness category output by the target liveness detection model is consistent with the distribution of the predicted multimodal liveness classification result under the liveness category output by the multimodal liveness detection model. This consistency may be such that the difference between the two is within a twelfth preset range, constraining the distribution pattern of the second predicted liveness classification result under the liveness category output by the target liveness detection model to approach the distribution pattern of the predicted multimodal liveness classification result under the liveness category output by the multimodal liveness detection model. Furthermore, the distribution of the second predicted liveness classification result under the attack category output by the target liveness detection model is consistent with the distribution of the predicted multimodal liveness classification result under the attack category output by the multimodal liveness detection model. This consistency may be such that the difference between the two is within a thirteenth preset range, constraining the distribution pattern of the second predicted liveness classification result under the attack category output by the target liveness detection model to approach the distribution pattern of the predicted multimodal liveness classification result under the attack category output by the multimodal liveness detection model.

[0105] In some embodiments, the consistency of prediction distribution may include: the distribution of the second predicted feature under the liveness category output by the target liveness detection model is consistent with the distribution of the predicted multimodal fusion feature under the liveness category output by the multimodal liveness detection model, and the distribution of the second predicted liveness classification result under the liveness category output by the target liveness detection model is consistent with the distribution of the predicted multimodal liveness classification result under the liveness category output by the multimodal liveness detection model; and the distribution of the second predicted feature under the attack category output by the target liveness detection model is consistent with the distribution of the predicted multimodal fusion feature under the attack category output by the multimodal liveness detection model, and the distribution of the second predicted liveness classification result under the attack category output by the target liveness detection model is consistent with the distribution of the predicted multimodal liveness classification result under the attack category output by the multimodal liveness detection model. The consistency here can be referred to the relevant content in the above embodiments.

[0106] The distribution of predicted objects can be represented using class centers. The computing device 600 can constrain the training process of advanced knowledge distillation using the class centers of the liveness category and the attack category, respectively.

[0107] Specifically, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model, which may include at least one of the following: the class centers of the predicted objects under the liveness category output by the target liveness detection model are consistent with the class centers of the predicted objects under the liveness category output by the multimodal liveness detection model; and the distance between the predicted objects under the liveness category output by the target liveness detection model and the class center of their corresponding liveness category is consistent with the distance between the predicted objects under the liveness category output by the multimodal liveness detection model and the class center of their corresponding liveness category. Consistency can be defined as the difference between the two being within a preset range.

[0108] In other words, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model, which can include the following implementation methods:

[0109] In some embodiments, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model. This may include: the class centers of the predicted objects under the liveness category output by the target liveness detection model are consistent with the class centers of the predicted objects under the liveness category output by the multimodal liveness detection model. Consistency may mean that the difference between the two is within a fourteenth preset range, thus constraining the class centers of the predicted objects under the liveness category output by the target liveness detection model to approach the class centers of the predicted objects under the liveness category output by the multimodal liveness detection model.

[0110] In some embodiments, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model. This may include: the distance between the predicted objects under the liveness category output by the target liveness detection model and the class center of their corresponding liveness category is consistent with the distance between the predicted objects under the liveness category output by the multimodal liveness detection model and the class center of their corresponding liveness category. Consistency may mean that the difference between the two is within a fifteenth preset range, thus constraining the distance between the predicted objects under the liveness category output by the target liveness detection model and the class center of their corresponding liveness category to approach the distance between the predicted objects under the liveness category output by the multimodal liveness detection model and the class center of their corresponding liveness category.

[0111] In some embodiments, the distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model. This may include: the class centers of the predicted objects under the liveness category output by the target liveness detection model are consistent with the class centers of the predicted objects under the liveness category output by the multimodal liveness detection model, and the distance between the predicted objects under the liveness category output by the target liveness detection model and the class center of their corresponding liveness category is consistent with the distance between the predicted objects under the liveness category output by the multimodal liveness detection model and the class center of their corresponding liveness category. For details on this consistency, please refer to the relevant descriptions in the above embodiments.

[0112] Furthermore, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model, and may further include at least one of the following: the class centers of the predicted objects under the attack category output by the target liveness detection model are consistent with the class centers of the predicted objects under the attack category output by the multimodal liveness detection model; and the distance between the predicted objects under the attack category output by the target liveness detection model and the class center of their corresponding attack category is consistent with the distance between the predicted objects under the attack category output by the multimodal liveness detection model and the class center of their corresponding attack category. Consistency can be defined as the difference between the two being within a preset range.

[0113] In other words, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model. This can also include the following implementation methods:

[0114] In some embodiments, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model. This may further include: the class centers of the predicted objects under the attack category output by the target liveness detection model are consistent with the class centers of the predicted objects under the attack category output by the multimodal liveness detection model. Consistency can be achieved by the difference between the two falling within a sixteenth preset range, thereby constraining the class centers of the predicted objects under the attack category output by the target liveness detection model to approach the class centers of the predicted objects under the attack category output by the multimodal liveness detection model.

[0115] In some embodiments, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model. This may further include: the distance between the predicted objects under the attack category output by the target liveness detection model and the class center of their corresponding attack category is consistent with the distance between the predicted objects under the attack category output by the multimodal liveness detection model and the class center of their corresponding attack category. Consistency can be achieved by ensuring that the difference between the two is within a seventeenth preset range, thereby constraining the distance between the predicted objects under the attack category output by the target liveness detection model and the class center of their corresponding attack category to approach the distance between the predicted objects under the attack category output by the multimodal liveness detection model and the class center of their corresponding attack category.

[0116] In some embodiments, the distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model. This may further include: the class centers of the predicted objects under the attack category output by the target liveness detection model are consistent with the class centers of the predicted objects under the attack category output by the multimodal liveness detection model, and the distance between the predicted objects under the attack category output by the target liveness detection model and the class center of their corresponding attack category is consistent with the distance between the predicted objects under the attack category output by the multimodal liveness detection model and the class center of their corresponding attack category. For details on consistency, please refer to the relevant descriptions in the above embodiments.

[0117] During advanced knowledge distillation, the computing device 600 can input the training image of the target modality into an intermediate single-modality feature encoding network to extract features from the training image of the target modality, thereby obtaining the second predicted feature of the target modality. The second predicted feature of the target modality is then input into an intermediate single-modality liveness classification network to perform liveness detection based on the second predicted feature of the target modality, thereby obtaining the second predicted liveness classification result.

[0118] After obtaining the second predicted feature and the second predicted liveness classification result of the target modality, the computing device 600 can determine the third comprehensive loss based on the second predicted feature and / or the second predicted liveness classification result. Taking the determination of the third comprehensive loss by the computing device 600 based on the second predicted feature and the second predicted liveness classification result of the target modality as an example, the third comprehensive loss can be expressed as the following formula (3):

[0119] Loss_total3=Loss_cls3+x×Loss_c1+y×Loss_c2; (3)

[0120] In equation (3), Loss_total3 represents the third comprehensive loss; Loss_cls3 represents the second unimodal liveness classification loss, which can represent the fifth objective of the target liveness detection model during training; Loss_c1 represents the class center relation constraint loss, which can represent the sixth objective of the target liveness detection model during training; and Loss c2 represents the fine-grained class center relation constraint loss, which can represent the seventh objective of the target liveness detection model during training. x and y are the weights, respectively. Where x = 0 or 1, and y = 0 or 1. The training objective of the target liveness detection model during training may include the fifth objective. The training objective of the target liveness detection model during training may also include the sixth and / or seventh objectives.

[0121] The fifth objective may include ensuring consistency between the second predicted liveness classification result and its corresponding true liveness classification result. Consistency can be defined as the difference between the two falling within a third preset range. Loss_cls3 can be determined based on the difference between the second predicted liveness classification result and its corresponding true liveness classification result. Loss_cls aims to constrain the second predicted liveness classification result to approximate its corresponding true liveness classification result. It should be noted that when determining Loss_h, the computing device 600 needs to include single-modality training images among the training images corresponding to the multiple modalities of the predicted multimodal liveness classification result. During training, the computing device 600 can perform backpropagation based on the Loss_cls3 obtained in the current training iteration to update the parameters of the intermediate single-modality feature encoding network and the intermediate single-modality liveness classification network until training is complete.

[0122] The sixth objective may include the consistency between the class centers of predicted objects in the liveness category output by the target liveness detection model and the class centers of predicted objects in the liveness category output by the multimodal liveness detection model, and the consistency between the class centers of predicted objects in the attack category output by the target liveness detection model and the class centers of predicted objects in the attack category output by the multimodal liveness detection model. Consistency can be defined as the difference between the two being within a preset range. Loss_c1 can be determined based on at least one of the differences between the class centers of predicted objects in the liveness category output by the target liveness detection model and the multimodal liveness detection model, and the differences between the class centers of predicted objects in the attack category output by the target liveness detection model and the multimodal liveness detection model. For example, computing device 600 can determine Loss_c1 based on the difference between the class center of the predicted object under the liveness category output by the target liveness detection model and the class center of the predicted object under the liveness category output by the multimodal liveness detection model. Alternatively, it can determine Loss_c1 based on the difference between the class center of the predicted object under the attack category output by the target liveness detection model and the class center of the predicted object under the attack category output by the multimodal liveness detection model. Furthermore, it can determine Loss_c1 based on the weighted sum of the differences between the class centers of the predicted object under the liveness category output by the target liveness detection model and the class centers of the predicted object under the attack category output by the target liveness detection model and the class centers of the predicted object under the attack category output by the multimodal liveness detection model. Loss_c1 aims to constrain the class centers of predicted objects under the liveness category output by the target liveness detection model to be close to the class centers of predicted objects under the liveness category output by the multimodal liveness detection model, and to constrain the class centers of predicted objects under the attack category output by the target liveness detection model to be close to the class centers of predicted objects under the attack category output by the multimodal liveness detection model.

[0123] Taking the class center corresponding to the liveness category as an example, the computing device 600 can determine the class center of the predicted object under the liveness category output by the target liveness detection model in various ways. For example, the computing device 600 can determine the class center of the predicted object under the liveness category based on the average value of the predicted object under the liveness category output by the target liveness detection model. Similarly, the computing device 600 can also obtain the class center of the predicted object under the attack category output by the target liveness detection model, the class center of the predicted object under the liveness category output by the multimodal liveness detection model, and the class center of the predicted object under the attack category output by the multimodal liveness detection model. Loss_c1 can constrain the class center of the predicted object output by the target liveness detection model to be as consistent as possible with the class center of the multimodal liveness detection model, thereby enabling the target liveness detection model to fully learn the prior knowledge of the multimodal liveness detection model and improve the liveness detection accuracy of the target liveness detection model.

[0124] The distance between the class centers of predicted objects under the liveness category output by the target liveness detection model characterizes the compactness of the predicted objects under the liveness category output by the target liveness detection model. A higher compactness indicates a more dispersed distribution of predicted objects under the liveness category output by the target liveness detection model, leading to greater difficulty in fitting the model. A well-trained multimodal liveness detection model already possesses high liveness detection performance, meaning it fits the training objects well. The computing device 600 constrains the class centers of predicted objects under the liveness category output by the target liveness detection model to be as consistent as possible with the class centers of predicted objects under the liveness category output by the multimodal liveness detection model. The same principle applies to attack categories. The class center constraint makes the distribution of features extracted by the student model in the feature space similar to that of the teacher model, making the distribution of features extracted by the student model more compact and thus more robust to noise, which is conducive to liveness classification.

[0125] The seventh objective may include the distance between the predicted object in the liveness category output by the target liveness detection model and the class center of its corresponding liveness category, and the distance between the predicted object in the liveness category output by the multimodal liveness detection model and the class center of its corresponding liveness category. Consistency can be defined as the difference between the two being within a preset range. Loss_c2 can be determined based on at least one of the following: the difference between the distance between the predicted object in the liveness category output by the target liveness detection model and the class center of its corresponding liveness category, and the difference between the distance between the predicted object in the attack category output by the target liveness detection model and the class center of its corresponding attack category, and the difference between the distance between the predicted object in the attack category output by the target liveness detection model and the class center of its corresponding attack category, and the difference between the distance between the predicted object in the attack category output by the multimodal liveness detection model and the class center of its corresponding attack category. For example, computing device 600 can determine Loss_c2 based on the difference between the distance between the predicted object under the liveness category output by the target liveness detection model and the class center of its corresponding liveness category, and the distance between the predicted object under the liveness category output by the multimodal liveness detection model and the class center of its corresponding liveness category. Alternatively, it can determine Loss_c2 based on the difference between the distance between the predicted object under the attack category output by the target liveness detection model and the class center of its corresponding attack category, and the distance between the predicted object under the attack category output by the multimodal liveness detection model and the class center of its corresponding attack category. It can also determine Loss_c2 based on the weighted sum of the differences between the distance between the predicted object under the liveness category output by the target liveness detection model and the class center of its corresponding liveness category, and the distance between the predicted object under the attack category output by the target liveness detection model and the class center of its corresponding attack category, and the distance between the predicted object under the attack category output by the multimodal liveness detection model and the class center of its corresponding attack category. Loss_c2 aims to further constrain the class centers of the predicted objects output by the target liveness detection model to be as consistent as possible with the class centers of the multimodal liveness detection model. Alternatively, it can be understood as constraining the class centers of the predicted objects output by the target liveness detection model to approach the class centers of the multimodal liveness detection model. This allows the target liveness detection model to learn more fully from the prior knowledge of the multimodal liveness detection model, thereby further improving the liveness detection accuracy of the target liveness detection model.

[0126] When determining the distance between the predicted object under the liveness category output by the target liveness detection model and the class center of its corresponding liveness category, the computing device 600 can determine the distance between the predicted object under the liveness category output by the target liveness detection model and the class center of its corresponding liveness category based on the Euclidean distance, L2 distance, etc.

[0127] The distance between the predicted object in the liveness category output by the target liveness detection model and the class center of its corresponding liveness category represents the degree to which the predicted object deviates from the class center of its corresponding liveness category. A larger deviation indicates a more dispersed distribution of the predicted objects in the liveness category output by the target liveness detection model, leading to greater difficulty in fitting the model. A well-trained multimodal liveness detection model already possesses high liveness detection performance, meaning it fits the training objects well. The computing device 600, by constraining the degree to which the predicted object in the liveness category output by the target liveness detection model deviates from the class center of its corresponding liveness category as closely as possible to the degree to which the predicted object in the liveness category output by the multimodal liveness detection model deviates from the class center of its corresponding liveness category, further constrains the distribution of the predicted objects in the liveness category output by the target liveness detection model to be closer to the distribution of the predicted objects in the liveness category output by the multimodal liveness detection model. Referring to the above introduction to Loss_c2 for liveness categories, the role of Loss_c2 for attack categories can be understood, and will not be elaborated upon here.

[0128] During training, the computing device 600 can perform backpropagation based on the Loss_c1 obtained from the current training to update the parameters of the intermediate single-modal feature encoding network and / or the intermediate single-modal liveness classification network until the third comprehensive loss is less than the third loss value, or the number of training iterations reaches the preset number of training iterations, or the training accuracy of the preset target liveness detection model reaches the preset accuracy, etc., and then training ends.

[0129] Fine-grained class center constraints can further improve the consistency between the distribution of features extracted by the student model and the distribution of features extracted by the teacher model in the feature space, making the distribution of features extracted by the student model more compact in the feature space, thereby further improving robustness to noise and further improving the accuracy of liveness classification.

[0130] The intermediate single-modal feature encoding network and intermediate single-modal liveness classification network obtained at the end of the advanced knowledge distillation training are the trained target liveness detection model.

[0131] In some embodiments, a mapping network can be added during the advanced knowledge distillation process. This mapping network can be a fully connected layer configured to perform feature mapping on the second predicted features output by the intermediate unimodal feature encoding network, obtaining remapped second predicted features. Feature mapping can include operations such as rotation, scaling, and translation. Correspondingly, the prediction object can also include the remapped second predicted features. The computing device 600 can then determine a third comprehensive loss based on the remapped second predicted features. The feature mapping can further improve the performance and accuracy of the target liveness detection model. The mapping module ensures that the training results of the pre-knowledge distillation are not significantly modified, but still improves the performance and accuracy of liveness detection.

[0132] The above content describes the implementation process of progressive knowledge distillation. In some embodiments, the computing device 600 can also perform one-time knowledge distillation on the preset target liveness detection model, that is, perform joint knowledge distillation based on the second comprehensive loss and the third comprehensive loss. For example, the computing device 600 can determine the liveness classification loss, the class center relation constraint loss, and the fine-grained class center relation constraint loss based on the first predicted feature and / or the first predicted liveness classification result. That is to say, the computing device 600 can replace the second predicted feature and the second predicted liveness classification result in the above formula (3) with the first predicted feature and the first predicted liveness classification result to determine the third comprehensive loss, and jointly train based on the third comprehensive loss and the second comprehensive loss to obtain the trained target liveness detection model.

[0133] After obtaining the trained target liveness detection model, it can be output to the terminal device 200. This allows the terminal device 200 to perform liveness detection based on the acquired target image and obtain the target liveness detection result. For example, the terminal device 200 inputs the target image into the target liveness detection model, causing the model to output either a liveness probability P1 or an attack probability P2. The liveness probability P1 represents the probability that the target image is a live object. The attack probability P2 represents the probability that the target image is an attack object. The terminal device 200 can determine the target liveness detection result based on either the liveness probability P1 or the attack probability P2. For example, if the liveness probability P1 is greater than a set threshold T1, the target image is identified as a live object; if the liveness probability P1 is less than the set threshold T1, the target image is identified as an attack object. Similarly, if the attack probability P2 is greater than a set threshold T2, the target image is identified as an attack object; if the attack probability P2 is less than the set threshold T2, the target image is identified as a live object.

[0134] Continue reading Figure 3 After step S140, the method P100 may further include step S160.

[0135] S160: Output the target liveness detection result.

[0136] The computing device 600 can output the liveness detection result in several ways. For example, the computing device 600 can visualize the liveness detection result. The visualization can take many forms, such as displaying the result on a screen, or issuing a prompt message about the liveness detection result using sound and light, etc.

[0137] In summary, the liveness detection method P100 and system 001 provided in this specification acquire a single-modal target image of the target user and perform multimodal liveness detection on the target image based on the single-modal image using a target liveness detection model, thereby outputting the obtained target liveness detection result. In this scheme, the single-modal liveness detection model is obtained by knowledge distillation using a multimodal liveness detection model to obtain the target liveness detection model. The target liveness detection model can characterize the mapping relationship between the single-modal image and the multimodal liveness detection result obtained based on the multimodal image, enabling the single-modal liveness detection model to possess the liveness detection performance of the multimodal liveness detection model, thus achieving an effect close to that of multimodal liveness detection when performing liveness detection based on single-modal images. Furthermore, since this scheme performs liveness detection based on single-modal images, it can save computing power, improve liveness detection efficiency, and reduce the hardware cost of the image acquisition module. Meanwhile, the target liveness detection model is a lightweight model based on single-modal image detection, which has low requirements for computing power. It can be deployed on terminal devices or remote servers. When deployed on terminal devices, image transmission can be avoided, further saving computing time, while avoiding the risk of privacy leakage during data transmission.

[0138] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing liveness detection. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the liveness detection method P100 described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the liveness detection method P100 described herein. The program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0139] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0140] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0141] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0142] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0143] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0144] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting liveness, comprising: Acquire a target image of the target user, wherein the target image is a single-modal image of the target user captured under the target modality; Multimodal liveness detection is performed on the target image based on a target liveness detection model to obtain target liveness detection results. The target liveness detection model is a single-modal liveness detection model obtained by knowledge distillation based on the training image of the target modality and the multimodal liveness detection model. The multimodal liveness detection model is a model for liveness detection based on images of multiple modalities. The knowledge distillation includes pre-knowledge distillation, which includes: inputting the training image of the target modality into a preset target liveness detection model to obtain a first predicted feature and a first predicted liveness classification result of the target modality; and inputting training images of multiple modalities, including the training image of the target modality, into the multimodal liveness detection model. A liveness detection model is used to obtain predicted multimodal fusion features and predicted multimodal liveness classification results. Based on the difference between the first predicted liveness classification result and the predicted multimodal liveness classification result, the mutual information between the first predicted feature and the predicted multimodal fusion feature, and the conditional self-information of the first predicted feature of the target modality output by the multimodal liveness detection model when the target liveness detection model outputs the first predicted feature of the target modality, a preset target liveness detection model is trained to obtain the target liveness detection model. Wherein, the mutual information approaches a first preset value, and the conditional self-information approaches a second preset value. Output the target liveness detection results.

2. The method according to claim 1, wherein, The target modality is one of the multiple modalities corresponding to the multimodal liveness detection model.

3. The method according to claim 1, wherein, The training objective of the pre-knowledge distillation also includes: training the preset target liveness detection model based on the difference between the first predicted liveness classification result and its corresponding real liveness classification result.

4. The method according to claim 1, wherein, The mutual information quantity approaching the first preset value indicates that the mutual information quantity approaches the maximum value, representing the maximization of mutual information quantity; and the conditional self-information quantity approaching the second preset value indicates that the conditional self-information quantity is minimized.

5. The method according to claim 1, wherein, The knowledge distillation also includes: Based on the multimodal liveness detection model, an advanced knowledge distillation is performed on the target liveness detection model. The training objective of the advanced knowledge distillation includes prediction distribution consistency, which includes: The distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model; and The distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model.

6. The method according to claim 5, wherein, The distribution of predicted objects under the liveness category output by the target liveness detection model is consistent with the distribution of predicted objects under the liveness category output by the multimodal liveness detection model, including at least one of the following: The class center of the predicted object under the liveness category output by the target liveness detection model is consistent with the class center of the predicted object under the liveness category output by the multimodal liveness detection model; and The distance between the predicted object under the liveness category output by the target liveness detection model and the class center of its corresponding liveness category is consistent with the distance between the predicted object under the liveness category output by the multimodal liveness detection model and the class center of its corresponding liveness category.

7. The method according to claim 5, wherein, The distribution of predicted objects under the attack category output by the target liveness detection model is consistent with the distribution of predicted objects under the attack category output by the multimodal liveness detection model, including at least one of the following: The class center of the predicted object under the attack category output by the target liveness detection model is consistent with the class center of the predicted object under the attack category output by the multimodal liveness detection model; and The distance between the predicted object under the attack category output by the target liveness detection model and the class center of its corresponding attack category is consistent with the distance between the predicted object under the attack category output by the multimodal liveness detection model and the class center of its corresponding attack category.

8. The method according to claim 5, wherein, The prediction objects include the predicted features output by the model and / or the predicted liveness classification results.

9. The method according to claim 5, wherein, The training objectives of the advanced knowledge distillation also include: The second predicted liveness classification result output by the target liveness detection model is consistent with its corresponding true liveness classification result.

10. The method according to claim 1, wherein, The training objective of the multimodal liveness detection model includes: the predicted liveness classification results of multiple modalities output by the multimodal liveness detection model are consistent with their corresponding true liveness classification results.

11. The method according to claim 10, wherein, The training objective of the multimodal liveness detection model also includes at least one of the following: The multimodal liveness detection model outputs multiple predicted liveness classification results for multiple modalities that are consistent with the fused predicted multimodal liveness classification results; and The predicted features of multiple modalities output by the multimodal liveness detection model are consistent.

12. A liveness detection system, comprising: At least one storage medium storing at least one instruction set for performing liveness detection; as well as At least one processor is communicatively connected to the at least one storage medium. When the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the method according to any one of claims 1-11 according to the instructions of the at least one instruction set.

Images