Method, system and device for detecting living body

Abstract

The training method, the live body detection method and the system of the live body detection model provided in the specification obtain multi-modal training data and single-modal training data corresponding to a main mode. The multi-modal training data can include training data of multiple modes, and the multiple modes include a main mode and at least one auxiliary mode. Then, based on the multi-modal training data, a preset first live body detection model corresponding to each auxiliary mode of the at least one auxiliary mode is trained to obtain an auxiliary live body detection model corresponding to each auxiliary mode. Furthermore, based on the multi-modal training data, the single-modal training data and the auxiliary live body detection model, a preset second live body detection model corresponding to the main mode is subjected to multi-modal knowledge distillation to obtain a target live body detection model. This scheme can improve the training precision of the target live body detection model corresponding to the main mode, and further improve the accuracy of live body detection.

Description

Technical Field

[0001] This specification relates to the field of image recognition, and in particular to a training method for a liveness detection model, a liveness detection method, and a system. Background Technology

[0002] Compared to traditional identity verification methods such as passwords and verification codes, facial recognition is more efficient and convenient. Therefore, it is widely used in scenarios such as finance, transportation, and identity verification (verifying a user's true identity). However, facial recognition also faces security issues such as liveness detection attacks. To detect various types of liveness attacks and ensure the security of facial recognition systems, liveness detection has become a crucial component.

[0003] Currently, using multimodal data can improve the accuracy of liveness detection. However, in some application scenarios, only single-modal data is available. Liveness detection models trained with single-modal data have lower accuracy, thus resulting in lower overall liveness detection accuracy.

[0004] Therefore, there is a need to provide a training method, a liveness detection method, and a system for a liveness detection model with higher accuracy. Summary of the Invention

[0005] This specification provides a training method, a liveness detection method, and a system for a more accurate liveness detection model.

[0006] In a first aspect, this specification provides a training method for a liveness detection model, comprising: obtaining multimodal training data and single-modal training data corresponding to a primary modality, wherein the multimodal training data includes training data for multiple modalities, the multiple modalities including the primary modality and at least one auxiliary modality; training a preset first liveness detection model corresponding to each of the at least one auxiliary modality based on the multimodal training data to obtain an auxiliary liveness detection model corresponding to each auxiliary modality; and performing multimodal knowledge distillation on a preset second liveness detection model corresponding to the primary modality based on the multimodal training data, the single-modal training data, and the auxiliary liveness detection model to obtain a target liveness detection model.

[0007] In some embodiments, the plurality of modalities includes at least two modalities selected from color images, depth images, infrared images, or thermal imaging images, the primary modality includes one of the plurality of modalities, and the at least one auxiliary modality includes at least one modality other than the primary modality among the plurality of modalities.

[0008] In some embodiments, training a preset first liveness detection model corresponding to each of the at least one auxiliary modalities to obtain an auxiliary liveness detection model corresponding to each auxiliary modality includes: obtaining a preset first liveness detection model corresponding to each auxiliary modality; selecting current modality training data corresponding to each auxiliary modality from the multimodal training data; and training the corresponding preset first liveness detection model based on the current modality training data to obtain an auxiliary liveness detection model corresponding to each auxiliary modality.

[0009] In some embodiments, the multimodal training data includes at least one first user sample as a first user image sample in each of the plurality of modalities; and the step of performing multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality to obtain a target liveness detection model includes: using the auxiliary liveness detection model to extract features from the first user image sample of the corresponding modality to obtain auxiliary modality user features in each auxiliary modality, and performing multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality based on the auxiliary modality user features, the single-modality training data, and the multimodal training data to obtain the target liveness detection model.

[0010] In some embodiments, the auxiliary liveness detection model includes a feature extraction network, the feature extraction network including multiple network layers; and the step of using the auxiliary liveness detection model to extract features from the first user image samples of the corresponding modality to obtain auxiliary modality user features corresponding to each auxiliary modality includes: selecting candidate first user image samples corresponding to the auxiliary liveness detection model from the first user image samples, inputting the candidate first user image samples into the feature extraction network to obtain a user feature set corresponding to the candidate first user image samples, and selecting user features output by at least one network layer from the multiple network layers in the user feature set to obtain auxiliary modality user features corresponding to each auxiliary modality.

[0011] In some embodiments, the unimodal data includes the second user image sample of the at least one second user sample under the main modality; and the multimodal knowledge distillation of the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model includes: using the preset second liveness detection model to extract features from the second user image sample to obtain the main modality user features corresponding to the main modality, and performing hybrid training on the preset second liveness detection model based on the main modality user features, the auxiliary modality user features, the unimodal training data and the multimodal training data to obtain the target liveness detection model.

[0012] In some embodiments, the step of performing hybrid training on the preset second liveness detection model to obtain the target liveness detection model includes: training the preset second liveness detection model based on the unimodal training data to obtain a first candidate liveness detection model; performing feature knowledge distillation on the first candidate liveness detection model based on the primary modality user features, the auxiliary modality user features, and the multimodal training data to obtain a second candidate liveness detection model; and using the second candidate liveness detection model as the preset second liveness detection model, and returning to execute the step of training the preset second liveness detection model based on the unimodal training data until the preset second liveness detection model converges to obtain the target liveness detection model.

[0013] In some embodiments, performing feature knowledge distillation on the first candidate liveness detection model to obtain a second candidate liveness detection model includes: selecting training data corresponding to the main modality from the multimodal training data to obtain main modality training data; using the first candidate liveness detection model to extract features from the main modality training data to obtain user liveness features; determining a feature distillation loss based on the user liveness features, the main modality user features, and the auxiliary modality user features; and converging the first candidate liveness detection model based on the feature distillation loss to obtain the second candidate liveness detection model.

[0014] In some embodiments, the first candidate liveness detection model includes a primary modality feature extraction network and a feature generation network corresponding to each auxiliary modality; and the step of using the first candidate liveness detection model to extract features from the primary modality training data to obtain user liveness features includes: using the primary modality feature extraction network to extract features from the primary modality training data to obtain a first modality user feature corresponding to the primary modality; based on the first modality user feature and the multimodal training data, using the feature generation network to generate a second modality user feature corresponding to each auxiliary modality; and using the first modality user feature and the second modality user feature as the user liveness features.

[0015] In some embodiments, during the training process constrained by the feature distillation loss, the primary modality user features and the auxiliary modality user features are respectively aligned with the corresponding modality features in the user liveness features. The feature distillation loss includes at least one of feature distance loss, instance relation loss, and feature space loss. During the training process constrained by the feature distance loss, the feature distance between modal user features of the same modality is less than a preset first distance threshold, and the feature distance between modal user features of different modalities is greater than a preset second distance threshold. During the training process constrained by the instance relation loss, the relation features corresponding to instances under the same modality and the relation features corresponding to instances under different modalities are also constrained. Furthermore, during the training process constrained by the feature space loss, the similarity between feature spaces under the same modality is greater than a preset first similarity threshold, and the similarity between feature spaces under different modalities is less than a preset second similarity threshold.

[0016] In some embodiments, the step of performing hybrid training on the preset second liveness detection model to obtain the target liveness detection model includes: performing feature knowledge distillation on the preset second liveness detection model based on the primary modality user features, the auxiliary modality user features, and the multimodal training data to obtain a first candidate liveness detection model; training the first candidate liveness detection model based on the single modality training data to obtain a second candidate liveness detection model; and using the second candidate liveness detection model as the preset second liveness detection model, and returning to perform the step of performing feature knowledge distillation on the preset second liveness detection model until the preset second liveness detection model converges to obtain the target liveness detection model.

[0017] Secondly, this specification also provides a liveness detection method, comprising: obtaining a target user image in a single modality; extracting features from the target user image based on a target liveness detection model corresponding to the single modality to obtain target user liveness features, wherein the target user liveness features include user liveness features in multiple modalities, the multiple modalities including the single modality; and determining the liveness detection result of the target user based on the target user liveness features, and outputting the liveness detection result.

[0018] In some embodiments, the plurality of modalities includes at least two modalities selected from color images, depth images, infrared images, or thermal imaging images, and the single modality includes one of the plurality of modalities.

[0019] In some embodiments, the target liveness detection model includes a feature extraction network corresponding to the single modality and a feature generation network corresponding to the target modality, wherein the target modality includes modalities other than the single modality among the plurality of modalities; and the step of extracting features from the target user image to obtain target user liveness features includes: inputting the target user image into the feature extraction network to obtain a first user liveness feature corresponding to the single modality; generating a second user liveness feature corresponding to the target modality using the feature generation network based on the target user image and the first user liveness feature; and fusing the first user liveness feature and the second user liveness feature to obtain the target user liveness feature.

[0020] In some embodiments, determining the liveness detection result of the target user based on the liveness characteristics of the target user includes: determining the attack probability of the target user based on the liveness characteristics of the target user; and performing a target operation based on the attack probability, the target operation including a first operation or a second operation, wherein: the first operation includes determining that the attack probability is greater than a preset probability threshold and determining that the liveness detection result of the target user is an attacking user; and the second operation includes determining that the attack probability is less than the preset probability threshold and determining that the liveness detection result of the target user is a live user.

[0021] Thirdly, this specification also provides a training system for a liveness detection model, comprising: at least one storage medium storing at least one instruction set for training the liveness detection model; and at least one processor communicatively connected to the at least one storage medium, wherein, when the training system for the liveness detection model is running, the at least one processor reads the at least one instruction set and executes the training method for the liveness detection model described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0022] Fourthly, this specification also provides a liveness detection system, comprising: at least one storage medium storing at least one instruction set for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0023] As can be seen from the above technical solutions, the training method, liveness detection method, and system for the liveness detection model provided in this specification obtain multimodal training data and single-modal training data corresponding to the main modality. The multimodal training data may include training data for multiple modalities, including the main modality and at least one auxiliary modality. Then, based on the multimodal training data, a preset first liveness detection model corresponding to each of the at least one auxiliary modality is trained to obtain an auxiliary liveness detection model corresponding to each auxiliary modality. Furthermore, based on the multimodal training data, single-modal training data, and auxiliary liveness detection models, multimodal knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model. Since this solution can use multimodal training data and single-modal training data to distill important information from at least one auxiliary modality into the preset second liveness detection model corresponding to the main modality, the preset second liveness detection model of a single modality can learn knowledge from other modalities. Therefore, the training accuracy of the target liveness detection model corresponding to the main modality can be improved, thereby improving the accuracy of liveness detection.

[0024] The training methods for the liveness detection model, the liveness detection methods, and other functions of the system provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the training methods for the liveness detection model, the liveness detection methods, and the system provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0027] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0028] Figure 3 A flowchart illustrating a training method for a liveness detection model according to embodiments of this specification is shown; and

[0029] Figure 4A schematic diagram is shown illustrating an application scenario of knowledge distillation of an RGB liveness detection model according to an embodiment of this specification;

[0030] Figure 5 A schematic diagram of a process for training an RGB-corresponding target liveness detection model is shown according to an embodiment of this specification;

[0031] Figure 6 A schematic diagram illustrating a visualization analysis of characteristics before and after distillation, according to embodiments of this specification, is shown; and

[0032] Figure 7 A schematic flowchart of a liveness detection method provided according to an embodiment of this specification is shown. Detailed Implementation

[0033] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0034] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0035] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0036] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0037] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0038] Liveness detection: In biometrics, it is a technique used to determine whether a user is a real person, rather than a target of attacks such as printed photos, masks, or head models.

[0039] Multimodal knowledge distillation: This method uses knowledge distillation to distill important information from multimodal data into a single-modal liveness detection model, thereby enabling the single-modal liveness detection model to have stronger capabilities without increasing any computational cost.

[0040] The liveness detection model provided in this specification can be applied to any liveness detection scenario in biometric processes, such as facial payment, access control, attendance, and identity verification. In liveness detection scenarios, the liveness detection method described in this specification can also be used to perform liveness detection on the target user's image. Besides the above-mentioned liveness detection scenarios, it can also be applied to any other liveness detection scenario, which will not be elaborated upon here. The biometric features may include, but are not limited to, one or more of the following: facial image, iris, sclera, fingerprint, palm print, voiceprint, and skeletal projection.

[0041] Those skilled in the art should understand that the liveness detection model training method, liveness detection method, and system described in this specification are also within the scope of protection of this specification when applied to other application scenarios.

[0042] Figure 1 This diagram illustrates an application scenario of a liveness detection system 100 provided according to an embodiment of this specification. The liveness detection system 100 (hereinafter referred to as System 100) can be applied to liveness detection in any scenario, such as liveness detection in scenarios like facial payment, access control, attendance, and identity verification, etc. Figure 1 As shown, system 100 may include user 110, client 120, server 130 and network 140.

[0043] User 110 can be the user who triggers the liveness detection. User 110 can trigger the liveness detection operation on client 120. User 110 can be the target user to be detected, or it can be the administrator of the liveness detection system 100.

[0044] Client 120 can be a device for performing liveness detection on a target user image in response to a liveness detection operation by user 110. In some embodiments, the liveness detection method can be executed on client 120. In this case, client 120 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 120 may include a hardware device with data processing capabilities and the necessary programs required to drive the hardware device. Figure 1 As shown, client 120 can communicate with server 130. In some embodiments, server 130 can communicate with multiple clients 120. In some embodiments, client 120 can interact with server 130 via network 140 to receive or send messages, etc. In some embodiments, client 120 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality controllers, augmented reality headsets, augmented reality glasses, augmented reality controllers, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, client 120 may be a device with positioning technology for locating the position of client 120.

[0045] In some embodiments, client 120 may have one or more applications (APPs) installed. The APPs provide user 110 with the ability and interface to interact with the outside world via network 140. These APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, client 120 may have a target APP installed. The target APP can provide client 120 with a target user image or perform liveness detection on the target user image. In some embodiments, user 110 can also trigger a liveness detection request for the target user through the target APP. The target APP can respond to the liveness detection request by executing the liveness detection method described in this specification. The liveness detection method will be described in detail later.

[0046] Server 130 may be a server providing various services, such as a server performing liveness detection on the target user image obtained by client 120, or a server providing other services when client 120 performs liveness detection. In some embodiments, the liveness detection search method may be executed on server 130. In this case, server 130 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, server 130 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device. Server 130 may be communicatively connected to multiple clients 120 and receive data sent by clients 120.

[0047] Network 140 serves as a medium to provide a communication connection between client 120 and server 130. Network 140 can facilitate the exchange of information or data. Figure 1 As shown, client 120 and server 130 can connect to network 140 and transmit information or data to each other through network 140. In some embodiments, network 140 can be any type of wired or wireless network, or a combination thereof. For example, network 140 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth networks. TM ZigBee TMA network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 140 may include one or more network access points. For example, network 140 may include a wired or wireless network access point, such as a base station or an internet exchange point, through which one or more components of client 120 and server 130 can connect to network 140 to exchange data or information.

[0048] It should be understood that Figure 1 The number of clients 120, servers 130, and networks 140 shown is merely illustrative. Any number of clients 120, servers 130, and networks 140 can be used depending on implementation needs.

[0049] It should be noted that the liveness detection method can be executed entirely on the client 120, entirely on the server 130, or partially on the client 120 and partially on the server 130.

[0050] One example is a schematic diagram illustrating an application scenario for a training system for a liveness detection model. Figure 1 As shown, the training system for the liveness detection model can train a target liveness detection model. The target liveness model is then used in system 100 to perform liveness detection on the target user. For details, please refer to the previous text, which will not be repeated here.

[0051] It is understood that, in the specific embodiments of this application, multimodal training data, unimodal training data, target user images or other user data and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] Figure 2 A hardware structure diagram of a computing device 200 provided according to an embodiment of this specification is shown. The computing device 200 can execute the training method and / or liveness detection method of the liveness detection model described in this specification. The training method and / or liveness detection method of the liveness detection model are described in other parts of this specification. When the training method and / or liveness detection method of the liveness detection model is executed on a client 120, the computing device 200 can be the client 120. When the training method and / or liveness detection method of the liveness detection model is executed on a server 130, the computing device 200 can be the server 130. When the training method and / or liveness detection method of the liveness detection model can be executed partly on the client 120 and partly on the server 130, the computing device 200 can be both the client 120 and the server 130.

[0053] like Figure 2 As shown, the computing device 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, the computing device 200 may also include a communication port 240 and an internal communication bus 210. Additionally, the computing device 200 may also include I / O components 250.

[0054] The internal communication bus 210 can connect different system components, including storage medium 230, processor 220 and communication port 240.

[0055] I / O component 250 supports input / output between computing device 200 and other components.

[0056] Communication port 240 is used for data communication between computing device 200 and the outside world. For example, communication port 240 can be used for data communication between computing device 200 and network 140. Communication port 240 can be a wired communication port or a wireless communication port.

[0057] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the training method and / or liveness detection method of the liveness detection model provided in this specification.

[0058] At least one processor 220 can be communicatively connected to at least one storage medium 230 and a communication port 240 via an internal communication bus 210. At least one processor 220 is used to execute the at least one instruction set described above. When the computing device 200 is running, at least one processor 220 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the training method and / or liveness detection method of the liveness detection model provided in this specification. The processor 220 can execute all the steps included in the training method and / or liveness detection method of the liveness detection model. The processor 220 can be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 220 is described in this specification for computing device 200. However, it should be noted that computing device 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 220 of computing device 200 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0059] Figure 3 A flowchart of a training method 300 for a liveness detection model according to an embodiment of this specification is shown. As previously described, computing device 200 can execute the training method 300 for the liveness detection model of this specification. Specifically, processor 220 can read an instruction set stored in its local storage medium and then execute the training method 300 for the liveness detection model of this specification according to the instructions in the instruction set. Figure 3 As shown, method 300 may include:

[0060] S320: Obtain multimodal training data and single-modal training data corresponding to the dominant modality.

[0061] The multimodal training data includes training data for multiple modalities, which may include a primary modality and at least one auxiliary modality. The multiple modalities may include at least two of the following: color images, depth images, near-infrared (NIR) images, or thermal imaging images. The color images may include images in at least one color space, such as RGB images, black and white images, grayscale images, or other color images, etc. The primary modality may include one of the multiple modalities, and the at least one auxiliary modality may include at least one modality other than the primary modality. For example, if the multiple modalities include RGB images, depth images, and NIR images, and the primary modality is an RGB image, then the auxiliary modality may include at least one of depth images and NIR images. The multimodal training data may include user image samples corresponding to each modality of user samples collected in a multimodal environment. The single-modal training data may include user image samples of user samples collected in a single-modal environment.

[0062] There are several ways to obtain multimodal training data and single-modal training data corresponding to the dominant modality, as follows:

[0063] For example, processor 220 can receive multimodal training data and single-modal training data corresponding to the main modality uploaded by user 110 through client 120 or terminal; or, it can acquire user images of user samples through multiple modal image acquisition devices to obtain multimodal training data, and acquire user images of user samples through the image acquisition device corresponding to the main modality to obtain single-modal training data corresponding to the main modality; or, it can obtain multimodal training data and single-modal training data corresponding to the main modality from a network or user image database; or, when the amount of multimodal training data and single-modal training data or the memory is large, it can also receive a liveness detection model training request, which may include the storage address of multimodal training data and single-modal training data, and obtain multimodal training data and single-modal training data corresponding to the main modality based on the storage address, and so on.

[0064] S340: Based on multimodal training data, train the preset first liveness detection model corresponding to each auxiliary modality in at least one auxiliary modality to obtain the auxiliary liveness detection model corresponding to each auxiliary modality.

[0065] For example, the processor 220 can obtain a preset first liveness detection model corresponding to each auxiliary modality, select the current modality training data corresponding to each auxiliary modality from the multimodal training data, and train the corresponding preset first liveness detection model based on the current modality training data to obtain the auxiliary liveness detection model corresponding to each auxiliary modality.

[0066] The current modality training data can be the training data corresponding to each auxiliary modality in the multimodal training data. For example, taking NIR images as an auxiliary modality, the corresponding current modality training data can include the training data corresponding to NIR images in the multimodal training data. Based on the current modality training data, there are multiple ways to train the corresponding preset first liveness detection model. For example, the processor 220 can obtain the liveness label of each user sample in the current modality training data, and based on the liveness label and the current modality training data, train the corresponding preset first liveness detection model with the constraint that the target liveness classification loss is less than a preset value, thereby obtaining the auxiliary liveness detection model corresponding to each auxiliary modality.

[0067] Among them, the target liveness classification loss constrains the liveness classification result output by the preset first liveness detection model to approach the liveness label during the training process.

[0068] It should be noted that, taking the auxiliary modalities including NIR images and Depth images as an example, the auxiliary liveness detection model corresponding to NIR and the auxiliary liveness detection model corresponding to Depth can be trained using the current modal training data. The trained auxiliary liveness detection model corresponding to NIR and the auxiliary liveness detection model corresponding to Depth can respectively contain all the important information or data of Depth data and NIR data, which means that these two auxiliary liveness detection models can learn the important information or data of the corresponding auxiliary modality in multimodal data.

[0069] S360: Based on multimodal training data, single-modal training data, and an auxiliary liveness detection model, multimodal knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model.

[0070] The multimodal training data may include at least one first user template in each of the multiple modalities, representing the first user image samples.

[0071] Multimodal knowledge distillation can be understood as distilling the knowledge of the auxiliary modal into the preset second liveness detection model corresponding to the main modal. This allows the preset second liveness detection model to learn the important information of the auxiliary modal, so that it still has the prior knowledge of the auxiliary modal in a single-modal deployment environment. Therefore, it can improve the performance of the target liveness detection model and thus improve the accuracy of liveness detection.

[0072] Among them, there are several ways to perform multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality based on multimodal training data, single-modal training data, and auxiliary liveness detection model. The specific methods are as follows:

[0073] For example, processor 220 can use an auxiliary liveness detection model to extract features from the first user image sample of the corresponding modality to obtain auxiliary modality user features under each auxiliary modality. Based on the auxiliary modality user features, single-modality training data, and multi-modality training data, multi-modality knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model, as follows:

[0074] S362: Use an auxiliary liveness detection model to extract features from the first user image sample of the corresponding modality to obtain the auxiliary modality user features under each auxiliary modality.

[0075] The auxiliary modality user features may include user features of the corresponding modality extracted using a trained auxiliary liveness detection model. The auxiliary liveness detection model may include a feature extraction network, which may include multiple network layers.

[0076] There are several ways to use an auxiliary liveness detection model to extract features from the first user image sample of the corresponding modality, as follows:

[0077] For example, the processor 220 can select candidate first user image samples corresponding to the auxiliary liveness detection model from the first user image samples, input the candidate first user image samples into the feature extraction network to obtain the user feature set corresponding to the candidate first user image samples, and select user features output by at least one network layer from the multiple network layers in the user feature set to obtain the auxiliary modality user features corresponding to each auxiliary modality.

[0078] S364: Based on auxiliary modality user features, single-modality training data, and multimodality training data, multimodal knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model.

[0079] The unimodal user data may include at least one second user image sample in the primary modality. The first user sample and the second user sample may be the same user or different users.

[0080] There are several ways to perform multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality, as follows:

[0081] For example, the processor 220 can use a preset second liveness detection model to extract features from the second user image sample to obtain the main modality user features corresponding to the main modality, and perform mixed training on the preset second liveness detection model based on the main modality user features, auxiliary modality user features, single modality training data and multimodality training data to obtain the target liveness detection model.

[0082] The method of using a preset second liveness detection model to extract features from the second user image sample is similar to the method of using a preset first liveness detection model to extract features from the first user image sample, as detailed above, and will not be repeated here.

[0083] After extracting the main modality user features corresponding to the main modality, the processor 220 can perform hybrid training on the preset second liveness detection model based on the main modality user features, auxiliary modality user features, single-modality training data, and multimodality training data to obtain the target liveness detection model. There are several ways to perform hybrid training on the preset second liveness detection model. For example, the processor 220 can train the preset second liveness detection model based on single-modality training data to obtain a first candidate liveness detection model; perform feature knowledge distillation on the first candidate liveness detection model based on the main modality user features, auxiliary modality user features, and multimodality training data to obtain a second candidate liveness detection model; and use the second candidate liveness detection model as the preset liveness detection model, then return to execute the step of training the preset second liveness detection model based on single-modality training data until the preset second liveness detection model converges, thus obtaining the target liveness detection model.

[0084] The preset second liveness detection model may include a main modality feature extraction network corresponding to the main modality, a feature generation network corresponding to each auxiliary modality, and a liveness classification network. There are various ways to train the preset second liveness detection model based on single-modality training data. For example, the processor 220 can train the main modality feature extraction network and the liveness classification network of the preset second liveness detection model based on single-modality training data to obtain a first candidate liveness detection model. The method of training the feature extraction network and the liveness classification network of the preset second liveness detection model is similar to the method of training the corresponding preset first liveness detection model based on the current modality training data, as detailed above, and will not be repeated here.

[0085] After training a preset second liveness detection model based on single-modal training data, the processor 220 can perform feature knowledge distillation on the first candidate liveness detection model based on the main modality user features, auxiliary modality user features, and multimodal training data to obtain the second candidate liveness detection model. There are several ways to perform feature knowledge distillation on the first candidate liveness detection model. For example, the processor 220 can select the training data corresponding to the main modality from the multimodal training data to obtain the main modality training data. It can then use the first candidate liveness detection model to extract features from the main modality training data to obtain user liveness features. Based on the user liveness features, the main modality user features, and the auxiliary modality user features, it can determine the feature distillation loss information and, based on the distillation loss information, converge the first candidate liveness detection model to obtain the second candidate liveness detection model.

[0086] The first candidate liveness detection model may include a main modality feature extraction network and a feature generation network for each auxiliary modality. There are various ways to extract features from the main modality training data using the first candidate liveness detection model. For example, the processor 220 may use the main modality feature extraction network to extract features from the main modality training data to obtain the first modality user features corresponding to the main modality. Based on the first modality user features and the multimodal training data, the feature generation network generates the second modality user features corresponding to each auxiliary modality. Finally, the first and second modality user features are used as user liveness features.

[0087] After extracting the user liveness features, the processor 220 can determine the feature distillation loss information based on the user liveness features, the primary modality user features, and the secondary modality user features. The feature distillation loss constrains the alignment of the primary modality user features and the secondary modality user features with the corresponding modal features in the user liveness features during training. The feature distillation loss includes at least one of feature distance loss, instance relation loss, and feature space loss.

[0088] Specifically, the feature distance loss constraint during training stipulates that the feature distance between modal user features of the same modality is less than a preset first distance threshold, and the feature distance between modal user features of different modalities is greater than a preset second distance threshold. There are various ways to determine the feature distance loss. For example, the processor 220 determines the feature distance between the second modal user feature and the corresponding auxiliary modal user feature, and determines the feature distance loss based on the feature distance. Alternatively, it can determine the feature similarity between the second modal user feature and the corresponding auxiliary modal user feature, and determine the feature distance loss based on the feature similarity, and so on.

[0089] There are various types of feature distances, such as L2 distance or other types of feature distances, etc.

[0090] The instance relationship loss constrains the relationship features of instances within the same modality and instances within different modalities during the training process. There are multiple ways to determine the instance relationship loss. For example, the processor 220 can determine the first relationship features between instances in different modalities and the second relationship features between instances in the same modality based on first modal user features, second modal user features, primary modal user features, and auxiliary modal user features, and then determine the instance relationship loss based on the first and second relationship features.

[0091] Specifically, the feature space loss constraint during training ensures that the similarity between feature spaces within the same modality is greater than a preset first similarity threshold, and the similarity between feature spaces in different modalities is less than a preset second similarity threshold. There are various ways to determine the feature space loss. For example, the processor 220 can determine the first similarity of the feature space within the same modality and the second similarity of the feature spaces in different modalities based on first modality user features, second modality user features, main modality user features, and auxiliary modality user features, and then determine the feature space loss based on the first and second similarities.

[0092] The processor 220 can use at least one of the feature distance loss, instance relation loss, and feature space loss as the feature distillation loss. After determining the feature distillation loss, the first candidate liveness detection model can be converged based on the feature distillation loss to obtain the trained second candidate liveness detection model. There are several ways to converge the first candidate liveness detection model. For example, the processor 220 can update the network parameters of the first candidate liveness detection model using a gradient descent algorithm based on the feature distillation loss to obtain the trained second candidate liveness detection model. Alternatively, other parameter update algorithms can be used to update the network parameters of the first candidate liveness detection model based on the feature distillation loss to obtain the trained second candidate liveness detection model.

[0093] After training the first candidate liveness detection model, the processor 220 can use the trained second candidate liveness detection model as the preset second liveness detection model, and return to execute the step of training the preset second liveness detection model based on the single-modality training data until the preset second liveness detection model converges, thus obtaining the target liveness detection model.

[0094] In some embodiments, the processor 220's hybrid training of the preset second liveness detection model may further include: the processor 220 performing feature knowledge distillation on the preset second liveness detection model based on the main modality user features, auxiliary modality user features, and multimodal training data to obtain a first candidate liveness detection model; training the first candidate liveness detection model based on the single modality training data to obtain a second candidate liveness detection model; and using the second candidate liveness detection model as the preset second liveness detection model, and returning to perform the step of performing feature knowledge distillation on the preset second liveness detection model until the preset second liveness detection model converges to obtain the target liveness detection model.

[0095] It should be noted that when performing hybrid training on the preset second liveness detection model, you can first use single-modal training data to train the preset second liveness detection model, or you can first use multi-modal training data to perform feature knowledge distillation on the preset second liveness detection model. The specific training process is the same in both hybrid training methods.

[0096] Taking RGB images as the primary mode and NIR and Depth images as auxiliary modes as an example, this scheme uses knowledge distillation to distill important information from the NIR and Depth modes into the RGB mode model. Application scenarios include... Figure 4 As shown, multimodal training data in a multimodal application environment and single-modal training data in a pure RGB application environment are collected. Using the single-modal and multimodal training data, key information from the NIR and Depth modes is distilled into the target liveness detection model corresponding to the RGB mode. The process of training the RGB-corresponding target liveness detection model using knowledge distillation can be described as follows: Figure 5 As shown, the specific details are as follows:

[0097] (1) Collect multimodal application environment data (RGB image, NIR image and Depth image), and train the Depth liveness detection model and NIR liveness detection model respectively using the multimodal training data. At this time, the trained Depth liveness detection model and NIR liveness detection model contain all the important information of the Depth data and NIR data in the multimodal data.

[0098] (2) The trained Depth liveness detection model and NIR liveness detection model are used to extract features from the Depth data and NIR data in the multimodal training data, respectively, so as to obtain the modal user features corresponding to the Depth data and the modal user features corresponding to the NIR data. The modal user features corresponding to the Depth data and the modal user features corresponding to the NIR data are used as the feature knowledge to be distilled.

[0099] (3) In single RGB modal application scenarios, collect environmental data in RGB modality (user images in RGB modality) to obtain single modal training data.

[0100] (4) Use the preset RGB liveness detection model to extract features from the single-modal training data to obtain the modal user features corresponding to the extracted RGB data.

[0101] (5) The preset RGB liveness detection model is trained using a combination of multimodal and single-modal training data. During the mixed training process, a knowledge distillation algorithm can be used to distill the Depth modality information and NIR modality information into the single RGB model, allowing the model to learn important information from the other two modalities. In the process of feature distillation, in order to ensure feature alignment within the same modality, distance information between features, relationship information between instances, and change information in the feature space can be introduced for knowledge distillation, thereby obtaining an enhanced liveness detection model in the single RGB modality. This enhanced liveness detection model is then used as the target liveness detection model corresponding to the RGB modality. By introducing this information, the student network (the preset RGB liveness detection model) is helped to imitate the teacher network (the NIR liveness detection model and the Depth liveness detection model), improving the learning ability of the student network and giving it good robustness.

[0102] It should be noted that in this solution, data collected in a multimodal application environment can be used to distill the model in a pure RGB application environment, thereby significantly enhancing the capabilities of the model in the pure RGB application environment. Furthermore, the target liveness detection model corresponding to the RGB modality achieves a significant performance improvement without increasing computational load, outperforming models trained using only single RGB modality data. Even in a single RGB deployment environment, some prior knowledge of Depth and NIR is still retained. The t-SNE visualization method was used to visualize and analyze the features before and after distillation, as detailed below. Figure 6 As shown in the visualization results, for samples that are inseparable from the features of the single RGB classification model, the distilled model achieves excellent classification performance. At the same time, the number of false positives and difficult-to-classify samples is significantly reduced, demonstrating the effectiveness of cross-modal feature distillation.

[0103] The processor 220 performs multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality based on multimodal training data, single-modal training data and auxiliary liveness detection model to obtain the target liveness detection model. Then, it can perform liveness detection on the target user image of the target user based on the target liveness detection model. Figure 7A flowchart of a liveness detection method 400 according to an embodiment of this specification is shown. As previously described, the computing device 200 can execute the liveness detection method 400 of this specification. Specifically, the processor 220 can read an instruction set stored in its local storage medium and then execute the liveness detection method 400 of this specification according to the instructions in the instruction set. Figure 7 As shown, method 400 may include:

[0104] S420: Obtain the target user image in unimodal mode.

[0105] The target user image may include an image of the user's biometric features in a single modality. These biometric features may include, but are not limited to, one or more of facial images, iris images, scleral images, fingerprints, palm prints, voiceprints, and skeletal projections. The single modality includes one of multiple modalities, which may include at least two modalities selected from color images, depth images, infrared images, or thermal imaging images, as detailed above.

[0106] There are several ways to obtain the target user image in a single modality, including the following:

[0107] For example, processor 220 can receive a target user image in unimodal mode uploaded by user 110 through client 120 or terminal; or, it can acquire at least one user image of the target user through a unimodal image acquisition device to obtain a target user image; or, it can select at least one unimodal user image of the target user from a network or image database to obtain a target user image; or, if there are many target user images or a large amount of memory, it can also receive a liveness detection request, which includes the storage address of the target user image in unimodal mode, and obtain the target user image based on the storage address, etc.

[0108] S440: Based on a single-modal corresponding target liveness detection model, feature extraction is performed on the target user image to obtain the target user liveness features.

[0109] The target user features include user liveness features across multiple modalities, including a single modality.

[0110] The target liveness detection model may include a feature extraction network corresponding to a single modality and a feature generation network corresponding to a target modality, wherein the target modality includes modalities other than the single modality among multiple modalities.

[0111] Among them, there are several ways to extract features from target user images based on a single-modality target liveness detection model, as follows:

[0112] For example, the processor 220 can input the target user image into the feature extraction network to obtain the first user liveness feature corresponding to the single modality, and based on the target user image and the first user liveness feature, use the feature generation network to generate the second user liveness feature corresponding to the target modality, and fuse the first user liveness feature and the second user liveness feature to obtain the target user liveness feature.

[0113] The second user liveness feature may include user liveness features related to the target modality in the target user image. Based on the target user image and the first user liveness feature, there are various ways to generate the second user liveness feature corresponding to the target modality using a feature generation network. For example, the processor 220 can input the target user image and / or the first user liveness image features into the feature generation network to obtain the second user liveness feature corresponding to the target modality; or, it can input the target user image and / or the first user liveness image features into the feature generation network to obtain candidate user images corresponding to the target modality, extract features from the candidate user images, and obtain the second user liveness feature corresponding to the target modality, and so on.

[0114] S460: Based on the liveness characteristics of the target user, determine the liveness detection result of the target user and output the liveness detection result.

[0115] The liveness detection result may include either the target user being an attacking user or a live user.

[0116] There are several ways to determine the liveness detection result of a target user based on the target user's liveness characteristics, as follows:

[0117] For example, processor 220 can determine the attack probability of the target user based on the target user's liveness characteristics, and perform the target operation based on the attack probability, thereby obtaining the liveness detection result of the target user.

[0118] The attack probability can include the probability that the target user is the attacker. There are several ways to determine the attack probability of the target user based on their liveness characteristics. For example, the processor 220 can input the target user's liveness characteristics into the liveness classification network of the target liveness detection model to obtain the attack probability of the target user.

[0119] The target operation may include a first operation or a second operation. The first operation may include determining that the attack probability is greater than a preset probability threshold, and determining that the target user's liveness detection result is that of an attacking user. The second operation may include determining that the attack probability is less than a preset probability threshold, and determining that the target user's liveness detection result is that of a live user.

[0120] After determining the liveness detection result of the target user, the processor 220 can output the liveness detection result. There are several ways to output the liveness detection result. For example, the processor 220 can directly send the liveness detection result to the client 120, terminal or server corresponding to the user 110, so that the client 120, terminal or server can respond to the target user or the request corresponding to the target user based on the liveness detection result. Alternatively, the liveness detection result can be directly visualized, and so on.

[0121] There are several ways to visualize the liveness detection results. For example, the processor 220 can directly display the liveness detection result, or it can display the liveness detection result through sound and light (for example, by broadcasting the liveness detection result by voice, or by displaying different types of liveness detection results by displaying different colored lights, or by displaying the liveness detection result through sound and light linkage), or it can display the liveness detection result for a specific type of liveness detection result (for example, only displaying the liveness detection result for the attacking user type, or only displaying the liveness detection result for the live user type, etc.), and so on.

[0122] In some embodiments, after determining or outputting the liveness detection result of the target user, the processor 220 may respond to the target user or the request corresponding to the target user based on the liveness detection result. There may be various ways to respond. For example, the processor 220 may directly intercept the target user or the request corresponding to the target user, or the processor 220 may directly perform secondary verification on the target user and, based on the secondary verification result, provide a final response to the target user or the request corresponding to the target user, and so on.

[0123] In summary, the liveness detection model training method 300, liveness detection method 400, and system 100 provided in this specification obtain multimodal training data and single-modal training data corresponding to the main modality. The multimodal training data may include training data for multiple modalities, including the main modality and at least one auxiliary modality. Then, based on the multimodal training data, a preset first liveness detection model corresponding to each of the at least one auxiliary modality is trained to obtain an auxiliary liveness detection model corresponding to each auxiliary modality. Furthermore, based on the multimodal training data, single-modal training data, and auxiliary liveness detection models, multimodal knowledge distillation is performed on a preset second liveness detection model corresponding to the main modality to obtain a target liveness detection model. Since this scheme can use multimodal training data and single-modal training data to distill important information from at least one auxiliary modality into the preset second liveness detection model corresponding to the main modality, the preset second liveness detection model of a single modality can learn knowledge from other modalities. Therefore, the training accuracy of the target liveness detection model corresponding to the main modality can be improved, thereby improving the accuracy of liveness detection.

[0124] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for training a liveness detection model and / or performing liveness detection. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the liveness detection model training method 300 and / or liveness detection method 400 described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 200, the program code causes the computing device 200 to perform the steps of the liveness detection model training method 300 and / or liveness detection method 400 described in this specification. The program product for implementing the above methods may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computing device 200. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages.The program code can be executed entirely on computing device 200, partially on computing device 200, as a standalone software package, partially on computing device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0127] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0128] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0129] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0130] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A training method for a liveness detection model, comprising: Obtain multimodal training data and single-modal training data corresponding to the main modality. The multimodal training data includes training data for multiple modalities, including the main modality and at least one auxiliary modality. The multimodal training data includes at least one first user sample as a first user image sample in each of the multiple modalities, and the single-modal training data includes at least one second user sample as a second user image sample in the main modality. Based on the multimodal training data, a preset first liveness detection model corresponding to each of the at least one auxiliary modalities is trained to obtain an auxiliary liveness detection model corresponding to each auxiliary modality; and Based on the multimodal training data, the unimodal training data, and the auxiliary liveness detection model, multimodal knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model; wherein the multimodal knowledge distillation of the preset second liveness detection model corresponding to the main modality includes: The auxiliary liveness detection model is used to extract features from the first user image sample of the corresponding modality to obtain the auxiliary modality user features under each auxiliary modality; The preset second liveness detection model is used to extract features from the second user image sample to obtain the dominant modality user features corresponding to the dominant modality; and Based on the primary modality user features, the auxiliary modality user features, the single-modality training data, and the multimodality training data, the preset second liveness detection model is trained in a hybrid manner to obtain the target liveness detection model.

2. The training method for the liveness detection model according to claim 1, wherein, The plurality of modalities includes at least two modalities selected from color images, depth images, infrared images, or thermal imaging images. The primary modality includes one of the plurality of modalities, and the at least one auxiliary modality includes at least one modality other than the primary modality among the plurality of modalities.

3. The training method for the liveness detection model according to claim 1, wherein, The step of training a preset first liveness detection model corresponding to each of the at least one auxiliary modalities to obtain an auxiliary liveness detection model corresponding to each auxiliary modality includes: Obtain the preset first liveness detection model corresponding to each auxiliary modality; Select the current modality training data corresponding to each auxiliary modality from the multimodal training data; and Based on the current modality training data, the corresponding preset first liveness detection model is trained to obtain the auxiliary liveness detection model corresponding to each auxiliary modality.

4. The training method for the liveness detection model according to claim 1, wherein, The step of performing multimodal knowledge distillation on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model includes: Based on the auxiliary modality user features, the single modality training data, and the multimodality training data, multimodal knowledge distillation is performed on the preset second liveness detection model corresponding to the main modality to obtain the target liveness detection model.

5. The training method for the liveness detection model according to claim 4, wherein, The auxiliary liveness detection model includes a feature extraction network, which comprises multiple network layers; and The step of using the auxiliary liveness detection model to extract features from the first user image sample of the corresponding modality to obtain the auxiliary modality user features corresponding to each auxiliary modality includes: Candidate first user image samples corresponding to the auxiliary liveness detection model are selected from the first user image samples. The candidate first user image sample is input into the feature extraction network to obtain the user feature set corresponding to the candidate first user image sample, and In the user feature set, at least one network layer output user feature is selected from the plurality of network layers to obtain the auxiliary modality user feature corresponding to each auxiliary modality.

6. The training method for the liveness detection model according to claim 4, wherein, The step of performing hybrid training on the preset second liveness detection model to obtain the target liveness detection model includes: The preset second liveness detection model is trained based on the single-modal training data to obtain the first candidate liveness detection model; Based on the primary modality user features, the auxiliary modality user features, and the multimodal training data, feature knowledge distillation is performed on the first candidate liveness detection model to obtain a second candidate liveness detection model; and The second candidate liveness detection model is used as the preset second liveness detection model, and the step of training the preset second liveness detection model based on the single-modality training data is returned to be executed until the preset second liveness detection model converges, thus obtaining the target liveness detection model.

7. The training method for the liveness detection model according to claim 6, wherein, The step of performing feature knowledge distillation on the first candidate liveness detection model to obtain the second candidate liveness detection model includes: The training data corresponding to the dominant mode is selected from the multimodal training data to obtain the dominant mode training data; The first candidate liveness detection model is used to extract features from the main modality training data to obtain user liveness features; Based on the user liveness features, the primary modality user features, and the secondary modality user features, the feature distillation loss is determined; and Based on the feature distillation loss, the first candidate liveness detection model is converged to obtain the second candidate liveness detection model.

8. The training method for the liveness detection model according to claim 7, wherein, The first candidate liveness detection model includes a main modality feature extraction network and a feature generation network corresponding to each auxiliary modality; as well as The step of using the first candidate liveness detection model to extract features from the main modality training data to obtain user liveness features includes: The dominant modality feature extraction network is used to extract features from the dominant modality training data to obtain the first modality user features corresponding to the dominant modality. Based on the first modality user features and the multimodal training data, the feature generation network generates second modality user features corresponding to each auxiliary modality, and The first modality user features and the second modality user features are used as the user liveness features.

9. The training method for the liveness detection model according to claim 7, wherein, During the training process constrained by the feature distillation loss, the main modality user features and the auxiliary modality user features are respectively aligned with the features of the corresponding modality in the user liveness features. The feature distillation loss includes at least one of feature distance loss, instance relation loss and feature space loss. During the training process constrained by the feature distance loss, the feature distance between modal user features of the same modality is less than a preset first distance threshold, and the feature distance between modal user features of different modalities is greater than a preset second distance threshold. The instance relation loss constraint training process includes relation features corresponding to instances in the same modality and relation features corresponding to instances in different modalities; and During the feature space loss constraint training process, the similarity between feature spaces in the same modality is greater than a preset first similarity threshold, and the similarity between feature spaces in different modalities is less than a preset second similarity threshold.

10. The training method for the liveness detection model according to claim 4, wherein, The step of performing hybrid training on the preset second liveness detection model to obtain the target liveness detection model includes: Based on the primary modality user features, the auxiliary modality user features, and the multimodal training data, feature knowledge distillation is performed on the preset second liveness detection model to obtain a first candidate liveness detection model; The first candidate liveness detection model is trained based on the single-modal training data to obtain the second candidate liveness detection model; and The second candidate liveness detection model is used as the preset second liveness detection model, and the step of performing feature knowledge distillation on the preset second liveness detection model is returned until the preset second liveness detection model converges, thus obtaining the target liveness detection model.

11. A method for detecting liveness, comprising: Obtain the target user image in a single modality; Based on the target liveness detection model corresponding to the single modality, feature extraction is performed on the target user image to obtain target user liveness features. These target user liveness features include user liveness features across multiple modalities, including the single modality. The target liveness detection model includes a feature extraction network corresponding to the single modality and a feature generation network corresponding to the target modality, wherein the target modality includes modalities other than the single modality among the plurality of modalities; The step of extracting features from the target user image to obtain target user liveness features includes: inputting the target user image into the feature extraction network to obtain a first user liveness feature corresponding to the single modality; generating a second user liveness feature corresponding to the target modality using the feature generation network based on the target user image and the first user liveness feature; and fusing the first user liveness feature and the second user liveness feature to obtain the target user liveness feature; and Based on the liveness characteristics of the target user, the liveness detection result of the target user is determined and output.

12. The liveness detection method according to claim 11, wherein, The plurality of modalities includes at least two modalities selected from color images, depth images, infrared images, or thermal imaging images, and the single modality includes one of the plurality of modalities.

13. The liveness detection method according to claim 11, wherein, The step of determining the liveness detection result of the target user based on the target user's liveness features includes: Based on the target user's liveness characteristics, determine the attack probability of the target user; and Based on the attack probability, a target operation is performed, wherein the target operation includes a first operation or a second operation, wherein: The first operation includes determining that the attack probability is greater than a preset probability threshold, determining that the liveness detection result of the target user is that of the attacking user, and The second operation includes determining that the attack probability is less than the preset probability threshold and determining that the liveness detection result of the target user is a live user.

14. A training system for a liveness detection model, comprising: At least one storage medium storing at least one instruction set for training a liveness detection model; as well as At least one processor is communicatively connected to the at least one storage medium. When the training system of the liveness detection model is running, the at least one processor reads the at least one instruction set and executes the training method of the liveness detection model according to any one of the at least one instruction set.

15. A liveness detection system, comprising: At least one storage medium storing at least one instruction set for performing liveness detection; as well as At least one processor is communicatively connected to the at least one storage medium. When the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method according to any one of claims 11-13 according to the instructions of the at least one instruction set.

Images