Liveness detection methods and systems

Abstract

The liveness detection method and system provided in this specification acquire multiple original modal images of the target object's region to be detected, input these images into a first model to obtain corresponding original modal feature vectors, determine at least one stable modality among the multiple modalities based on these feature vectors, and perform liveness detection on the target object based on at least one stable modality to obtain a liveness detection result indicating whether the target object is alive. This approach can improve the stability of liveness detection performance.

Description

Technical Field

[0001] This specification relates to the field of Internet of Things (IoT) technology, and in particular to a liveness detection method and system. Background Technology

[0002] Facial recognition technology has been widely used in various fields, bringing great convenience to people's production and life. However, it has also brought security risks, the most common and threatening of which is liveness detection. This involves attackers using mobile phone photos, printed paper, and high-precision masks to forge facial images, attempting to pass liveness detection and harm users' property and information security. Therefore, liveness detection is necessary.

[0003] Currently, liveness detection is primarily achieved through deep learning models. One approach involves training a separate liveness detection model for each of multiple modalities, such as RGB (red-green-blue), near-infrared (NIR), and 3D (three-dimensional) modalities. Each model then makes a liveness decision based on facial data from that modality, and the decisions from these multiple models are then fused to obtain the final liveness detection result. Alternatively, another approach trains a single liveness detection model for multiple modalities, inputting facial data from all modalities into this model for fusion and decision-making to obtain the final liveness detection result. However, deep learning models experience performance degradation when the application data domain differs significantly from the training data domain.

[0004] Therefore, there is a need to provide a liveness detection method with stable performance. Summary of the Invention

[0005] This specification provides a liveness detection method and system that exhibits stable liveness detection performance across different data domains.

[0006] Firstly, this specification provides a liveness detection method, which involves acquiring multiple original modal images of the target object's region to be detected, wherein the multiple original modal images correspond to images actually acquired under multiple modalities of an image acquisition module; inputting the multiple original modal images into a first model to obtain corresponding multiple original modal feature vectors; determining at least one stable modality among the multiple modalities based on the multiple original modal feature vectors, wherein the stable modality refers to a modality whose stability of liveness detection performance for different data domains meets preset requirements; and performing liveness detection on the target object based on the at least one stable modality to obtain a liveness detection result indicating whether the target object is a live object.

[0007] In some embodiments, the liveness detection of the target object based on the at least one stable modality includes: if there is an unstable modality among the plurality of modalities, then perturbing the original modal image of the unstable modality among the plurality of modalities and combining it with the original modal image of the at least one stable modality to generate an updated multimodal input image; feeding the updated multimodal input image back to the input end of the first model to obtain the updated multimodal liveness decision result corresponding to the updated multimodal input image; and outputting the updated multimodal liveness decision result as the final decision result.

[0008] In some embodiments, perturbing the original modal image of the unstable modality among the plurality of modalities includes: replacing the original modal image of the unstable modality with 0; or reducing the image quality of the original modal image of the unstable modality.

[0009] In some embodiments, inputting the plurality of original modal images into the first model and obtaining corresponding initial multimodal liveness decision results includes: obtaining feature maps of the plurality of modalities by multimodal feature encoding of the plurality of original modal images; and performing feature fusion decision on the feature maps of the plurality of modalities to determine the initial multimodal liveness decision results corresponding to the plurality of original modal images.

[0010] In some embodiments, the step of performing liveness detection on the target object based on the at least one stable modality includes: if there is no unstable modality among the plurality of modalities, outputting the initial multimodal liveness decision result as the final decision result.

[0011] In some embodiments, the step of inputting the plurality of original modal images into a first model to obtain the corresponding plurality of original modal feature vectors includes: performing multimodal feature encoding on the plurality of original modal images under the plurality of modalities to obtain the feature maps of the plurality of modalities; and performing multi-head feature encoding on the feature maps of the plurality of modalities to determine the original modal feature vectors corresponding to the plurality of modalities.

[0012] In some embodiments, determining at least one stable mode among the plurality of modes includes: determining a plurality of comprehensive stability indices corresponding to the plurality of modes, wherein each comprehensive stability index indicates whether each mode is a stable mode.

[0013] In some embodiments, determining the multiple comprehensive stability indices corresponding to the multiple modalities includes: designating each of the multiple modalities as a target modality; performing cross-modal transformation on the original modality feature vector corresponding to the target modality to generate at least one corresponding transmodal feature vector, wherein the at least one transmodal feature vector corresponds to other modalities among the multiple modalities; determining a first stability index corresponding to the target modality based on the similarity between the feature map of the original modality image and the feature map of the at least one transmodal image; and / or performing single-modal classification on the original modality feature vector corresponding to the target modality to obtain the liveness decision result corresponding to the target modality; and determining a second stability index corresponding to the target modality based on the similarity between the liveness decision result and the liveness decision results corresponding to other modalities among the multiple modalities.

[0014] In some embodiments, determining the multiple comprehensive stability indices corresponding to the multiple modes includes determining one of the first stability index or the second stability index as the comprehensive stability index.

[0015] In some embodiments, determining the comprehensive stability index based on the first stability index and the second stability index includes: if the stability indicated by the first stability index and the second stability index both meet preset requirements, then the comprehensive stability index is determined to be a stable mode.

[0016] In some embodiments, determining the comprehensive stability index based on the first stability index and the second stability index includes: if the stability indicated by at least one of the first stability index and the second stability index does not meet a preset requirement, then determining the comprehensive stability index as an unstable mode.

[0017] In some embodiments, the first model includes a first feature extraction network, a feature fusion network, and a second feature extraction network; the first model is trained using the following method: acquiring a first training image set of the plurality of modalities; performing training iterations based on the first training image set of the plurality of modalities until a first comprehensive loss function is less than a preset value, the training iterations including: updating the parameter values ​​of the first model based on the first comprehensive loss function determined in the previous training iteration; inputting the first training image set of the plurality of modalities into the first feature extraction network to determine the feature maps corresponding to the first training image set of the plurality of modalities; inputting the feature maps corresponding to the first training image set of the plurality of modalities into the feature fusion network to determine the initial liveness decision result and its corresponding first loss function; inputting the feature maps corresponding to the first training image set of the plurality of modalities into the second feature extraction network to determine the feature vectors corresponding to the plurality of modalities and their corresponding second loss functions; wherein, the first comprehensive loss function is a weighted sum of the first loss function and the second loss function.

[0018] In some embodiments, obtaining the training image set of the plurality of modalities includes: obtaining the original training image set of the plurality of modalities; generating random numbers for each modality in the plurality of modalities to obtain random numbers for the plurality of modalities; and performing either a first operation or a second operation based on the random numbers for the plurality of modalities, wherein the first operation includes: if all random numbers for the plurality of modalities are greater than or equal to a preset value, then determining the original training image set of the plurality of modalities as the first training image set of the plurality of modalities; the second operation includes: if the random number for at least one modality in the plurality of modalities is less than a preset value, then perturbing the original training image set of the at least one modality to obtain at least one perturbed training image set corresponding to at least one modality, and combining it with the original training image set corresponding to the unperturbed modality to generate the first training image set of the plurality of modalities.

[0019] Secondly, this specification also provides a liveness detection system, comprising: at least one storage medium storing at least one instruction set for performing liveness detection; and at least one processor communicatively connected to the at least one storage medium, wherein, when the liveness detection system is running, the at least one processor reads the at least one instruction set and executes the liveness detection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0020] As can be seen from the above technical solutions, the liveness detection method and system provided in this specification, after acquiring multiple original modal images of the target object's detection region, inputs these multiple original modal images into a first model to obtain corresponding multiple original modal feature vectors. Based on these original modal feature vectors, at least one stable modality is determined among the multiple modalities. This stable modality refers to a modality whose stability of liveness detection performance across different data domains meets preset requirements. Based on at least one stable modality, liveness detection is performed on the target object to obtain a liveness detection result indicating whether the target object is alive. Since this solution determines at least one stable modality from multiple modalities and performs liveness detection on the target object based on at least one stable modality, discarding unstable modalities, it avoids the impact of unstable modalities on liveness detection performance in cross-domain scenarios. The stable modality determines the liveness detection performance, making the liveness detection performance more stable when the data domain of the training dataset differs from that of the application data domain.

[0021] Other functions of the liveness detection methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the liveness detection methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system provided according to an embodiment of this specification is shown.

[0024] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0025] Figure 3 A flowchart of a liveness detection method provided according to an embodiment of this specification is shown;

[0026] Figure 4 A schematic block diagram of a liveness detection method provided according to an embodiment of this specification is shown.

[0027] Figure 5 A flowchart of a training method for a first model provided according to an embodiment of this specification is shown;

[0028] Figure 6 A flowchart of a training method for a second model provided according to an embodiment of this specification is shown. Detailed Implementation

[0029] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0030] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0031] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0032] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0033] For ease of description, the terms used in this specification are explained as follows:

[0034] Liveness detection and interception: In a facial recognition system, liveness detection and interception are performed on attacks, such as those made using photos from mobile phones, paper photos, or masks.

[0035] Multimodal images: refer to different types of images captured by different image acquisition modules, such as RGB, NIR, 3D and thermal imaging images.

[0036] Multimodal liveness detection: Liveness detection is performed using data from multiple modalities, including RGB, NIR, 3D, and thermal imaging.

[0037] Different data domains: Images acquired based on different imaging conditions in different scenarios, such as the physical environment. For example, the imaging conditions and scenarios for the same user in different situations such as payment and access control are different, resulting in different imaging effects. Due to changes in the acquisition conditions and the environment in which the equipment is set up, different data domains are generated.

[0038] Cross-domain liveness detection: refers to using the same liveness detection algorithm to detect liveness across different data domains.

[0039] Cross-domain fluctuations: Liveness detection performance becomes unstable when facing different data domains. For example, a liveness detection model may exhibit high performance on the training dataset, but show a decline in performance when the application dataset differs from the training dataset, resulting in inconsistent liveness detection performance. For instance, when switching from a daytime scene to a nighttime scene, RGB and thermal imaging modalities show significant fluctuations, while NIR and 3D modalities show smaller fluctuations because they are not affected by ambient light and temperature.

[0040] In existing technologies, multimodal liveness detection methods are mainly used for liveness detection protection on devices such as smartphones and Internet of Things (IoT) devices. Common multimodal liveness detection methods mainly include the following two approaches: In one approach, a separate liveness detection model (e.g., RGB liveness model + NIR liveness model + 3D liveness model) is first trained for each modality. Each individual liveness detection model is then used to perform liveness detection for each modality, obtaining the liveness decision result corresponding to each modality. The decision results of multiple liveness detection models are then fused to obtain the final liveness detection result (e.g., averaging the decision results of multiple liveness models). The advantage of this approach is that each modality is independent, allowing for rapid updates to the liveness detection model corresponding to each modality. However, the disadvantage is that when a modality is missing or the image quality of a modality is poor, the liveness detection performance of the entire system becomes unstable, exhibiting significant fluctuations. Another approach involves using data from multiple modalities as input to train a single deep learning model for liveness detection. This method can reduce system stability issues caused by missing modalities and quality problems through data perturbation and network structure design, and it requires less storage and computational resources compared to single-modal fusion decision-making methods. However, both of these approaches use data-driven deep learning models for liveness detection. A significant drawback of deep learning models is that when the application data domain differs greatly from the training data domain—that is, in cross-domain liveness detection scenarios—liveness detection performance drops drastically. Furthermore, since multimodal liveness detection methods are based on data from multiple modalities, in cross-domain scenarios, liveness detection performance is affected by multiple modalities. The final liveness detection performance often depends on the modality with the greatest fluctuation, leading to a bottleneck effect in liveness detection, where the performance is close to the modality with the worst liveness detection performance.

[0041] The liveness detection method and system provided in this specification obtain the original modal feature vector corresponding to each modality in multiple modalities, evaluate the magnitude of cross-domain variability based on the original modal feature vector of each modality in multiple modalities, determine the modality with smaller cross-domain variability, and select the modality with smaller cross-domain variability for liveness detection, thereby discarding the modality with larger cross-domain variability, so as to achieve the purpose of stabilizing the liveness detection performance.

[0042] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0043] The liveness detection method provided in this manual can be applied to any scenario requiring identity verification, such as facial recognition payment services in offline retail stores, AI vending machines, access control systems, or other scenarios requiring facial recognition payment. The next step of identity verification is only performed when the liveness detection result of the detected object confirms that the user is alive. For example, in a payment scenario, the liveness detection method provided in this manual can be used to detect the user's liveness, and facial recognition is only performed on the user after confirming that the user is alive. Furthermore, facial recognition payment is only allowed after confirming that the user is a legitimate user. Similarly, in an access control scenario, the liveness detection method provided in this manual can be used to detect the user's liveness, and facial recognition is only performed on the user after confirming that the user is alive. Access is only allowed after confirming that the user is a legitimate user. Or, in an information query scenario, the liveness detection method provided in this manual can be used to detect the user's liveness, and facial recognition is only performed on the user after confirming that the user is alive. Information queries are only allowed for the user after confirming that the user is a legitimate user.

[0044] Those skilled in the art should understand that the liveness detection method described in this specification can also be applied to other use cases within the scope of protection of this specification.

[0045] Figure 1 A schematic diagram illustrating an application scenario of a liveness detection system 001 provided according to an embodiment of this specification is shown. For example... Figure 1 As shown, the liveness detection system 001 (hereinafter referred to as system 001) may include: a target user 100, a terminal device 200, a server 300, and a network 400. The terminal device 200 and the server 300 are both connected to the network 400.

[0046] In this context, the target user 100 can be the user to be subjected to liveness detection. The target user 100 can trigger the liveness detection function on the terminal device 200 to perform liveness detection on the target user. For example, in a payment service, the target user 100 can enter the facial recognition payment stage through a series of operations; or in an access control scenario, the target user can enter the facial recognition verification stage through a series of operations, thereby triggering the liveness detection function.

[0047] The terminal device 200 can be a device having multiple image acquisition modules. These multiple image acquisition modules can be at least two of RGB image acquisition modules, infrared image acquisition modules, NIR image acquisition modules, 3D image acquisition modules, and thermal imaging image acquisition modules. In some embodiments, a liveness detection method can be executed on the terminal device 200. In this case, the terminal device 200 can store data or instructions for executing the liveness detection method described herein, and can execute or be used to execute the data or instructions. In some embodiments, the terminal device 200 may include hardware devices with data processing capabilities and the necessary programs required to drive the hardware devices. Figure 1 As shown, terminal device 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple terminal devices 200. In some embodiments, terminal device 200 can interact with server 300 through network 400 to receive or send messages, such as receiving or sending images or liveness detection results of multiple modalities. In some embodiments, terminal device 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, mobile devices may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, smart home devices may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, smart mobile devices may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, virtual reality devices or augmented reality devices may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, virtual reality devices or augmented reality devices may include Google Glass, head-mounted displays, VR, etc. In some embodiments, built-in devices in motor vehicles may include in-vehicle computers, in-vehicle televisions, etc.

[0048] In some embodiments, the terminal device 200 may have one or more applications (APPs) installed. The APP provides the target user 100 with the ability and interface to interact with the outside world via the network 400. APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the terminal device 200 may have a target APP installed. The target user 100 can access the target APP through the terminal device 200 and obtain images and liveness detection results in multiple modalities. In some embodiments, the target user 100 can also trigger a liveness detection request through the target APP. The target APP can respond to the liveness detection request and execute the liveness detection method described in this specification. The liveness detection method will be described in detail later.

[0049] Server 300 may be a server providing various services, such as a backend server supporting multiple modal images and liveness detection results acquired on terminal device 200. In some embodiments, the liveness detection method can be executed on server 300. In this case, server 300 may store data or instructions for executing the liveness detection method described herein, and may execute or be used to execute the data or instructions. In some embodiments, server 300 may include hardware devices with data processing capabilities and necessary programs to drive the hardware devices. Server 300 may be communicatively connected to multiple terminal devices 200 and receive data sent by the terminal devices 200.

[0050] Network 400 serves as a medium to provide a communication connection between terminal device 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, terminal device 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable network, wired network, fiber optic network, telecommunications network, intranet, Internet, local area network (LAN), wide area network (WAN), wireless local area network (WLAN), metropolitan area network (MAN), public switched telephone network (PSTN), Bluetooth network. TM ZigBee TMA network, a near-field communication (NFC) network, or a similar network. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or internet exchange points, through which one or more components of terminal device 200 and server 300 can connect to network 400 to exchange data or information.

[0051] It should be understood that Figure 1 The number of terminal devices 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, any number of terminal devices 200, servers 300, and networks 400 can be included.

[0052] It should be noted that the liveness detection method can be executed entirely on the terminal device 200, entirely on the server 300, or partially on the terminal device 200 and partially on the server 300.

[0053] Figure 2 A hardware structure diagram of a computing device 500 provided according to an embodiment of this specification is shown. The computing device 500 can execute the liveness detection method described in this specification. The liveness detection method is described in other parts of this specification. When the liveness detection method is executed on a terminal device 200, the computing device 500 can be the terminal device 200. When the liveness detection method is executed on a server 300, the computing device 500 can be the server 300. When the liveness detection method can be executed partly on the terminal device 200 and partly on the server 300, the computing device 500 can be both the terminal device 200 and the server 300.

[0054] like Figure 2 As shown, the computing device 500 may include at least one storage medium 530 and at least one processor 520. In some embodiments, the computing device 500 may also include a communication port 550 and an internal communication bus 510. Additionally, the computing device 500 may also include I / O components 560.

[0055] The internal communication bus 510 can connect different system components, including storage medium 530, processor 520 and communication port 550.

[0056] I / O component 560 supports input / output between computing device 500 and other components.

[0057] Communication port 550 is used for data communication between computing device 500 and the outside world. For example, communication port 550 can be used for data communication between computing device 500 and network 400. Communication port 550 can be a wired communication port or a wireless communication port.

[0058] Storage medium 530 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 532, a read-only storage medium (ROM) 534, or a random access storage medium (RAM) 536. Storage medium 530 also includes at least one set of instructions stored in the data storage device. Instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that perform the liveness detection methods provided in this specification.

[0059] At least one processor 520 can be communicatively connected to at least one storage medium 530 and a communication port 550 via an internal communication bus 510. The at least one processor 520 is used to execute the at least one instruction set described above. When the computing device 500 is running, the at least one processor 520 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the liveness detection method provided in this specification. The processor 520 can execute all the steps included in the liveness detection method. The processor 520 can be in the form of one or more processors. In some embodiments, the processor 520 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 520 is described in the computing device 500 of this specification. However, it should be noted that the computing device 500 in this specification may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by a single processor or by multiple processors in combination, as described in this specification. For example, if the processor 520 of the computing device 500 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 520 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0060] Figure 3A flowchart of a liveness detection method P100 according to an embodiment of this specification is shown. As previously described, the computing device 500 can execute the liveness detection method P100 of this specification. Specifically, the processor 520 can read an instruction set stored in its local storage medium and then execute the liveness detection method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0061] S110: Acquire multiple original modal images of the target object's region to be detected.

[0062] The target object is the object to be detected for liveness detection. When the target object is a target user, the body part to be detected can be a human body part from which biometric features can be extracted, such as the face, iris, or fingerprint.

[0063] In this context, multiple original modal images correspond to the images actually acquired in multiple modalities by the image acquisition module. The image acquisition module integrates camera modules with multiple modalities. Different modalities have different imaging characteristics and are subject to different external interference points. By acquiring images of the target area of ​​the object through multiple modal camera modules, original modal images corresponding to multiple modalities can be obtained. The original modality can be understood as the original modality of the image acquisition module; the original modal image is the same as the original modality of the image acquisition module, and the modality type has not changed. For example, the image acquisition module may include at least two of the following: RGB camera module, infrared camera module, NIR camera module, 3D camera module, and thermal imaging camera module. The original modal images of the RGB camera module, infrared camera module, NIR camera module, 3D camera module, and thermal imaging camera module are respectively RGB image, infrared image, NIR image, 3D image, and thermal imaging image.

[0064] S120: Input multiple original modal images into the first model to obtain the corresponding multiple original modal feature vectors.

[0065] The first model can be a liveness detection model based on multi-modal image fusion detection with multiple output terminals. Each output terminal corresponds to a modality and can output the original modality feature vector corresponding to that modality. The first model performs feature encoding on the original modality images of multiple modalities to obtain the original modality feature vectors corresponding to multiple modalities.

[0066] It should be understood that the original modal feature vector corresponds to the original modal image. By performing feature encoding on the original modal image acquired by the image acquisition module, the feature vector of the original modal image of the image acquisition module can be obtained.

[0067] There are several ways to input multiple original modal images into the first model to obtain the corresponding multiple original modal feature vectors, as follows:

[0068] For example, the processor 520 obtains feature maps of multiple modalities by performing multimodal feature encoding on multiple original modal images under multiple modalities, and then performs multi-head feature encoding on the feature maps of multiple modalities to determine the original modal feature vectors corresponding to multiple modalities. Here, multi-head feature encoding refers to performing feature encoding on the feature map corresponding to each modality in multiple modalities, thereby obtaining the original modal feature vector corresponding to each modality in multiple modalities.

[0069] Figure 4 A schematic diagram illustrating the principle of the liveness detection method provided in the embodiments of this specification. Figure 4 As shown, the first model 600 includes a multimodal feature encoding module 610 and a multi-head feature encoding module 630. The multimodal feature encoding module 610 can be a ResNet network, the feature fusion decision module 620 can be a transform block network, and the multi-head feature encoding module 630 can be a multi-layer perceptron (MLP) with multiple outputs. It should be understood that... Figure 4 The explanation of RGB mode, NIR mode and 3D mode as multiple modes is illustrative and can be exemplified by those skilled in the art. Other numbers and modes can be selected as multiple modes according to actual needs.

[0070] Please continue reading. Figure 4 The original modal images of RGB, NIR, and 3D modes are input into the multimodal feature encoding module 610. Feature encoding is performed on the original modal image corresponding to each of the RGB, NIR, and 3D modes, resulting in feature maps for each modality. These feature maps can be understood as data representing the distribution of a certain feature on the original modal image. After obtaining the feature maps of the RGB, NIR, and 3D modes, they are input into the multi-head feature encoding module 630. This multi-head feature encoding module 630 can be understood as including multiple parallel feature encoding branches, each corresponding to a different modality. It receives the feature map of the corresponding modality and performs feature encoding to obtain the original modal feature vectors corresponding to the RGB, NIR, and 3D modes.

[0071] S130: Based on multiple original mode feature vectors, determine at least one stable mode among multiple modes.

[0072] Among them, stable mode refers to the mode in which the stability of liveness detection performance in different data domains meets the preset requirements.

[0073] There are several ways to determine at least one stable mode among multiple modes based on multiple original mode feature vectors, as follows:

[0074] For example, processor 520 determines multiple comprehensive stability indices corresponding to multiple modes based on multiple original mode feature vectors. The comprehensive stability index corresponding to each mode in the multiple modes indicates whether each mode is a stable mode. By using the multiple comprehensive stability indices corresponding to multiple modes, at least one stable mode among the multiple modes is determined.

[0075] There are several ways to determine multiple comprehensive stability indices corresponding to multiple modes based on multiple original mode feature vectors, as follows:

[0076] For example, processor 520 denotes each of the multiple modalities as a target modality; and performs cross-modal transformation on the original modality feature vector corresponding to the target modality to generate at least one corresponding transmodal feature vector, wherein the original modality feature vector is the feature vector of the original modality image under the target modality, and the at least one transmodal feature vector corresponds to other modalities among the multiple modalities; and determines a first stability index corresponding to the target modality based on the similarity between the feature map of the original modality image and the feature map of the at least one transmodal image; and / or performs single-modal classification on the original modality feature vector corresponding to the target modality to obtain the liveness decision result corresponding to the target modality; and determines a second stability index corresponding to the target modality based on the similarity between the liveness decision result and the liveness decision results corresponding to other modalities among the multiple modalities. This approach can be further elaborated into the following three different methods:

[0077] For example, the processor 520, after designating each of the multiple modalities as the target modality, performs cross-modal transformation on the original modal feature vector corresponding to the target modality, generates at least one corresponding transmodal feature vector, and determines the first stability index corresponding to the target modality based on the similarity between the feature map of the original modal image and the feature map of at least one transmodal image.

[0078] For example, the processor 520 denotes each of the multiple modalities as the target modality, performs single-modality classification on the original modality feature vector corresponding to the target modality, obtains the liveness decision result corresponding to the target modality, and determines the second stability index corresponding to the target modality based on the similarity between the liveness decision result corresponding to the target modality and the liveness decision results corresponding to other modalities.

[0079] For example, the processor 520, after designating each of the multiple modalities as the target modality, performs cross-modal transformation on the original modal feature vector corresponding to the target modality to generate at least one corresponding transmodal feature vector. Based on the similarity between the feature map of the original modal image and the feature map of at least one transmodal image, it determines the first stability index corresponding to the target modality. It also performs single-modal classification on the original modal feature vector corresponding to the target modality to obtain the liveness decision result corresponding to the target modality. Based on the similarity between the liveness decision result corresponding to the target modality and the liveness decision results corresponding to other modalities, it determines the second stability index corresponding to the target modality.

[0080] Among the three implementation methods mentioned above, there are multiple ways to perform cross-modal transformation on the original modality feature vector corresponding to the target modality to generate at least one corresponding converted modality feature vector, and / or perform single-modal classification on the original modality feature vector corresponding to the target modality to obtain the liveness decision result corresponding to the target modality. Specifically, they can be as follows:

[0081] For example, the processor 520 inputs multiple original modal feature vectors corresponding to multiple modalities into the second model to obtain at least one modal transformation feature vector and / or liveness decision result corresponding to each modality in the multiple modalities.

[0082] Please continue reading. Figure 4 The second model 700 includes a cross-modal feature conversion module 710 and a single-modal classification module 720. The cross-modal feature conversion module 710 can be a multi-layer MLP network, and the single-modal classification module 720 can be a network with fully connected layers and softmax layers. For example, if the first model sets the multimodal input images as RGB images, NIR images, and 3D images, taking RGB mode as the target mode, cross-modal feature conversion for RGB mode involves converting the feature vectors corresponding to RGB images into feature vectors corresponding to NIR images and 3D images, respectively. Multiple original modal feature vectors corresponding to multiple modalities are input into the cross-modal feature conversion module 710 to perform cross-modal feature conversion on the original modal feature vectors corresponding to each modality, obtaining at least one converted modal feature vector for each modality. And / or, multiple original modal feature vectors corresponding to multiple modalities are input into the single-modal classification module 720 to perform liveness decision on the original modal feature vectors corresponding to each modality, obtaining the liveness decision result for each modality. For example, the single-modal classification module 720 can perform liveness decisions on the original modal feature vectors corresponding to RGB images, NIR images, and 3D images respectively, thereby obtaining the liveness decision results corresponding to RGB images, NIR images, and 3D images.

[0083] After determining the first stability index and the second stability index, the processor 520 can determine the comprehensive stability index corresponding to the target mode based on the first stability index and / or the second stability index. Specifically, determining the comprehensive stability index corresponding to the target mode based on the first stability index and / or the second stability index can be done in the following ways:

[0084] For example, processor 520 determines one of a first stability metric or a second stability metric as the overall stability metric. For example, the first stability metric may be used as the overall stability metric, or the second stability metric may be determined as the overall stability metric.

[0085] For example, the processor 520 determines the comprehensive stability index corresponding to the target mode based on the first stability index and the second stability index.

[0086] There are several ways to determine the comprehensive stability index corresponding to the target mode based on the first and second stability indices, as detailed below:

[0087] For example, if the processor 520 determines that the overall stability index is a stable mode when it determines that the stability indicated by the first stability index and the second stability index both meet the preset requirements.

[0088] For example, if the processor 520 determines that the overall stability index is an unstable mode if at least one of the first stability index and the second stability index indicates that the stability does not meet the preset requirements. For instance, if the stability indicated by the first stability index or the second stability index does not meet the preset requirements, or if neither the first stability index nor the second stability index meets the preset requirements, the overall stability index is determined to be an unstable mode.

[0089] The determination process of the first stability index and the second stability index will be explained below:

[0090] For ease of understanding, each mode to be transformed across modalities is denoted as the target mode. After obtaining at least one transformation feature vector corresponding to the target mode, the first stability index corresponding to the target mode can be determined by calculating the cosine similarity between the original mode feature vector corresponding to the target mode and each transformation feature vector.

[0091] There are several ways to determine the first stability index corresponding to the target mode based on the cosine similarity between the original mode feature vector corresponding to the target mode and each transformed mode feature vector. Specifically, they can be as follows:

[0092] For example, processor 520 converts at least one cosine similarity between the original modality feature vector corresponding to the target modality and at least one transformed modality feature vector into at least one first score, and obtains a first comprehensive score corresponding to the target modality based on the average of the weighted sum of the at least one first score, and performs one of operation 1 and operation 2 based on the first comprehensive score. Operation 1 includes determining that if the first comprehensive score corresponding to the target modality is less than a threshold, the first stability index corresponding to the target modality does not meet a preset requirement, i.e., the first stability index corresponding to the target modality indicates an unstable mode. Operation 2 includes determining that if the first comprehensive score corresponding to the target modality is greater than or equal to a threshold, the first stability index corresponding to the target modality meets a preset requirement, i.e., the first stability index corresponding to the target modality indicates a stable mode.

[0093] Taking multiple modalities, including RGB, NIR, and 3D modalities, with RGB modality as the target modality, as an example, this implementation obtains the feature vectors corresponding to the conversion of the RGB modality to the NIR modality and the feature vectors corresponding to the conversion to the 3D modality. Specifically, it calculates the cosine similarity between the feature vectors corresponding to the RGB modality and the converted NIR modality, and the cosine similarity between the feature vectors corresponding to the RGB modality and the converted 3D modality. After converting the two cosine similarities into a first score, the two first scores are weighted, summed, and averaged to obtain the first comprehensive score for the RGB modality.

[0094] The first score represents the feature consistency score between the original mode feature vector corresponding to the target mode and each transition mode feature vector, serving as an indicator to evaluate the cross-domain volatility of each mode. When the target mode corresponds to one transition mode feature vector, the first score between the feature vector corresponding to the target mode and the transition mode feature vector is determined as the first comprehensive score of the target mode. When the target mode corresponds to at least two transition mode feature vectors, the first comprehensive score of the target mode is determined by the average of the weighted sums of at least two first scores between the feature vector corresponding to the target mode and at least two transition mode feature vectors. When the first comprehensive score is below a threshold, it indicates that the stability of the target mode differs significantly from the stability of other modes, i.e., the stability of the target mode is low, and the first stability index of the target mode can be set as an indicator of an unstable mode. When the first comprehensive score is above or equal to the threshold, it indicates that the stability of the target mode differs little or no from the stability of other modes, i.e., the stability of the target mode is high, and the first stability index of the target mode can be set as an indicator of a stable mode.

[0095] For ease of understanding, each mode to be used for a liveness decision is referred to as the target mode. After obtaining the liveness decision result corresponding to the target mode, the second stability index corresponding to the target mode can be determined by calculating the similarity between the liveness decision result of the target mode and the liveness decision results of other modes.

[0096] Among them, based on the similarity between the liveness decision results of the target mode and the liveness decision results of other modes, there are multiple ways to determine the realization of the second stability index corresponding to the target mode, which can be as follows:

[0097] For example, processor 520 converts the similarity between the liveness decision result corresponding to the target modality and the liveness decision result corresponding to at least one other modality among multiple modalities into at least one second score, and obtains a second comprehensive score corresponding to the target modality based on the average of the weighted sum of the at least one second score, and performs one of operations 3 and 4 based on the second comprehensive score. Operation 3 includes: if the second comprehensive score corresponding to the target modality is less than a threshold, then it is determined that the second stability index corresponding to the target modality does not meet a preset requirement, i.e., the second stability index corresponding to the target modality indicates an unstable modality. Operation 4 includes: if the second comprehensive score corresponding to the target modality is greater than or equal to a threshold, then it is determined that the second stability index corresponding to the target modality meets a preset requirement, i.e., the second stability index corresponding to the target modality indicates a stable modality.

[0098] Taking a multi-modal approach, including RGB, NIR, and 3D modalities, with RGB being the target modality, this implementation obtains the liveness decision results for each modality: RGB, NIR, and 3D. The liveness decision results can be identified using "1" and "0" to represent the liveness of the attacker, respectively. During model computation, "1" and "0" can also be used as vectors for cosine similarity calculation, thus obtaining the similarity between the liveness decision results of the RGB modality and the NIR modality, as well as the similarity with the 3D modality. For example, if the liveness decision result for the RGB modality is identified as "1," and the liveness decision results for the NIR and 3D modalities are identified as "0" and "1," respectively, then the similarity between the RGB and NIR modalities is low (e.g., setting the second score to 0). The similarity between the RGB and 3D modalities is high (e.g., setting the score to 100), then the second comprehensive score for the RGB modality is 50 based on the weighted average of 0 and 100. Assuming the threshold is 50 points, the target mode is determined to be a stable mode.

[0099] The second score represents the consistency score between the prediction results of the target modality's live decision and those of at least one other modality among multiple modalities, serving as another indicator to assess the magnitude of cross-domain volatility for each modality. The second comprehensive score can be understood as the average prediction consistency score of the target modality obtained by calculating the pairwise similarity among multiple modalities. When this average prediction consistency score is less than a threshold, it indicates that the stability of the target modality differs significantly from the stability of other modalities, classifying it as an unstable modality. Conversely, when the average prediction consistency score is greater than or equal to the threshold, it indicates that the stability of the target modality is not different from the stability of other modalities, classifying it as a stable modality. Specifically, when the second comprehensive score is below the threshold, it indicates low stability of the target modality, and the second stability index of the target modality can be set to indicate an unstable modality. When the second comprehensive score is greater than or equal to the threshold, it indicates high stability of the target modality, and the second stability index of the target modality can be set to indicate a stable modality.

[0100] S140: Based on at least one stable mode, perform liveness detection on the target object to obtain a liveness detection result indicating whether the target object is a live object.

[0101] In some embodiments, inputting multiple original modal images into the first model can also obtain corresponding initial multimodal liveness decision results.

[0102] There are several ways to input multiple original modal images into the first model to obtain the corresponding initial multimodal liveness decision results, as follows:

[0103] For example, the processor 520 obtains a multimodal feature map by encoding multiple original modal images under multiple modalities through multimodal feature encoding, and performs feature fusion decision on the multimodal feature map to determine the initial multimodal liveness decision result corresponding to the multiple original modal images.

[0104] Please continue reading. Figure 4 The first model also includes a feature fusion decision module 620. After inputting the original modal images of RGB, NIR and 3D modalities into the multimodal feature encoding module 610 to obtain the feature maps corresponding to the original modal images of RGB, NIR and 3D modalities, the feature maps corresponding to the original modal images of RGB, NIR and 3D modalities can also be input into the feature fusion decision module 620 to perform feature fusion and decision based on the feature maps corresponding to the original modal images of RGB, NIR and 3D modalities, and obtain an initial multimodal liveness decision result. This initial multimodal liveness decision result indicates whether the target object is a live object.

[0105] After obtaining the initial multimodal liveness decision results, if there are no unstable modes among the multiple modes, the initial multimodal liveness decision results can be directly output as the final liveness decision results.

[0106] After obtaining the initial multimodal liveness decision result, if there is an unstable mode among the multiple modes, it is necessary to perturb the original modal image of the unstable mode and combine it with the original modal image of at least one stable mode to generate an updated multimodal input image. The updated multimodal input image is then fed back to the input of the first model to obtain the updated multimodal liveness decision result corresponding to the updated multimodal input image, and the updated multimodal liveness decision result is output as the final decision result.

[0107] There are several ways to perturb the original modality image of an unstable modality, including replacing it with zeros or reducing its quality. Data perturbation aims to transform the unstable modality into a missing modality for liveness detection. By perturbing the data, the influence of the unstable modality image on liveness detection performance can be suppressed, thereby enabling the liveness detection model to exhibit stable performance across different data domains.

[0108] The processor 520 feeds back the updated multimodal input image to the input of the first model to obtain the updated multimodal liveness decision result corresponding to the updated multimodal input image. There are several ways to achieve this. For example, the updated multimodal input image is fed back to the input of the first model to obtain the updated attack probability p2 corresponding to the updated multimodal input image. When p is greater than the threshold T, it is determined to be an attack target. When p2 is less than or equal to the threshold T, it is determined to be a liveness target.

[0109] The processor 520 can input multiple original modal images into the first model to obtain the initial multimodal liveness decision result in several ways, such as as follows: For example, multiple original modal images under multiple modalities can be input into the first model to obtain the attack probability p1 corresponding to the multimodal input image. When p1 is greater than the threshold T, it is determined to be an attack target. When p1 is less than or equal to the threshold T, it is determined to be a liveness target.

[0110] The above embodiments describe how to use the first model and the second model. Before using the first model and the second model, they need to be trained. The training process of the first model and the second model will be described below.

[0111] Figure 5A flowchart of a training method P200 for a first model according to an embodiment of this specification is shown. As previously described, the computing device 500 can execute the training method P200 for the first model of this specification. Specifically, the processor 520 can read an instruction set stored in its local storage medium and then execute the training method P200 for the first model of this specification according to the instructions in the instruction set. Figure 5 As shown, method P200 may include:

[0112] S210: Obtain the first training image set for multiple modalities.

[0113] There are several ways to obtain the first training image set for multiple modalities, as follows:

[0114] For example, after the processor 520 acquires the original training image set for multiple modalities, it generates random numbers for each modality and, based on the random numbers from the multiple modalities, selects to execute either a first operation or a second operation. The first operation includes: if all random numbers from the multiple modalities are greater than or equal to a preset value, then the original training image set for the multiple modalities is determined as the first training image set for the multiple modalities. The second operation includes: if the random number for at least one modality is less than the preset value, then the original training image set for at least one modality is perturbed to obtain at least one perturbed training image set corresponding to at least one modality, and this perturbed training image set is combined with the original training image set of the unperturbed modality to generate the first training image set for the multiple modalities.

[0115] The processor 520 can generate random numbers for each mode using a random number generation method. For ease of subsequent comparison, the random number can also be a random number between 0 and 100.

[0116] Here, the original training image set for at least one modality is perturbed to set at least one modality to a missing state. For example, when the random number for the RGB modality is less than 25, the training images in the original training image set for the RGB modality are perturbed to obtain the perturbed training image set corresponding to the RGB modality.

[0117] S220: Iteratively train based on the first training image set of multiple modalities until the first comprehensive loss function is less than the preset value.

[0118] When the first comprehensive loss function is less than the preset value, it means that the first comprehensive loss function has been minimized. At this time, the iterative training ends, and the final first model can be obtained based on the parameter values ​​of the first model at the end of training.

[0119] Iterative training includes:

[0120] S221: Update the parameter values ​​of the first model based on the first comprehensive loss function determined in the previous training iteration.

[0121] The first comprehensive loss function is a weighted sum of the first loss function and the second loss function. The first comprehensive loss function can be expressed as follows: (1)

[0122] Loss = Loss1 + Loss2; (1)

[0123] In equation (1), Loss is the first comprehensive loss function, Loss1 is the first loss function, and Loss2 is the second loss function.

[0124] The difference between Loss1 and Loss2 lies in that Loss1 is a classification loss function applied to the multimodal fusion decision result, while Loss2 is a classification loss function applied to the single-modal decision result for each of the multiple modalities. When updating the parameter values ​​of the first model based on the first comprehensive loss function determined in the previous training iteration, backpropagation is performed based on Loss1 to update the parameter values ​​of the feature fusion network and the first feature extraction network, and backpropagation is performed based on the classification loss function of each modality in Loss2 to update the parameter values ​​of the second feature extraction network and the first feature extraction network. After the feature fusion network is trained, it becomes... Figure 4 The feature fusion decision module in the middle. After the first feature extraction network is trained, it becomes... Figure 4 The multimodal feature encoding module in the middle. The second feature extraction network, after being trained, becomes... Figure 4 The first part is the multi-head feature encoding module. The second feature extraction network can be understood as including multiple feature encoding branch networks. Each feature encoding branch network accepts the feature map of a mode for feature extraction and performs backpropagation according to the classification loss function of each mode to update the parameter values ​​of the corresponding feature encoding branch network.

[0125] For example, if multiple modalities include RGB, NIR, and 3D modalities, the second feature extraction network can be understood as a network structure comprising RGB feature encoding branches, NIR feature encoding branches, and 3D feature encoding branches. The RGB feature encoding branch performs feature extraction and liveness detection on the feature maps of the RGB modality, obtaining the corresponding feature vector and liveness detection result for the RGB modality. Similarly, the NIR feature encoding branch performs feature extraction and liveness detection on the feature maps of the NIR modality, obtaining the corresponding feature vector and liveness detection result for the NIR modality. Finally, the 3D feature encoding branch performs feature extraction and liveness detection on the feature maps of the 3D modality, obtaining the corresponding feature vector and liveness detection result for the 3D modality. Then, during backpropagation, the parameter values ​​of the RGB modality feature encoding branch network and the first feature extraction network are updated according to the classification loss function corresponding to the RGB modality; the parameter values ​​of the NIR modality feature encoding branch network and the first feature extraction network are updated according to the classification loss function corresponding to the NIR modality; and the parameter values ​​of the 3D modality feature encoding branch network and the first feature extraction network are updated according to the classification loss function corresponding to the 3D modality.

[0126] It's important to note that the classification loss function for the mode with missing states is not backpropagated. Using the example above, assuming the RGB mode, NIR mode, and 3D mode are missing in the current training iteration, the classification loss functions for the NIR and 3D modes are backpropagated, but the classification loss function for the RGB mode is not.

[0127] After updating the parameter values ​​of the first model, training can continue based on the updated first model. The training includes the following steps S222, S223 and S224.

[0128] S222: Input the first training image set of multiple modalities into the first feature extraction network to determine the feature map corresponding to the first training image set of multiple modalities.

[0129] The first feature extraction network can be a residual network, such as ResNet50 or ResNet18, which are network structures used for feature extraction.

[0130] The first training image set for each modality includes multiple training images. The first feature extraction network extracts features from the training images in the first training image set for each modality to obtain the feature map corresponding to each training image in the first training image set for each modality. For example, if the first training image set for the RGB modality includes 50 training images, then feature extraction is performed on each of the 50 training images for the RGB modality to obtain the feature map corresponding to each training image in the first training image set for the RGB modality.

[0131] S223: Input the feature maps corresponding to the first training image set of multiple modalities into the feature fusion network to determine the initial liveness decision result and its corresponding first loss function.

[0132] The feature fusion network can be a transform block network, that is... Figure 4 The feature fusion decision module in the middle.

[0133] Specifically, the feature fusion network performs feature fusion decisions based on the feature maps corresponding to training images in the first training image set across multiple modalities, thereby obtaining an initial liveness decision result. Then, based on the difference between the initial liveness decision result and the labels corresponding to the training images in the first training image set, a classification loss function, i.e., the first loss function Loss1, is determined. Here, the label indicates whether the training object in the training image is a live object.

[0134] S224: Input the feature maps corresponding to the first training image set of multiple modalities into the second feature extraction network to determine the feature vectors corresponding to the multiple modalities and their corresponding second loss functions.

[0135] The second feature extraction network can be an artificial neural network (ANN), such as a multilayer MLP.

[0136] The second feature extraction network extracts features based on the feature maps corresponding to the training images in the first training image set of each modality in multiple modalities, thereby obtaining the feature vectors corresponding to the training images in the first training image set of each modality in multiple modalities. It also performs liveness decision based on the feature maps corresponding to the training images in the first training image set of each modality in multiple modalities, thereby obtaining the liveness decision training result for each modality, i.e., the liveness classification result for each modality.

[0137] It should be noted that the second loss function here refers to the loss function corresponding to each modality among multiple modalities. The loss function corresponding to each modality is determined based on the difference between the liveness decision training result of each modality and the label corresponding to the training image in the first training image set of each modality. The loss function corresponding to each modality can also be understood as a classification loss function. Here, the label indicates whether the training object in the training image is a live object.

[0138] It is worth noting that, in some embodiments, when iteratively training based on a first training image set of multiple modalities, the training termination condition may also be that the number of training iterations reaches a preset number.

[0139] Figure 6A flowchart of a training method P300 for a second model according to an embodiment of this specification is shown. As previously described, the computing device 500 can execute the training method P300 for the second model of this specification. Specifically, the processor 520 can read the instruction set stored in its local storage medium and then execute the training method P300 for the second model of this specification according to the instructions in the instruction set. Figure 6 As shown, method P300 may include:

[0140] S310: Obtain a second training image set with multiple modalities.

[0141] Here, the second training image set for multiple modalities and the first training image set for multiple modalities in step S210 can be the same training image set including the same training images, or they can be two training image sets including completely different training images, or they can be two training image sets with partially overlapping training images. The embodiments in this specification do not limit this.

[0142] S320: Iterate the training based on the second training image set with multiple modalities until the training termination condition is met to obtain the second model.

[0143] The training termination conditions include: the second comprehensive loss function reaching a preset value, or the training iterations reaching a preset number. The second comprehensive loss function includes a third loss function and a fourth loss function, which can be specifically expressed as the following formula (2):

[0144] Loss' = Loss3 + Loss4; (2)

[0145] In equation (2), Loss' is the second comprehensive loss function, Loss3 is the third loss function, and Loss4 is the fourth loss function.

[0146] Iterative training includes:

[0147] S321: Update the parameter values ​​of the second model based on the second comprehensive loss function determined in the previous training iteration.

[0148] S322: Input the second training image set of multiple modalities into the trained first model to obtain the original modal feature vectors corresponding to the training images in the second training image set of multiple modalities.

[0149] S323: For each of the multiple modes, denoted as the target mode.

[0150] S324: Input the original modality feature vector corresponding to the training image of the target modality in the second training image set into the cross-modality feature transformation network to determine at least one modality-transformation feature vector corresponding to the target modality and its corresponding third loss function, wherein at least one modality-transformation feature vector corresponds to other modalities among multiple modalities.

[0151] The cross-modal feature transformation network can be a multi-layer perceptron (MLP). Once trained, the cross-modal feature transformation network becomes... Figure 4 The cross-modal feature transformation module in the network performs cross-modal feature transformation based on the original modality feature vector corresponding to the training image of the target modality, obtaining at least one transformed modality feature vector corresponding to the target modality. The network also determines the cross-modal feature transformation loss function (the third loss function) for the target modality by averaging the weighted sums of the L2 distances between the original modality feature vector and the at least one transformed modality feature vector. Then, backpropagation is performed based on the third loss function to update the parameter values ​​of the cross-modal feature transformation network.

[0152] S325: Input the original modality feature vector corresponding to the training image of the target modality in the second training image set into the single-modality classification network to determine the liveness decision result corresponding to the target modality and its corresponding fourth loss function.

[0153] The unimodal classification network comprises multiple unimodal classifiers, each of which can be a network structure consisting of fully connected layers and a softmax layer. Once trained, this unimodal classification network becomes... Figure 4 The single-modal classification module in the system.

[0154] Taking RGB, NIR, and 3D modalities as examples, a single-modal classification network includes a first modality classifier, a second modality classifier, and a third modality classifier. The original modality feature vector corresponding to the training image of the RGB modality is input into the first modality classifier for liveness detection, obtaining the liveness detection result for the RGB modality. Based on the difference between the liveness detection result and the label for the RGB modality, the classification loss function for the RGB modality can be determined. Then, backpropagation is performed based on the classification loss function for the RGB modality to update the parameter values ​​of the first modality classifier. The processing steps for the NIR and 3D modalities are similar to those for the RGB modality; please refer to the RGB modality processing steps for details, which will not be elaborated here.

[0155] In this model, the cross-modal feature transformation network and the unimodal classification network are adjusted simultaneously. This means that during each training iteration, the parameter values ​​of both the cross-modal feature transformation network and the unimodal classification network need to be updated separately. Training ends when the second comprehensive loss function is minimized or when the preset number of training iterations is reached, resulting in the second model.

[0156] It is understandable that in the trained second model, the cross-modal feature transformation network aims to determine the first stability metric, and the unimodal classification network aims to determine the second stability metric. As described above, the comprehensive stability metric can be determined independently based on either the first or second stability metric. Therefore, the training process here can also involve iteratively training the cross-modal feature transformation network or the unimodal classification network. Correspondingly, the second comprehensive loss function is either the third or fourth loss function, and the remaining iterative training process can be found in the descriptions in S324 or S325 above.

[0157] It should be understood that the training method for the first model can be performed by a computing device other than computing device 500, and the training method for the second model can also be performed by a computing device other than computing device 500. After the first and second models are trained, they are deployed on computing device 500 for liveness detection.

[0158] In summary, the liveness detection method P100 and system 001 provided in this specification, after acquiring multiple original modal images of the target object's detection region, input these images into a first model to obtain corresponding original modal feature vectors. Based on these feature vectors, at least one stable modality is determined from among the multiple modalities. This stable modality refers to a modality whose liveness detection performance stability meets preset requirements across different data domains. Based on this at least one stable modality, liveness detection is performed on the target object to obtain a liveness detection result indicating whether the target object is alive. Since this scheme determines at least one stable modality from multiple modalities and performs liveness detection on the target object based on this stable modality, discarding unstable modalities, it avoids the impact of unstable modalities on liveness detection performance in cross-domain scenarios. The stable modality determines the liveness detection performance, making the liveness detection performance more stable when the data domain of the training dataset differs from that of the application.

[0159] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing liveness detection. When executed by a processor, the executable instructions instruct the processor to implement the steps of the liveness detection method P100 of this specification. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on a computing device 500, the program code causes the computing device 500 to perform the steps of the liveness detection method P100 described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing device 500. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 500, partially on computing device 500, as a standalone software package, partially on computing device 500 and partially on a remote computing device, or entirely on a remote computing device.

[0160] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0161] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure may be presented by way of example only and may not be restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0162] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0163] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0164] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0165] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting liveness, comprising: Multiple original modal images of the target object to be detected are acquired, and the multiple original modal images correspond to the images actually acquired in multiple modalities of the image acquisition module; The multiple original modal images are input into the first model to obtain the corresponding multiple original modal feature vectors and the initial multimodal liveness decision results; Based on the multiple original modal feature vectors, at least one stable modality among the multiple modalities is determined, and it is determined whether there is an unstable modality among the multiple modalities. The stable modality refers to the modality whose stability of liveness detection performance for different data domains meets the preset requirements. as well as If the unstable mode exists, the original modal image of the unstable mode is perturbed and combined with the original modal image of at least one stable mode to generate an updated multimodal input image. The final decision result is determined based on the updated multimodal input image. If the unstable mode does not exist, the initial multimodal liveness decision result is output as the final decision result.

2. The method according to claim 1, wherein, The final decision result output based on the updated multimodal input image includes: The updated multimodal input image is fed back to the input of the first model to obtain the updated multimodal liveness decision result corresponding to the updated multimodal input image; and The updated multimodal liveness decision result is output as the final decision result.

3. The method according to claim 2, wherein, The data perturbation of the original mode image of the unstable mode among the multiple modes includes: Replace the original mode image of the unstable mode with 0; or The image quality of the original mode image of the unstable mode is reduced.

4. The method according to claim 1, wherein, The step of inputting the multiple original modal images into the first model to obtain the corresponding multiple original modal feature vectors and the initial multimodal liveness decision results includes: The feature maps of the multiple modalities are obtained by performing multimodal feature encoding on the original modal images of the multiple modalities; and Feature fusion decision is performed on the feature maps of the multiple modalities to determine the initial multimodal liveness decision result corresponding to the multiple original modal images.

5. The method according to claim 1, wherein, The step of inputting the plurality of original modal images into the first model to obtain the corresponding plurality of original modal feature vectors includes: The feature maps of the multiple modalities are obtained by performing multimodal feature encoding on the original modal images of the multiple modalities; and Multi-head feature encoding is performed on the feature maps of the multiple modalities to determine the original modal feature vectors corresponding to the multiple modalities.

6. The method according to claim 1, wherein, Determining at least one stable mode among the plurality of modes includes: determining a plurality of comprehensive stability indices corresponding to the plurality of modes, wherein each comprehensive stability index indicates whether each mode is a stable mode.

7. The method according to claim 6, wherein determining the plurality of comprehensive stability indices corresponding to the plurality of modes includes: Each of the plurality of modes is denoted as the target mode; as well as Perform cross-modal transformation on the original modality feature vector corresponding to the target modality to generate at least one corresponding transmodal feature vector, wherein the at least one transmodal feature vector corresponds to other modalities among the plurality of modalities; and determine a first stability index corresponding to the target modality based on the similarity between the feature map of the original modality image and the feature map of the at least one transmodal image; and / or Perform unimodal classification on the original modality feature vector corresponding to the target modality to obtain the liveness decision result corresponding to the target modality; And based on the similarity between the live decision result and the live decision results corresponding to other modes among the plurality of modes, a second stability index corresponding to the target mode is determined.

8. The method according to claim 7, wherein determining the plurality of comprehensive stability indices corresponding to the plurality of modes includes, The first stability index or the second stability index is determined as the comprehensive stability index.

9. The method according to claim 7, wherein determining the plurality of comprehensive stability indices corresponding to the plurality of modes includes determining the comprehensive stability index corresponding to the target mode based on the first stability index and the second stability index.

10. The method according to claim 9, wherein determining the comprehensive stability index based on the first stability index and the second stability index comprises: If the stability indicated by the first stability index and the second stability index both meet the preset requirements, then the comprehensive stability index is determined to be a stable mode.

11. The method according to claim 9, wherein determining the comprehensive stability index based on the first stability index and the second stability index comprises: If the stability indicated by at least one of the first stability index and the second stability index does not meet the preset requirements, then the comprehensive stability index is determined to be an unstable mode.

12. The method according to claim 1, wherein, The first model includes a first feature extraction network, a feature fusion network, and a second feature extraction network; The first model was trained using the following method: Obtain the first training image set for the multiple modalities; Training iterations are performed based on the first training image set of the multiple modalities until the first comprehensive loss function is less than a preset value. The training iterations include: The parameter values ​​of the first model are updated based on the first comprehensive loss function determined in the previous training iteration; The first training image set of the multiple modalities is input into the first feature extraction network to determine the feature map corresponding to the first training image set of the multiple modalities; The feature maps corresponding to the first training image set of the multiple modalities are input into the feature fusion network to determine the initial liveness decision result and its corresponding first loss function. The feature maps corresponding to the first training image set of the multiple modalities are input into the second feature extraction network to determine the feature vectors corresponding to the multiple modalities and their corresponding second loss functions; Wherein, the first comprehensive loss function is a weighted sum of the first loss function and the second loss function.

13. The method according to claim 12, wherein, The step of obtaining the training image set for the multiple modalities includes: Obtain the original training image set for the multiple modalities; For each of the multiple modes, generate a random number to obtain the random number for the multiple modes; and Based on the random numbers from the multiple modalities, one of the operations, either the first or the second, is selected for execution, wherein... The first operation includes determining the original training image set of the multiple modalities as the first training image set of the multiple modalities if the random numbers of the multiple modalities are all greater than or equal to a preset value. The second operation includes: if the random number of at least one of the multiple modalities is less than a preset value, then the original training image set of the at least one modality is perturbed to obtain at least one perturbed training image set corresponding to at least one modality, and combined with the original training image set corresponding to the unperturbed modality to generate a first training image set of the multiple modalities.

14. A liveness detection system, comprising: At least one storage medium storing at least one instruction set for performing liveness detection; as well as At least one processor is communicatively connected to the at least one storage medium. When the liveness detection device is running, the at least one processor reads the at least one instruction set and executes the method of any one of claims 1-13 according to the instructions of the at least one instruction set.

Images