A multi-modal living body detection model training method and a multi-modal living body detection method

Abstract

The application provides a multi-modal living body detection model training method, comprising: acquiring a sample channel image corresponding to a sample face; performing random shielding and no shielding processing on the sample channel image to obtain a corresponding target format image and label a sample confidence corresponding to the target format image; acquiring a sample training set according to the target format image and the sample confidence corresponding thereto, and training a preset living body detection neural network by using the sample training set to obtain a multi-modal living body detection model. The application trains the model by using a single-modal format image and a multi-modal format image to obtain a multi-modal unified model, can flexibly perform living body detection on the single-modal format image and the multi-modal format image, can be flexibly deployed for single-modal or multi-modal living body detection when applied to a product, greatly reduces the workload of maintaining multiple models, and improves the accuracy and stability of living body detection.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a multimodal liveness detection model training method and a multimodal liveness detection method. Background Technology

[0002] Liveness detection refers to methods for determining the true physiological characteristics of an individual in identity verification scenarios. It typically involves detecting combinations of actions such as blinking, opening the mouth, shaking the head, and nodding, using technologies like facial landmark localization and face tracking to verify whether the user is a real, living person. It is particularly important in facial recognition applications. Current technologies usually employ neural network models for liveness detection. However, during training, a fixed modality of samples is often used to train the initial model, resulting in a liveness detection neural network model that can only detect data of that fixed modality. This lack of flexibility in adapting to various scenarios leads to relatively low accuracy and stability in liveness detection. Summary of the Invention

[0003] This application provides a multimodal liveness detection model training method and a multimodal liveness detection method, which can solve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a method for training a multimodal liveness detection model, including acquiring sample channel images corresponding to sample faces; wherein, the sample channel images are obtained by stitching together a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image; performing random masking and unmasking processing on the sample channel images to obtain corresponding target format images and labeling the sample confidence levels corresponding to the target format images; wherein, the target format images include single-modal format images and multimodal format images, the single-modal format image is any channel image in the sample channel images, and the multimodal format image is a combination of multiple channel images in the sample images; obtaining a sample training set based on the target format images and their corresponding sample confidence levels, and using the sample training set to train a preset liveness detection neural network to obtain a multimodal liveness detection model.

[0005] Secondly, embodiments of this application provide a multimodal liveness detection method, including acquiring a face image to be identified, wherein the face image to be identified may include any one or more combinations of grayscale images, infrared images, or depth images; inputting the face image to be identified into a multimodal liveness detection model for detection processing to obtain a liveness detection result corresponding to the face image to be identified; wherein the multimodal liveness detection model is trained by the multimodal liveness detection model training method of the first aspect.

[0006] Thirdly, embodiments of this application provide a multimodal liveness detection model training device, comprising: an image acquisition unit for acquiring sample channel images corresponding to sample faces, wherein the sample channel images are obtained by stitching together at least a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image; a masking and annotation unit for randomly masking and not masking the sample channel images to obtain corresponding target format images and annotating the sample confidence scores corresponding to the target format images; wherein the target format images include single-modal format images and multimodal format images, the single-modal format image being any channel image in the sample channel images, and the multimodal format images being a combination of multiple channel images in the sample images; a training set acquisition unit for acquiring a sample training set based on the target format images and their corresponding sample confidence scores; and a training unit for training a preset liveness detection neural network based on the sample training set to obtain a multimodal liveness detection model.

[0007] Fourthly, embodiments of this application provide a multimodal liveness detection device, comprising: an acquisition unit for acquiring a face image to be identified; the face image to be identified may include any one or more of grayscale images, infrared images, or depth images; and a detection unit for inputting the face image to be identified into a multimodal liveness detection model for detection processing to obtain a liveness detection result corresponding to the face image to be identified; wherein the multimodal liveness detection model is trained using the multimodal liveness detection model training method of the first aspect.

[0008] Fifthly, embodiments of this application provide a multimodal liveness detection model training device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above or the method described in the second aspect above.

[0009] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0010] Compared with existing technologies, the embodiments of this application use single-modal format images and multi-modal format images to train the model, thereby obtaining a unified multi-modal model. This model can flexibly perform liveness detection on single-modal format images and multi-modal format images. When applied to products, it can be flexibly deployed according to single-modal or multi-modal models, which greatly reduces the workload of maintaining multiple models and improves the accuracy and stability of liveness detection. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a multimodal liveness detection model training method provided in the first embodiment of this application;

[0013] Figure 2 This is a schematic flowchart of a multimodal liveness detection method provided in the second embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the multimodal liveness detection model training device provided in the third embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the multimodal liveness detection device provided in the fourth embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the electronic device provided in the fifth embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0018] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0019] It should also be understood that the term "and / or" as used in this application specification and the appended claims refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations. Furthermore, in the description of this application specification and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0020] Figure 1This is a schematic flowchart illustrating a multimodal liveness detection model training method provided in this application embodiment. In this embodiment, the execution subject of the multimodal liveness detection model training method is an electronic device with multimodal liveness detection model training capabilities, such as a desktop computer, server, etc. Figure 1 The multimodal liveness detection model training method shown may include:

[0021] S101: Obtain the sample channel image corresponding to the sample face.

[0022] In this embodiment, the training of the multimodal liveness detection model primarily involves multimodal processing of the training samples. When acquiring samples, it is necessary to obtain sample channel images containing the sample faces. It is understood that when acquiring sample faces, faces from different races, ages, and genders can be obtained, thus acquiring sample channel images of different races, ages, and genders, improving the accuracy of liveness detection. The sample channel images are multi-channel images, and different channels can correspond to images of different modalities. Preferably, the sample channel images can be three-channel images, with the three channels representing grayscale, infrared, and depth images, respectively.

[0023] In some implementations, obtaining the sample channel image corresponding to the sample face may include: obtaining an initial sample color image and identifying the sample face from the initial sample color image; obtaining a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image corresponding to the sample face; and stitching the single-channel grayscale image, the single-channel infrared image, and the single-channel depth image together in the channel dimension to obtain the sample channel image corresponding to the sample face.

[0024] In one embodiment, the initial sample color image is an RGB image including a face. A pre-trained face recognition model is used to recognize the initial sample color image to obtain the sample face. Furthermore, after obtaining the sample face, it can be cropped based on its coordinate information to obtain an RGB image containing only the sample face, thereby reducing computational load and improving efficiency. It should be noted that this embodiment does not impose specific limitations on the face recognition model; it only needs to be able to recognize the sample face in the initial sample color image.

[0025] In one embodiment, obtaining the single-channel grayscale image, single-channel infrared image, and single-channel depth image corresponding to the sample face specifically includes: when obtaining the initial sample color image, simultaneously obtaining the initial infrared image and initial depth image corresponding to the aligned initial sample color image; after identifying the sample face in the initial sample color image, cropping the initial infrared image according to the position and size information of the sample face to obtain the single-channel infrared image corresponding to the sample face, and simultaneously cropping the initial depth image to obtain the single-channel depth image corresponding to the sample face; finally, converting the initial sample color image into an initial grayscale image, and obtaining the single-channel grayscale image corresponding to the sample face from the initial grayscale image.

[0026] Specifically, the formula for converting a grayscale image can be:

[0027] Gray=(R*299+G*587+B*114+500) / 1000

[0028] Where Gray represents the grayscale value, and R, G, and B represent the color values. Compared to directly using the initial sample color image for training, converting the initial color image to a single-channel grayscale image can remove information redundancy, reduce computational load, and improve training efficiency.

[0029] In one embodiment, a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image are stitched together along the channel dimension to obtain a sample channel image corresponding to the sample face. Preferably, the single-channel grayscale image, single-channel infrared image, and single-channel depth image have the same size, such as 112×112. Stitching images of the same size along the channel dimension yields a sample channel image of 3×112×112. It should be noted that by stitching traditional single-channel images into multi-channel images, not only are the common and different information of each face in the image preserved, but the relative spatial positions of the information between each face are also accurately preserved. This allows the network to be trained to extract the difference information between different channels and learn abstract information with relative relationships, thereby increasing the network's generalization ability.

[0030] S102: Randomly mask the sample channel images and do not mask them at all to obtain the corresponding target format images, and label the sample confidence levels corresponding to the target format images; wherein, the target format images include unimodal format images and multimodal format images.

[0031] In one embodiment, a target format image is obtained by randomly masking the sample channel image; that is, by randomly masking any or multiple channels of the sample channel image, a target format image is obtained after random masking. The target format image includes both single-modal and multi-modal format images. In other words, the target format image is not limited to a single format; it needs to encompass multiple modal formats. For example, when the target format image includes a single-modal format image, it can be any one of a single-channel grayscale image, a single-channel infrared image, or a single-channel depth image. When the target format image includes a multi-modal format image, it can be an image composed of a single-channel grayscale image and a single-channel infrared image, or an image composed of a single-channel grayscale image and a single-channel depth image, and so on.

[0032] In one implementation, the random masking process specifically includes: setting one or two of the pixels of a single-channel grayscale image, a single-channel infrared image, or a single-channel depth image in a channel of the sample channel image to 0, thereby obtaining a target format image corresponding to the sample channel image. Specifically, when setting the pixels of a single-channel grayscale image in a channel of the sample channel image to 0, the target format image corresponding to the sample channel image is an image composed of a single-channel infrared image and a single-channel depth image; when setting the pixels of a single-channel grayscale image and a single-channel infrared image in a channel of the sample channel image to 0, the target format image corresponding to the sample channel image is a single-channel depth image.

[0033] In another implementation, the device uses the sample channel image as the target format image corresponding to the sample channel image, that is, no masking is performed on any modality, and the target format image of all modal combinations is obtained.

[0034] Furthermore, it is necessary to obtain the sample confidence scores corresponding to the target format images, where the sample confidence scores are pre-labeled. Each target format image corresponds to a sample confidence score, which identifies the liveness detection result corresponding to the target format image. The confidence score can be labeled based on whether the face corresponding to each modality image in the target format image is real. For example, the sample confidence score is a value between 0 and 1. When the sample confidence score is greater than 0.5, the liveness detection result is identified as a real person; when the sample confidence score is less than or equal to 0.5, the liveness detection result is identified as a fake person.

[0035] S103: Obtain the sample training set based on the target format image and its corresponding sample confidence.

[0036] Specifically, the target format image and its corresponding sample confidence score are stored together. Each target format image and its corresponding sample confidence score is considered a sample, and multiple samples are stored to form a sample training set. It is understandable that the more numerous and diverse the samples in the sample training set, the higher the accuracy of the detection results of the trained model.

[0037] S104: Train the preset liveness detection neural network using the sample training set to obtain a multimodal liveness detection model.

[0038] In one embodiment, a preset liveness detection neural network can be pre-stored. This preset liveness detection neural network is used to train the desired multimodal liveness detection model. Generally, the preset liveness detection neural network has a simple structure but cannot obtain accurate liveness detection results. It needs to be trained with a sample training set to obtain a multimodal liveness detection model that can obtain accurate detection results.

[0039] Specifically, the training process includes: inputting the target format image into a preset liveness detection neural network to obtain the target confidence score, and calculating the target loss value based on the target confidence score, sample confidence score, and preset loss function; if the target loss value does not meet the preset termination condition, updating the preset liveness detection neural network based on the target loss value and returning to the previous step of inputting the target format image into the preset liveness detection neural network for processing to obtain the target confidence score; if the target loss value meets the preset termination condition, outputting the trained multimodal liveness detection model. It should be noted that the preset loss function can be the cross-entropy loss function, and there are no restrictions here.

[0040] In one implementation, to better learn the features of real and fake objects during the training process and obtain a high-precision multimodal liveness detection model, this embodiment pre-sets a feature fusion module in the liveness detection neural network to perform multi-scale feature fusion on the learned features. Specifically, the target format image is input into a multi-layered cascaded feature extraction module. Based on each layer of the feature extraction module, at least a first feature information, a second feature information, and a third feature information can be obtained sequentially. The first, second, and third feature information have different sizes, corresponding to low, medium, and high-dimensional feature information, respectively. The first and second feature information are downsampled to obtain a fourth and a fifth feature information, so that the size of the low and medium-dimensional feature maps is the same as the high-dimensional size. The third, fourth, and fifth feature information are input into the feature fusion module, and splicing is performed in the channel dimension to obtain fused feature information. The target confidence is determined based on the fused feature information, thereby obtaining the liveness detection result.

[0041] In one embodiment, each feature extraction module includes at least multiple convolutional layers and at least one pooling layer. The more convolutional layers, the richer the extracted features, and the higher the completeness of the features extracted by the neural network. Furthermore, after the target format image undergoes convolution in the convolutional layers, it also needs to be reduced by the pooling layer to obtain the corresponding feature information, thereby reducing the computational load on the image.

[0042] Figure 2 This is a schematic flowchart illustrating a multimodal liveness detection method provided in the second embodiment of this application. In this embodiment, the executing entity of the multimodal liveness detection method is an electronic device with multimodal liveness detection functionality, such as a door lock system or a facial recognition payment system. In one embodiment, the liveness detection method includes:

[0043] S201: Obtain the face image to be identified; wherein, the face image to be identified includes a target unimodal image or a target multimodal image.

[0044] In one embodiment, the target single-modal image includes any one of a target grayscale image, a target infrared image, or a target depth image. The target grayscale image can be a single-channel target grayscale image, the target infrared image can be a single-channel target infrared image, and the target depth image can be a single-channel target depth image.

[0045] In one embodiment, the target multimodal image includes: a first multimodal image stitched together from a target grayscale image and a target infrared image; a second multimodal image stitched together from a target infrared image and a target depth image; a third multimodal image stitched together from a target grayscale image and a target depth image; and a fourth multimodal image stitched together from a target grayscale image, a target infrared image, and a target depth image.

[0046] It should be noted that this embodiment does not limit whether the face image to be identified is unimodal or multimodal; it can be a target unimodal image or a target multimodal image. That is, in this embodiment, images of multiple modalities can be used for liveness detection.

[0047] S202: Input the face image to be identified into the multimodal liveness detection model for detection processing to obtain the liveness detection result corresponding to the face image to be identified; wherein, the multimodal liveness detection model is trained by the multimodal liveness detection model training method described above.

[0048] In one embodiment, the liveness detection result obtained in this embodiment is the target confidence score corresponding to the face image to be identified; wherein, the target confidence score identifies the liveness detection result. For example, the target confidence score is a value between 0 and 1. When the target confidence score is greater than 0.5, the liveness detection result is identified as a real person, and when the target confidence score is less than or equal to 0.5, the liveness detection result is identified as a fake person.

[0049] Compared with the prior art, the embodiments of this application can flexibly perform liveness detection on single-modal format images and multi-modal format images by adopting a unified multimodal model. When the model is deployed in the product, it can be flexibly deployed according to single-modal or multimodal, which greatly reduces the workload of maintaining multiple models and improves the accuracy and stability of liveness detection.

[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0051] Figure 3 This is a schematic diagram of a multimodal liveness detection model training device provided in the third embodiment of this application. This device is used to perform… Figure 1 The steps in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown; for parts not covered in detail, please refer to the detailed description. Figure 1 The relevant descriptions in the corresponding embodiments.

[0052] In one embodiment, the training device 3 includes: an image acquisition unit 310, used to acquire sample channel images corresponding to sample faces, wherein the sample channel images are obtained by stitching together at least a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image; a masking and annotation unit 320, used to randomly mask the sample channel images to obtain corresponding target format images, and to annotate the target format images to obtain corresponding sample confidence scores; wherein the target format images include single-modal format images and multi-modal format images; a training set acquisition unit 330, used to acquire a sample training set based on the target format images and their corresponding sample confidence scores; and a training unit 340, used to train a preset liveness detection neural network based on the sample training set to obtain a multi-modal liveness detection model.

[0053] Figure 4 This is a schematic diagram of the multimodal liveness detection device provided in the fourth embodiment of this application, which is used to perform... Figure 2 The steps in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown; for parts not covered in detail, please refer to the detailed description. Figure 2 The relevant descriptions in the corresponding embodiments.

[0054] In one embodiment, the detection device 4 includes: an acquisition unit 410 for acquiring a face image to be identified, wherein the face image to be identified includes a target unimodal image or a target multimodal image; and a detection unit 420 for inputting the face image to be identified into a multimodal liveness detection model for detection processing to obtain a liveness detection result corresponding to the face image to be identified; wherein the multimodal liveness detection model is trained using the above-described multimodal liveness detection model training method.

[0055] Figure 5 The fifth embodiment of this application provides a schematic diagram of an electronic device, which includes: a processor 50, a memory 51, and a computer program stored in the memory 51 and executable on the processor 50. When the processor executes the computer program, it implements the training method or multimodal liveness detection method of the multimodal liveness detection model mentioned in the above embodiments.

[0056] In one embodiment, the device may further include an acquisition unit, comprising one or more combinations of a grayscale camera, an RGB camera, and a depth camera. The grayscale camera, RGB camera, and depth camera are used to acquire grayscale images, RGB images, infrared images, and depth images containing a face, respectively, and send them to a processor for the processor to execute the aforementioned multimodal liveness detection model training method or multimodal liveness detection method. It should be noted that the acquisition unit is independently configured and connected to the electronic device via wired and / or wireless communication, or embedded in the electronic device as a component of the electronic device. Furthermore, the depth camera may be any one or more combinations of a structured light camera, a TOF camera, a binocular camera, or a multi-view camera; no limitation is made here.

[0057] In one embodiment, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0058] In one embodiment, the memory can be an internal storage unit of the device, such as a hard drive or RAM; or it can be an external storage device of the device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the device may include both internal and external storage units. The memory is used to store the computer program and other necessary programs and data. The memory can also be used to temporarily store data that has been output or will be output.

[0059] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0060] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0061] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0062] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0063] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0065] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training a multimodal liveness detection model, characterized in that, include: Obtain the sample channel image corresponding to the sample face; wherein, the sample channel image is obtained by stitching together a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image; the single-channel grayscale image, the single-channel infrared image, and the single-channel depth image have the same size; The sample channel images are randomly masked and unmasked to obtain corresponding target format images, and the sample confidence scores corresponding to the target format images are labeled. The target format images include unimodal format images and multimodal format images. The unimodal format image is any channel image from the sample channel images, and the multimodal format image is a combination of multiple channel images from the sample channel images. The sample confidence scores are pre-labeled. Each target format image corresponds to one sample confidence score, and the sample confidence score identifies the liveness detection result corresponding to the target format image. A sample training set is obtained based on the target format image and its corresponding sample confidence score, and a multimodal liveness detection model is obtained by training a preset liveness detection neural network using the sample training set, including: The target format image is input into a preset liveness detection neural network for processing to obtain the target confidence score, and the target loss value is calculated based on the target confidence score, the sample confidence score, and a preset loss function. If the target loss value does not meet the preset termination condition, the preset liveness detection neural network is updated according to the target loss value, and the process returns to input the target format image into the preset liveness detection neural network for processing to obtain the target confidence score. If the target loss value meets the preset termination condition, the trained multimodal liveness detection model is output. The preset liveness detection neural network includes a feature extraction module, a downsampling module, and a feature fusion module. The step of inputting the target format image into the preset liveness detection neural network for processing to obtain the target confidence score includes: The target format image is input into the feature extraction module, which is connected in a multi-layered series. Based on each layer of the feature extraction module, at least the first feature information, the second feature information, and the third feature information can be obtained sequentially. The first feature information, the second feature information, and the third feature information have different sizes. The downsampling module performs downsampling processing on the first feature information and the second feature information respectively to obtain the fourth feature information and the fifth feature information with the same size as the third feature information. The feature fusion module performs concatenation processing on the third, fourth, and fifth feature information along the channel dimension to obtain fused feature information, and determines the target confidence level based on the fused feature information.

2. The multimodal liveness detection model training method as described in claim 1, characterized in that, The process of obtaining the sample channel image corresponding to the sample face includes: Acquire an initial sample color image, an initial infrared image, and an initial depth image containing and aligned sample faces, and identify the sample faces in the initial sample color image; Based on the position information, size information and alignment relationship of the sample face, the initial infrared image and the initial depth image are cropped to obtain the single-channel infrared image and single-channel depth image containing only the sample face; Convert the initial sample color image to an initial grayscale image and obtain the single-channel grayscale image corresponding to the sample face; The single-channel grayscale image, the single-channel infrared image, and the single-channel depth image are stitched together along the channel dimension to obtain the sample channel image corresponding to the sample face.

3. The multimodal liveness detection model training method as described in claim 1, characterized in that, The step of randomly masking or not masking the sample channel images to obtain the corresponding target format image includes: Set one or two of the pixels of the single-channel grayscale image, the single-channel infrared image, or the single-channel depth image in the channel of the sample channel image to 0 to obtain the target format image corresponding to the sample channel image; and use the sample channel image as the target format image corresponding to the sample channel image.

4. A multimodal liveness detection method, characterized in that, include: Acquire a face image to be identified, wherein the face image to be identified may include any one or more combinations of grayscale image, infrared image or depth image; The face image to be identified is input into a multimodal liveness detection model for detection processing to obtain the liveness detection result corresponding to the face image to be identified; wherein, the multimodal liveness detection model is trained by the multimodal liveness detection model training method according to any one of claims 1-3.

5. The multimodal liveness detection method as described in claim 4, characterized in that, The face image to be identified may also include an RGB image. When the face image to be identified is an RGB image, the RGB image needs to be converted into a grayscale image, and then the grayscale image is input into the multimodal liveness detection model for detection processing to obtain the liveness detection result.

6. A multimodal liveness detection model training device, characterized in that, include: The image acquisition unit is used to acquire sample channel images corresponding to sample faces, wherein the sample channel images are obtained by stitching together at least a single-channel grayscale image, a single-channel infrared image, and a single-channel depth image; the single-channel grayscale image, the single-channel infrared image, and the single-channel depth image have the same size; A masking and annotation unit is used to randomly mask or not mask the sample channel images to obtain corresponding target format images and to annotate the sample confidence scores corresponding to the target format images. The target format images include unimodal and multimodal format images. The unimodal format image is any channel image from the sample channel images, and the multimodal format image is a combination of multiple channel images from the sample channel images. The sample confidence scores are pre-annotated. Each target format image corresponds to one sample confidence score, and the sample confidence score identifies the liveness detection result corresponding to the target format image. The training set acquisition unit is used to acquire a sample training set based on the target format image and its corresponding sample confidence. The training unit, used to train a preset liveness detection neural network based on the sample training set to obtain a multimodal liveness detection model, includes: The target format image is input into a preset liveness detection neural network for processing to obtain the target confidence score, and the target loss value is calculated based on the target confidence score, the sample confidence score, and a preset loss function. If the target loss value does not meet the preset termination condition, the preset liveness detection neural network is updated according to the target loss value, and the process returns to input the target format image into the preset liveness detection neural network for processing to obtain the target confidence score. If the target loss value meets the preset termination condition, the trained multimodal liveness detection model is output. The preset liveness detection neural network includes a feature extraction module, a downsampling module, and a feature fusion module. The step of inputting the target format image into the preset liveness detection neural network for processing to obtain the target confidence score includes: The target format image is input into the feature extraction module, which is connected in a multi-layered series. Based on each layer of the feature extraction module, at least the first feature information, the second feature information, and the third feature information can be obtained sequentially. The first feature information, the second feature information, and the third feature information have different sizes. The downsampling module performs downsampling processing on the first feature information and the second feature information respectively to obtain the fourth feature information and the fifth feature information with the same size as the third feature information. The feature fusion module performs concatenation processing on the third, fourth, and fifth feature information along the channel dimension to obtain fused feature information, and determines the target confidence level based on the fused feature information.

7. A multimodal liveness detection device, characterized in that, include: An acquisition unit is used to acquire a face image to be identified; the face image to be identified may include any one or more combinations of grayscale images, infrared images, or depth images; The detection unit is used to input the face image to be identified into a multimodal liveness detection model for detection processing to obtain the liveness detection result corresponding to the face image to be identified; wherein, the multimodal liveness detection model is trained by the multimodal liveness detection model training method according to any one of claims 1-3.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the multimodal liveness detection model as described in any one of claims 1 to 3, or the multimodal liveness detection method as described in any one of claims 4 to 5.

9. The electronic device as claimed in claim 8, characterized in that, The electronic device further includes a data acquisition device for acquiring one or more of grayscale images, RGB images, infrared images, and depth images containing a human face and transmitting them to the processor for the processor to execute the training method of the multimodal liveness detection model as described in any one of claims 1 to 3, or the multimodal liveness detection method as described in any one of claims 4 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the multimodal liveness detection model as described in any one of claims 1 to 3, or the multimodal liveness detection method as described in any one of claims 4 to 5.

Images