Living body recognition method and device, electronic equipment and storage medium

Abstract

The application provides a living body recognition method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring a to-be-recognized image of a target user under a target mode; inputting the to-be-recognized image into a first detection model to output a living body detection result of the target user, wherein the first detection model is obtained by constraining an initial first detection model by a first category token based on a second detection model, the second detection model is trained based on multi-modal image samples, and the first category token is a parameter when the second detection model performs living body detection on a user corresponding to an image sample. In the method, the second detection model can utilize data features of different modes in the multi-modal image samples to fully mine the relationship between the data. The first category token determined by the second detection model is used to constrain the initial first detection model, so that the first detection model can perform living body recognition on images of any mode, and the flexibility of living body recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a liveness detection method, device, electronic device, and storage medium. Background Technology

[0002] Facial recognition systems have been applied to many aspects of daily life, such as unlocking mobile phones and authenticating online banking. However, the widespread use of facial recognition also brings potential security risks, such as using non-living facial images for illegal activities. Therefore, in addition to facial recognition, it is necessary to determine whether the image to be recognized is from a living person, i.e., liveness detection.

[0003] Liveness detection algorithms typically employ single-modality image data, such as color images or near-infrared images alone. However, single-modality image recognition suffers from poor generalization. Thanks to the emergence of advanced sensors with multiple modalities, multimodal liveness detection methods have gained widespread attention. These advanced sensors can output images in various modalities, including red-green-blue (RGB) color images, near-infrared (NIR) images, and depth images. Liveness detection algorithms are increasingly evolving towards multimodal image recognition to improve generalization.

[0004] In related technologies, multimodal image-based liveness detection typically uses fixed-modal inputs. For example, RGB color images and NIR near-infrared images may be used as multimodal inputs, or RGB color images and depth images may be used. Using fixed-modal images for multimodal liveness detection limits its application scenarios, increases the deployment cost of liveness detection methods, and reduces their flexibility. Summary of the Invention

[0005] This invention provides a liveness detection method, device, electronic device, and storage medium to address the shortcomings of existing technologies in multimodal liveness detection when inputting images with fixed modalities, which have low flexibility, thereby improving the flexibility of multimodal liveness detection.

[0006] This invention provides a liveness detection method, comprising:

[0007] Acquire the target user's image to be identified in the target modality;

[0008] The image to be identified is input into the first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on the first category token determined by the second detection model. The second detection model is trained based on multimodal image samples, and the target modality is the modality in the multimodal model. The first category token is the parameter of the second detection model when performing liveness detection on the user corresponding to the image sample.

[0009] According to a liveness detection method provided by the present invention, the step of inputting the image to be identified into a first detection model to obtain the liveness detection result of the target user output by the first detection model includes:

[0010] The image to be identified is input into the first detection model to obtain at least two first tokens;

[0011] For each first token, extract the first token feature of the first token;

[0012] Add a first preset category token to the first token feature to obtain a second token feature;

[0013] The second token feature is updated to obtain the updated first preset category token;

[0014] Based on the updated first preset category token, the liveness detection result of the target user is determined.

[0015] According to the present invention, a liveness detection method is provided, wherein the first detection model is trained in the following manner:

[0016] The multimodal image samples are input into the initial first detection model to obtain at least two second tokens corresponding to each modality;

[0017] Extract the third token features of at least two second tokens corresponding to each modality;

[0018] Based on the third token features of different modalities, at least two first concatenated token features are obtained, and each first concatenated token feature includes at least one third token feature corresponding to a modality;

[0019] For each first concatenated token feature, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature;

[0020] The fourth token feature is updated to obtain the updated second preset category token;

[0021] The initial first detection model is constrained based on the updated second preset category token and the first category token to obtain the constraint result;

[0022] Based on the constraint results and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model.

[0023] According to a liveness detection method provided by the present invention, the step of constraining the initial first detection model based on the updated second preset category token and the first category token to obtain a constraint result includes:

[0024] Based on formula Constrain the initial first detection model;

[0025] Among them, f T Indicates the first category of tokens; F s This represents the set of each updated second-preset category token; F represents s The updated second preset category token.

[0026] According to the present invention, a liveness detection method is provided, wherein the second detection model is trained in the following manner:

[0027] The multimodal image samples are input into the initial second detection model to obtain at least two third tokens corresponding to each modality;

[0028] Extract the fifth token features of at least two third tokens corresponding to each modality;

[0029] The fifth token features of different modalities are concatenated to obtain the second concatenated token feature;

[0030] Add a third preset category token to the second concatenated token feature to obtain the sixth token feature;

[0031] Based on the sixth token feature, the first category of tokens is determined;

[0032] Based on the first category token, the network parameters of the initial second detection model are updated to obtain the second detection model.

[0033] According to a liveness detection method provided by the present invention, determining the liveness detection result of the target user based on the updated first preset category token includes:

[0034] Based on the updated first preset category token, determine the liveness detection score of the target user;

[0035] If the liveness detection score is greater than a preset score, the liveness detection result of the target user is determined to be a live user.

[0036] According to a liveness detection method provided by the present invention, the method further includes:

[0037] Obtain initial image samples for each modality;

[0038] A face detection bounding box is determined in the initial image sample, and the face detection bounding box is used to select the face region in the initial image sample;

[0039] Based on the coordinate information corresponding to the face detection box, the face detection box is expanded to obtain the expanded face detection box;

[0040] The face image is extracted from the expanded face detection box to obtain a face image sample;

[0041] The face image sample is scaled to obtain the image sample.

[0042] The present invention also provides a liveness detection device, comprising:

[0043] The acquisition module is used to acquire the image of the target user in the target modality.

[0044] The recognition module is used to input the image to be recognized into a first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality in the multimodal model. The first category token is a parameter used by the second detection model to perform liveness detection on the user corresponding to the image sample.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described liveness detection methods.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the liveness detection method as described above.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the liveness detection method as described above.

[0048] This invention provides a liveness detection method, apparatus, electronic device, and storage medium. The method involves acquiring an image of a target user in a target modality, inputting the image into a first detection model, and obtaining the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining an initial first detection model with a first-category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality within the multimodal model. The first-category token is the parameter used by the second detection model when performing liveness detection on the user corresponding to the image sample. In this method, the second detection model trained on multimodal image samples can fully utilize the data features of different modalities in the multimodal image samples to fully explore the relationships between data. By constraining the initial first detection model with the first-category token determined by the second detection model, the initial first detection model can learn the parameters of the second detection model, making the initial first detection model more fully learned and capable of learning information from image samples of various modalities. Furthermore, after inputting the image to be identified of any target modality in the multimodal model into the first detection model, the first detection model can perform liveness detection on the image to be identified, thereby achieving the purpose of liveness detection on the image to be identified of any modality, and improving the flexibility of the liveness detection method. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the liveness detection method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic flowchart of the model training process provided in the embodiments of the present invention;

[0052] Figure 3 This is a schematic block diagram illustrating the specific process of model training provided in this embodiment of the invention;

[0053] Figure 4a This is one of the schematic diagrams illustrating different modalities corresponding to token feature concatenation methods provided in the embodiments of the present invention;

[0054] Figure 4b This is the second schematic diagram of the token feature concatenation method corresponding to different modalities provided in the embodiments of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure of the liveness detection device provided in an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] It should be noted that the serial numbers assigned to the objects described in this invention, such as "first" and "second", are only used to distinguish the objects being described and do not have any sequential or technical meaning.

[0059] Security risks associated with facial recognition can stem from attacks such as printing attacks or replay attacks. Printing attacks involve using printed images of faces for facial recognition, while replay attacks involve playing videos containing faces for facial recognition. Liveness detection can mitigate these security risks. For applications with high security requirements, such as facial payments, highly generalizable liveness detection is necessary.

[0060] While single-modal image data, such as color or near-infrared images, offers some protection against known attack methods, its generalization ability is poor when new attack methods emerge. Utilizing multimodal data for liveness detection can improve generalization. However, existing multimodal liveness detection algorithms, although employing various image modalities, have relatively fixed input modalities, such as only accepting color and near-infrared images or only accepting color and depth images. This significantly limits the application scenarios, reduces flexibility, and increases deployment costs. Improving the flexibility of multimodal liveness detection has become a pressing issue.

[0061] Fixed-modality multimodal liveness detection algorithms focus only on data features related to the input modality during training, ignoring the interrelationships between data from other modalities. They fail to fully utilize the multimodal data information during training. Therefore, fixed-modality multimodal liveness detection algorithms cannot perform liveness detection on images of any modality other than the fixed modality.

[0062] Based on this, embodiments of the present invention provide a liveness detection method. This method trains an initial second detection model using multimodal image samples to obtain a second detection model. This second detection model can fully utilize the data features of different modalities in the multimodal image samples to fully explore the relationships between data. Furthermore, the initial first detection model is constrained by a first category token determined by the second detection model, enabling the trained first detection model to learn the parameters of the second detection model. This allows the initial first detection model to learn more fully and acquire information from image samples of various modalities, thereby enabling liveness detection for images of any modality. This not only reduces the cost of liveness detection but also improves the flexibility of the liveness detection method. The following is a detailed explanation... Figures 1 to 4b The liveness detection method provided in the embodiments of the present invention will be described.

[0063] Figure 1 This is a flowchart illustrating the liveness detection method provided in an embodiment of the present invention. This embodiment can be applied to liveness detection scenarios of any modality. The executing entity of this method can be an electronic device such as a mobile phone, smartwatch, tablet computer, computer, or a specially designed liveness detection device, or a liveness detection device installed in the electronic device. This liveness detection device can be implemented through software, hardware, or a combination of both. Figure 1 As shown, the liveness detection method includes steps 110 and 120.

[0064] Step 110: Obtain the image of the target user in the target modality.

[0065] Specifically, the target user is the user for whom liveness detection needs to be performed; the target modality can be the modality of an image set in the liveness detection scenario. Image modality can be understood as the type of image; a single modality is a single type of image, while a multimodal modality is multiple types of images. The target modality can be single-modal or multimodal. When acquiring the image to be identified of the target user in the target modality, images can be captured using devices with image acquisition capabilities such as mobile phones, cameras, or camcorders; alternatively, images can be retrieved from a database; or other methods can be used to acquire the image to be identified.

[0066] For example, the target modality can be a single modality of an RGB color image, a single modality of a NIR (near-infrared) image, or a single modality of a depth image. The target modality can also be a multimodal combination of an RGB color image and a NIR image, or a multimodal combination of an RGB color image and a depth image, or a multimodal combination of an NIR image and a depth image, or a multimodal combination of an RGB color image, a NIR image, and a depth image, etc.

[0067] For example, when the target modality is a multimodal image consisting of RGB color images and NIR near-infrared images, the image acquisition device acquires the RGB color image and the NIR near-infrared image of the target user in that target modality, and uses both the RGB color image and the NIR near-infrared image as images to be identified. When acquiring the RGB color image and the NIR near-infrared image of the target user in that target modality, these two types of images can be acquired simultaneously or separately.

[0068] Step 120: Input the image to be identified into the first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on the first category token determined by the second detection model. The second detection model is trained based on multimodal image samples, and the target modality is the modality in the multimodal model. The first category token is the parameter of the second detection model when performing liveness detection on the user corresponding to the image sample.

[0069] Specifically, the second detection model can be a network model obtained by training the initial second detection model based on multimodal image samples; the first detection model can be a network model obtained by training the initial first detection model based on the same image samples as the initial second detection model, and constraining the initial first detection model with the first category token determined by the second detection model during the training process.

[0070] In the field of image recognition technology, a token can be understood as a series of labels that interpret an image as a series of words similar to those in natural language; that is, it can be understood as a label or a parameter used for labeling. The first category of tokens can be parameters determined by the second detection model when performing liveness detection on the user corresponding to the image sample. This parameter can characterize whether the user corresponding to the image sample is alive.

[0071] The training of the initial first detection model is constrained by the first category token determined by the second detection model. This can be understood as follows: during the adjustment or updating of the network parameters of the initial first detection model, the first category token is used as a constraint condition for updating, thereby obtaining the first detection model. After training the first detection model, the image of the target user to be identified can be input into the first detection model to obtain the liveness detection result of the target user output by the first detection model.

[0072] When training the initial second detection model, the input is multimodal image samples, and the target modality is any modality within that multimodal model. This can be understood as the input image samples containing multiple types, and the target modality containing at least one of the types included in the image samples. For example, when training the initial second detection model, if the input multimodal image samples include RGB color images, NIR (near-infrared) images, and depth images, then the target modality can be at least one of these three image types.

[0073] The first detection model is obtained by constraining the initial first detection model with the first category tokens determined by the second detection model. For example, it can be a joint model training process using a teacher model and a student model based on knowledge distillation. The second detection model can be a teacher model, trained on multimodal image samples. The teacher model fully explores the relationships between image samples of different modalities, learns rich knowledge from the samples, and acts as a knowledge outputter. It can also impart knowledge to the student model through the first category tokens it determines during training. The first detection model can be a student model, acting as a knowledge receiver. It learns from the knowledge output by the teacher model and is trained using the first category tokens determined by the second detection model. This allows it to achieve similar functionality and output capabilities to the second detection model. Compared to the teacher model, the student model typically has simpler structure, fewer parameters, and higher computational efficiency.

[0074] Both the initial first detection model and the initial second detection model can include models composed of at least one of the following neural networks: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) Neural Network, and Deep Neural Network (DNN), but are not limited to these.

[0075] The liveness detection method provided in this invention acquires an image of a target user in a target modality, inputs the image into a first detection model, and obtains the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining an initial first detection model with a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality within the multimodal model. The first category token is the parameter used by the second detection model when performing liveness detection on the user corresponding to the image sample. In this method, the second detection model trained on multimodal image samples can fully utilize the data features of different modalities in the multimodal image samples to fully explore the relationships between data. By constraining the initial first detection model with the first category token determined by the second detection model, the initial first detection model can learn the parameters of the second detection model, making the initial first detection model learn more fully and capable of learning information from image samples of various modalities. Furthermore, after inputting the image to be identified of any target modality in the multimodal model into the first detection model, the first detection model can perform liveness detection on the image to be identified, thereby achieving the purpose of liveness detection on the image to be identified of any modality, and improving the flexibility of the liveness detection method.

[0076] Figure 2 This is a schematic flowchart of the model training process provided in an embodiment of the present invention, such as... Figure 2 As shown, taking RGB color images, NIR near-infrared images, and depth images as examples, model training can include two processes: training an initial second detection model to obtain a second detection model, and training an initial first detection model based on the second detection model to obtain a first detection model. Training the initial second detection model to obtain the second detection model can be achieved by first inputting the RGB color image, NIR near-infrared image, and depth image into the initial second detection model to update its network parameters. Training the initial first detection model based on the second detection model to obtain the first detection model can be achieved by inputting the RGB color image, NIR near-infrared image, and depth image into the initial first detection model, and constraining the initial first detection model with the first category token output by the second detection model to obtain the first detection model. After training the first detection model, the process of performing liveness detection on the image to be recognized can involve inputting an image of any modality to be recognized into the first detection model to obtain the liveness detection result output by the first detection model. Based on the above model training process, the initial second detection model can learn information from image data containing multiple modalities, fully explore the relationships between data, and obtain better liveness detection results. When the model is applied, based on the trained first detection model, liveness detection can be performed on images of any modality to obtain liveness detection results.

[0077] The following describes the examples of training the second detection model and training the first detection model.

[0078] In one embodiment, the second detection model can be trained in the following manner:

[0079] Multimodal image samples are input into the initial second detection model to obtain at least two third tokens corresponding to each modality; fifth token features are extracted from the at least two third tokens corresponding to each modality; the fifth token features of different modalities are concatenated to obtain the second concatenated token features; a third preset category token is added to the second concatenated token features to obtain the sixth token features; the first category token is determined based on the sixth token features; the network parameters of the initial second detection model are updated based on the first category token to obtain the second detection model.

[0080] Specifically, when training the initial second detection model, the input multimodal image samples should include image samples from all modalities. For example, image samples from all modalities include image samples from RGB color images, image samples from NIR near-infrared images, and image samples from depth images.

[0081] For example, the initial second detection model may include a third linear projection layer, which may be a network structure layer containing at least one convolutional layer. For instance, the third linear projection layer may be a single convolutional layer, or it may be two convolutional layers. By inputting multimodal image samples into the third linear projection layer of the initial second detection model, at least two third tokens corresponding to each modality can be obtained from the output of the third linear projection layer. These third tokens may be sub-image labels representing each modality image sample output by the third linear projection layer.

[0082] Figure 3 This is a schematic block diagram illustrating the specific process of model training provided in this embodiment of the invention. Taking RGB color images, NIR near-infrared images, and depth images as examples, RGB color images, NIR near-infrared images, and depth images are simultaneously acquired from a living face using an image acquisition device, and label values ​​are set for each image to obtain multimodal image samples. Alternatively, non-living facial images, including RGB color images, NIR near-infrared images, and depth images, can be obtained from a database, and label values ​​are set for each image to obtain multimodal image samples. The label values ​​represent the true value indicating whether an image is a real, living image. Other methods can also be used to obtain multimodal image samples, which will not be elaborated here.

[0083] likeFigure 3 As shown, during the initial training of the second detection model, multimodal image samples of RGB color images, NIR near-infrared images, and depth images are input into the linear projection layer, that is, into the third linear projection layer of the initial second detection model. The third linear projection layer can output the third token corresponding to the RGB color image, the third token corresponding to the NIR near-infrared image, and the third token corresponding to the depth image. The third token corresponding to each modality image can be understood as the sub-image label of the sub-image obtained after image segmentation.

[0084] After obtaining the third token corresponding to each modality, feature extraction needs to be performed on each third token. This can be done through the third feature extraction layer of the initial second detection model. For example, the third feature extraction layer could be a Transformer layer. By inputting each third token into the third feature extraction layer of the initial second detection model for feature extraction, the features of the fifth token corresponding to each third token can be obtained. Figure 3 As shown in the figure, 1, 2, 3, and 4 represent a fifth token feature; 5, 6, 7, and 8 represent a fifth token feature; and 9, 10, 11, and 12 represent a fifth token feature. After obtaining the fifth token features corresponding to each modality, all fifth token features are concatenated to obtain a second concatenated token feature. Further, a third preset category token is added to the second concatenated token feature to obtain a sixth token feature. The third preset category token can be an initial value of the category token preset during the initial training of the second detection model; for example, the third preset category token could be 0, 0.1, or 0.01.

[0085] Figure 4a This is one of the schematic diagrams illustrating different modalities corresponding to token feature concatenation methods provided in the embodiments of the present invention, such as... Figure 4aAs shown, during the initial training of the second detection model, the fifth token feature corresponding to the RGB color image is represented by 1, 2, 3, and 4; the fifth token feature corresponding to the NIR near-infrared image is represented by 5, 6, 7, and 8; and the fifth token feature corresponding to the depth image is represented by 9, 10, 11, and 12. Concatenating the fifth token features corresponding to the RGB color image, the NIR near-infrared image, and the depth image yields the second concatenated token feature. Adding a third preset category token to this second concatenated token feature yields the sixth token feature corresponding to the fifth token feature. Figure 4a The token features represented by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 and cls are the sixth token features, where cls represents the third preset category token.

[0086] The third fusion feature layer can be a network structure layer used for feature fusion in the initial second detection model. The third fusion feature layer can consist of at least one Transformer layer. For example, the third fusion feature layer in the initial second detection model is a network structure layer composed of three Transformer layers. This third fusion feature layer can be understood as the feature fusion block in the initial second detection model. Inputting the sixth token feature into the third fusion feature layer of the initial second detection model allows for feature fusion, resulting in the first category token output by the third fusion feature layer. This first category token can be understood as the parameters for user liveness detection corresponding to the image samples output by the initial second detection model during training. For example, the first category token is denoted as f. T .

[0087] Figure 3 The thick arrows shown indicate the process of training the initial second detection model, and the thin arrows indicate the process of training the initial first detection model. For example... Figure 3As shown, during the training of the second detection model, after obtaining the sixth token feature, the sixth token feature is input into the third fusion feature layer of the initial second detection model for feature fusion. This third fusion feature layer, for example, consists of three Transformer layers. The Transformer layers in the third fusion feature layer can share weights, i.e., share the network parameters of each Transformer layer. The third fusion feature layer can output a fused feature of the sixth token feature, which includes an updated third preset category token. This updated third preset category token is input into the classification head for classification, obtaining the liveness detection result output by the initial second detection model. Based on the detection result obtained from the classification head, it can be determined whether the image sample is a live image sample, i.e., the liveness detection result output by the initial second detection model is obtained. For example, if the classification threshold of the classification head is set to 0.5, when the first category token input to the classification head is greater than 0.5, the detection result of the input multimodal image sample being a live image sample can be obtained; when the first category token input to the classification head is less than or equal to 0.5, the detection result of the input multimodal image sample being a non-live image sample can be obtained. Based on the result and the label information corresponding to the image samples, the loss is determined, and the model parameters of the initial second detection model are updated based on this loss. This process is iterated until the required number of iterations is reached or the model converges. The final model obtained is then determined as the trained second detection model. The category token output by the trained second detection model is the first category token.

[0088] The network parameters of the initial second detection model are updated based on the liveness detection results and the label values ​​of the recorded multimodal image samples. This allows the liveness detection results of the initial second detection model to gradually approach the label values ​​during training. In other words, updating the network parameters of the initial second detection model causes its output value to converge until training is complete, resulting in the second detection model. Optionally, the cross-entropy loss function can be used as the objective function to train the initial second detection model for the classification layer.

[0089] In this embodiment, multimodal image samples are input into the initial second detection model to train the initial second detection model. This allows the initial second detection model to learn data information from multiple modal images, fully explore the relationships between data information, and enable the final second detection model to output a first-class token with high accuracy. Furthermore, the initial first detection model is constrained based on the first-class token, so that the liveness detection results obtained by the first detection model can converge with the detection results of the second detection model. This achieves the goal of enabling the first detection model to perform liveness recognition on images of any modality, improving the flexibility of liveness recognition.

[0090] After training the second detection model, the initial first detection model can be trained based on the first category token output by the second detection model to obtain the first detection model.

[0091] In one embodiment, the first detection model can be trained in the following manner:

[0092] Multimodal image samples are input into an initial first detection model to obtain at least two second tokens corresponding to each modality; third token features are extracted from the at least two second tokens corresponding to each modality; based on the third token features of different modalities, at least two first concatenated token features are obtained, each first concatenated token feature including at least one third token feature corresponding to a modality; for each first concatenated token feature, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature; the fourth token feature is updated to obtain an updated second preset category token; the initial first detection model is constrained based on the updated second preset category token and the first category token to obtain a constraint result; based on the constraint result and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model.

[0093] Specifically, when training the initial first detection model, multimodal image samples from the initial second detection model can be input. For example, if the multimodal image samples input when training the second detection model include RGB color images, NIR near-infrared images, and depth images, then the same three modal image samples can be input when training the first detection model.

[0094] For example, the initial first detection model may include a second linear projection layer, which may be a network structure layer containing at least one convolutional layer. For instance, the second linear projection layer may be a single convolutional layer or two convolutional layers. Inputting multimodal image samples into the second linear projection layer of the initial first detection model yields at least two second tokens corresponding to each modality output by the second linear projection layer. These second tokens may be sub-image labels representing each modality image sample output by the second linear projection layer. For example, inputting an RGB color image, a NIR near-infrared image, and a depth image into the second linear projection layer of the initial first detection model yields the second tokens corresponding to the RGB color image, the NIR near-infrared image, and the depth image output by the second linear projection layer.

[0095] After obtaining the second tokens corresponding to each modality, feature extraction needs to be performed on each second token. This can be done through the second feature extraction layer of the initial first detection model. The second feature extraction layer can be, for example, a Transformer layer. Inputting at least two second tokens corresponding to each modality into the second feature extraction layer of the initial first detection model yields the third token features corresponding to each modality, output by the second feature extraction layer. After obtaining the third token features corresponding to each modality, these features are concatenated in a permutation and combination manner to obtain at least two first concatenated token features. Each first concatenated token feature then includes at least one third token feature corresponding to a modality.

[0096] Furthermore, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature. The second preset category token can be an initial value of the category token preset when training the initial first detection model. For example, the second preset category token can be 0, 0.1, or 0.01.

[0097] like Figure 3 As shown, the thin arrows indicate the process of training the initial first detection model. When training the initial first detection model, after obtaining the third token features, the third token features are concatenated in a permutation and combination form to obtain at least two first concatenated token features.

[0098] Figure 4b This is the second schematic diagram of the token feature concatenation method corresponding to different modalities provided in the embodiments of the present invention, such as... Figure 4b As shown, during the initial training of the first detection model, the third token features corresponding to RGB color images are represented by 1, 2, 3, and 4; the third token features corresponding to NIR near-infrared images are represented by 5, 6, 7, and 8; and the third token features corresponding to depth images are represented by 9, 10, 11, and 12. By concatenating these third token features in various permutations and combinations, at least two first concatenated token features can be obtained. Adding a second preset category token to each first concatenated token feature yields the fourth token feature corresponding to that third token feature. Figure 4bIn the following, 1, 2, 3, 4 and cls represent a fourth token feature; 5, 6, 7, 8 and cls represent a fourth token feature; 1, 2, 3, 4, 5, 6, 7, 8 and cls represent a fourth token feature; 1, 2, 3, 4, 9, 10, 11, 12 and cls represent a fourth token feature; 5, 6, 7, 8, 9, 10, 11, 12 and cls represent a fourth token feature; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 and cls represent a fourth token feature; where cls represents the second preset category token.

[0099] The second fusion feature layer can be a network structure layer used for feature fusion in the initial first detection model, and the second fusion feature layer can consist of at least one Transformer layer. For example... Figure 3 As shown, the second fusion feature layer in the initial first detection model is a network structure consisting of three Transformer layers. The Transformer layers in the second fusion feature layer can share weights, that is, share the network parameters of each Transformer layer. This second fusion feature layer can be understood as the fusion feature extraction module in the initial first detection model.

[0100] Inputting the fourth token feature into the second fusion feature layer of the initial first detection model allows for feature fusion of the fourth token features, i.e., updating the fourth token features, resulting in the updated second preset category token output by the second fusion feature layer. The updated second preset category token can be understood as follows: based on the preset initial value of the second preset category token, the interaction and fusion of the fourth token features within the second fusion feature layer alters the initial value of the second preset category token, thus updating its value. The set of these updated second preset category tokens can be denoted as F. s For each fourth token feature, the updated second preset category token corresponding to each fourth token feature can be obtained, for example, in, This represents the updated second preset category token corresponding to the fourth token feature of the RGB color image; This represents the updated second preset category token corresponding to the fourth token feature of the NIR near-infrared image; This represents the updated second preset category token corresponding to the fourth token feature of the RGB color image + Depth image; This represents the updated second preset category token corresponding to the fourth token feature of the NIR image + Depth image; This represents the updated second preset category token corresponding to the fourth token feature of the RGB color image + NIR near-infrared image; This represents the updated second preset category token corresponding to the fourth token feature of the RGB color image + NIR near-infrared image + Depth image.

[0101] The initial first detection model is constrained based on the updated second preset category token and the first category token to obtain the constraint result. For example, it can be based on comparing the updated second preset category token output by the initial first detection model with the first category token output by the second detection model. Based on the comparison result, parameter constraints are applied when adjusting the network parameters of the initial first detection model.

[0102] Furthermore, based on the constraint results and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model. This can be understood as inputting the updated second preset category token into the classification layer of the initial first detection model to classify the updated second preset category token. Based on the detection result obtained from this classification layer, it can be determined whether the image sample corresponding to the updated second preset category token is an image sample provided by a live subject, i.e., the liveness detection result output by the initial first detection model is obtained. For example, if the classification threshold of the classification layer is set to 0.5, when the updated second preset category token input to the classification layer is greater than 0.5, the detection result of the input multimodal image sample being a live image sample can be obtained; when the updated second preset category token input to the classification layer is less than or equal to 0.5, the detection result of the input multimodal image sample being a non-live image sample can be obtained.

[0103] The network parameters of the initial first detection model are updated based on the liveness detection results and the label values ​​of the recorded multimodal image samples. This allows the liveness detection results of the initial first detection model to gradually approach the label values ​​during training. In other words, updating the network parameters of the initial first detection model causes its output value to converge until training is complete, resulting in the first detection model. Optionally, for the classification of the updated second preset category token, the cross-entropy loss function can be used as the objective function to train the initial first detection model.

[0104] In this embodiment, multimodal image samples are input into an initial first detection model to train the initial first detection model, and constraints are imposed on the initial first detection model based on a first category token, so that the liveness detection results obtained by the first detection model can be similar to the detection results of the second detection model. This enables the first detection model to perform liveness recognition on the target modality of the image to be identified, thereby achieving the purpose of liveness recognition on images of any modality and improving the flexibility of liveness recognition.

[0105] In one embodiment, the initial first detection model is constrained based on the updated second preset category token and the first category token to obtain a constraint result, including:

[0106] Based on formula Constrain the initial first detection model;

[0107] Among them, f T Indicates the first category of tokens; F s This represents the set of each updated second-preset category token; F represents s The updated second preset category token.

[0108] Specifically, when training the initial first detection model, multimodal image samples are input into the initial first detection model to obtain the fourth token features corresponding to each modality image sample. The fourth token features are input into the second fusion feature layer of the initial first detection model to fuse the features of each fourth token feature to obtain the updated second preset category token output by the second fusion feature layer. The updated second preset category tokens can be combined into a set.

[0109] For example, the set consisting of each updated second preset category token is F. s ,

[0110] In this embodiment, by inputting multimodal image samples into the second detection model, a first category token corresponding to the image sample can be obtained from the output of the second detection model, based on the above formula. The network parameters of the initial first detection model can be constrained and adjusted to optimize the network parameters, so that the updated second preset category token of the initial first detection model gradually converges with the first category token. This enables the first detection model to obtain detection results close to the real results when performing liveness detection on images of any modality, thereby improving the accuracy of liveness detection of the first detection model.

[0111] After training, the first detection model can perform liveness detection on the target user corresponding to the image to be identified, and identify whether the image to be identified is provided by a live person or a non-live person.

[0112] In one embodiment, the image to be identified is input into a first detection model to obtain the liveness detection result of the target user output by the first detection model. This can be achieved in the following way:

[0113] The image to be identified is input into the first detection model to obtain at least two first tokens; for each first token, the first token feature is extracted; a first preset category token is added to the first token feature to obtain the second token feature; the second token feature is updated to obtain the updated first preset category token; based on the updated first preset category token, the liveness detection result of the target user is determined.

[0114] Specifically, the first detection model may include a first linear projection layer, which may be a network structure layer containing at least one convolutional layer. For example, the first linear projection layer may be a single convolutional layer, or the first linear projection layer may be two convolutional layers.

[0115] By inputting the acquired image to be identified into the first linear projection layer of the first detection model, at least two first tokens can be obtained from the output of the first linear projection layer. These first tokens can be sub-image markers representing the image to be identified, output by the first linear projection layer. For example, if the target modality is a multimodal image consisting of RGB color and NIR images, inputting the target user's RGB color image and NIR image into the first linear projection layer of the first detection model can yield the first token corresponding to the RGB color image and the first token corresponding to the NIR image, both output by the first linear projection layer.

[0116] For example, after obtaining each first token corresponding to the image to be recognized, features can be extracted from the first token through a first feature extraction layer. The first feature extraction layer can be, for example, a Transformer layer. By inputting each first token corresponding to the image to be recognized into the first feature extraction layer of the first detection model, the first token features corresponding to the first token output by the first feature extraction layer can be obtained.

[0117] The first preset category token can be an initial value of a category token preset for the first detection model. For example, the first preset category token can be 0, 0.1, or 0.01. After obtaining the first token feature, the first preset category token is added to the first token feature to obtain the second token feature. The second token feature can be understood as the token feature obtained by concatenating the first token feature with the first preset category token.

[0118] The first fusion feature layer can be a network structure layer used for feature fusion in the first detection model. The first fusion feature layer can consist of at least one Transformer layer. For example, the first fusion feature layer in the first detection model is a network structure layer composed of three Transformer layers. This first fusion feature layer can be understood as the fusion feature extraction module in the first detection model. After completing the initial training of the first detection model to obtain the first detection model, the second fusion feature layer in the initial first detection model becomes the first fusion feature layer in the first detection model.

[0119] The second token feature is input into the first fusion feature layer of the first detection model. This allows for feature fusion of the second token feature, resulting in an updated first preset category token output by the first fusion feature layer. The updated first preset category token can be understood as follows: based on the preset initial value of the first preset category token, the second token feature, after interactive fusion within the first fusion feature layer, causes a corresponding change in the initial value of the first preset category token; that is, the value of the first preset category token is updated, resulting in the updated first preset category token.

[0120] In one embodiment, the liveness detection result of the target user is determined based on the updated first preset category token, which can be implemented in the following way:

[0121] Based on the updated first preset category token, determine the liveness detection score of the target user; if the liveness detection score is greater than the preset score, determine the liveness detection result of the target user as a live user.

[0122] Specifically, the first detection model may include a classification layer, which can be a network structure layer capable of classifying the updated first preset category token. By inputting the updated first preset category token into the classification layer of the first detection model, a liveness detection score for the target user can be obtained. The liveness detection score is a value calculated by the classification layer based on the updated first preset category token. Based on this liveness detection score and the preset score, the updated first preset category token can be classified, and the classification result is the liveness detection result, which determines whether the target user is a live user.

[0123] The preset score can be a threshold for classification based on the liveness detection score. The preset score can be determined based on statistical data of the liveness detection results or on the rules of the liveness recognition algorithm. This embodiment of the invention does not specifically limit the preset score or the method for determining it. If the liveness detection score is greater than the preset score, the liveness detection result of the target user can be determined as that of a live user.

[0124] For example, if the preset score is set to 0.5, when the updated first preset category token of the classification layer of the first detection model is greater than 0.5, the liveness detection result of the target user can be determined to be a live user; when the updated first preset category token of the classification layer of the first detection model is less than or equal to 0.5, the liveness detection result of the target user can be determined to be a non-live user.

[0125] In this embodiment, an updated first preset category token can be obtained based on the first linear projection layer, the first feature extraction layer, the first preset category token, and the first fusion feature layer of the first detection model. This enables image processing of the target modality's image to obtain image features, and the first preset category token is updated after fusing the image features. Furthermore, based on the updated first preset category token, the liveness detection score corresponding to the updated first preset category token can be determined through the classification layer of the first detection model. This yields the liveness detection result of the target user output by the first detection model, determining whether the target user is alive. This achieves the goal of performing liveness recognition on images of any modality using the first detection model, improving the flexibility of liveness recognition.

[0126] When training the initial second detection model, training the initial first detection model, or applying the first detection model for liveness detection, the images can be preprocessed to make the preprocessed images more suitable for model training or model application.

[0127] In one embodiment, the liveness detection method further includes: acquiring initial image samples of each modality; determining a face detection box in the initial image sample, the face detection box being used to select the face region in the initial image sample; expanding the face detection box based on the coordinate information corresponding to the face detection box to obtain an expanded face detection box; extracting a face image from the expanded face detection box to obtain a face image sample; and scaling the face image sample to obtain an image sample.

[0128] Specifically, the initial image samples can be image samples used during model training or images to be recognized when the model is applied. Face detection algorithms can be used to determine the face detection boxes in the initial image samples, that is, to select the image region where the face is located in the initial image using a box selection method.

[0129] Based on the coordinate information corresponding to the face detection bounding box, the face detection bounding box is expanded to obtain an expanded face detection bounding box. This expansion can be based on the pixel coordinates of the face detection bounding box in the image, ensuring that the expanded face detection bounding box completely includes all pixels of the face image. The face image is then extracted from the expanded face detection bounding box to obtain a face image sample; this face image sample is then scaled to obtain an image sample.

[0130] For example, given initial image samples of three modalities—RGB color image, NIR near-infrared image, and depth image—existing face detection algorithms are used to obtain corresponding face detection boxes in each image. These face detection boxes can be, for example, rectangular boxes. The face detection boxes are then expanded along both the length and width directions, and the faces are extracted according to the expanded face detection boxes to obtain face image samples. Finally, the face image samples are scaled, for example, to a size of 256*256 pixels, to obtain the final image sample.

[0131] For example, to facilitate model training and accelerate convergence, the scaled image samples can be normalized. Normalization can involve subtracting the mean and dividing by the variance. For instance, the scaled image samples can be subtracted by 128 from the mean and divided by 128 from the variance. This helps to unify the denominator during model training, thus accelerating convergence.

[0132] In this embodiment, face image samples are obtained based on the acquired initial image samples, and further image samples are obtained. The initial image samples can be preprocessed to make the preprocessed images more suitable for model training, which facilitates the smooth completion of model training and improves the efficiency of model training. When the first detection model after training uses the preprocessed images for application, it can quickly obtain liveness detection results and improve the success rate of model application.

[0133] The following describes the liveness detection device provided in the embodiments of the present invention. The liveness detection device described below can be referred to in correspondence with the liveness detection method described above.

[0134] Figure 5 This is a schematic diagram of the structure of the liveness detection device provided in an embodiment of the present invention, with reference to... Figure 5 As shown, the liveness detection device 500 includes:

[0135] The acquisition module 510 is used to acquire the image of the target user in the target modality.

[0136] The recognition module 520 is used to input the image to be recognized into the first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on the first category token determined by the second detection model. The second detection model is trained based on multimodal image samples, and the target modality is the modality in the multimodal model. The first category token is the parameter of the second detection model when performing liveness detection on the user corresponding to the image sample.

[0137] In one example embodiment, the recognition module 520 is specifically configured to: input the image to be recognized into a first detection model to obtain at least two first tokens; extract first token features for each first token; add a first preset category token to the first token features to obtain second token features; update the second token features to obtain an updated first preset category token; and determine the liveness detection result of the target user based on the updated first preset category token.

[0138] In one example embodiment, the first detection model is trained as follows: multimodal image samples are input into an initial first detection model to obtain at least two second tokens corresponding to each modality; third token features are extracted from the at least two second tokens corresponding to each modality; based on the third token features of different modalities, at least two first concatenated token features are obtained, each first concatenated token feature including at least one third token feature corresponding to a modality; for each first concatenated token feature, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature; the fourth token feature is updated to obtain an updated second preset category token; the initial first detection model is constrained based on the updated second preset category token and the first category token to obtain a constraint result; based on the constraint result and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model.

[0139] In one example embodiment, the initial first detection model is constrained based on the updated second preset category token and the first category token to obtain the constraint result, including:

[0140] Based on formula Constrain the initial first detection model;

[0141] Among them, f T Indicates the first category of tokens; F s This represents the set of each updated second-preset category token; F represents s The updated second preset category token.

[0142] In one example embodiment, the second detection model is trained as follows: multimodal image samples are input into an initial second detection model to obtain at least two third tokens corresponding to each modality; fifth token features are extracted from the at least two third tokens corresponding to each modality; the fifth token features of different modalities are concatenated to obtain a second concatenated token feature; a third preset category token is added to the second concatenated token feature to obtain a sixth token feature; a first category token is determined based on the sixth token feature; and the network parameters of the initial second detection model are updated based on the first category token to obtain the second detection model.

[0143] In one example embodiment, the identification module 520 is specifically used to: determine the liveness detection score of the target user based on the updated first preset category token; and determine the liveness detection result of the target user as a live user if the liveness detection score is greater than the preset score.

[0144] In one example embodiment, the liveness detection device 500 further includes a determination module, an expansion module, an extraction module, and a processing module;

[0145] The acquisition module 510 is also used to acquire initial image samples for each modality; the determination module is used to determine the face detection box in the initial image sample, and the face detection box is used to select the face region in the initial image sample; the expansion module is used to expand the face detection box based on the coordinate information corresponding to the face detection box to obtain the expanded face detection box; the extraction module is used to extract the face image from the expanded face detection box to obtain the face image sample; and the processing module is used to scale the face image sample to obtain the image sample.

[0146] The apparatus of this embodiment can be used to execute the method of any embodiment in the liveness detection method side embodiment. Its specific implementation process and technical effects are similar to those in the liveness detection method side embodiment. For details, please refer to the detailed description in the liveness detection method side embodiment, which will not be repeated here.

[0147] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a liveness detection method. This method includes: acquiring an image of a target user in a target modality; inputting the image to be identified into a first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining an initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality within the multimodal model. The first category token is a parameter used by the second detection model when performing liveness detection on the user corresponding to the image sample.

[0148] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the liveness detection method provided by the above methods. The method includes: acquiring an image of a target user in a target modality; inputting the image to be identified into a first detection model to obtain a liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining an initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality in the multimodality. The first category token is a parameter of the second detection model when performing liveness detection on the user corresponding to the image sample.

[0150] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the liveness detection method provided by the above methods. The method includes: acquiring an image of a target user to be identified in a target modality; inputting the image to be identified into a first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality in the multimodality. The first category token is a parameter of the second detection model when performing liveness detection on the user corresponding to the image sample.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A liveness detection method, characterized in that, include: Acquire the target user's image to be identified in the target modality; The image to be identified is input into a first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality in the multimodal model. The first category token is a parameter used by the second detection model to perform liveness detection on the user corresponding to the image sample. The first detection model was trained in the following manner: The multimodal image samples are input into the initial first detection model to obtain at least two second tokens corresponding to each modality; Extract the third token features of at least two second tokens corresponding to each modality; Based on the third token features of different modalities, at least two first concatenated token features are obtained, and each first concatenated token feature includes at least one third token feature corresponding to a modality; For each first concatenated token feature, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature; The fourth token feature is updated to obtain the updated second preset category token; Based on formula Constrain the initial first detection model; in, This represents the first category of tokens; This represents the set of each of the updated second preset category tokens; express The updated second preset category token mentioned above; Based on the constraint results and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model.

2. The liveness detection method according to claim 1, characterized in that, The step of inputting the image to be identified into the first detection model to obtain the liveness detection result of the target user output by the first detection model includes: The image to be identified is input into the first detection model to obtain at least two first tokens; For each first token, extract the first token feature of the first token; Add a first preset category token to the first token feature to obtain a second token feature; The second token feature is updated to obtain the updated first preset category token; Based on the updated first preset category token, the liveness detection result of the target user is determined.

3. The liveness detection method according to claim 1, characterized in that, The second detection model was trained in the following manner: The multimodal image samples are input into the initial second detection model to obtain at least two third tokens corresponding to each modality; Extract the fifth token features of at least two third tokens corresponding to each modality; The fifth token features of different modalities are concatenated to obtain the second concatenated token feature; Add a third preset category token to the second concatenated token feature to obtain the sixth token feature; Based on the sixth token feature, the first category of tokens is determined; Based on the first category token, the network parameters of the initial second detection model are updated to obtain the second detection model.

4. The liveness detection method according to claim 2, characterized in that, The step of determining the liveness detection result of the target user based on the updated first preset category token includes: Based on the updated first preset category token, determine the liveness detection score of the target user; If the liveness detection score is greater than a preset score, the liveness detection result of the target user is determined to be a live user.

5. The liveness detection method according to any one of claims 1-3, characterized in that, The method further includes: Obtain initial image samples for each modality; A face detection bounding box is determined in the initial image sample, and the face detection bounding box is used to select the face region in the initial image sample; Based on the coordinate information corresponding to the face detection box, the face detection box is expanded to obtain the expanded face detection box; The face image is extracted from the expanded face detection box to obtain a face image sample; The face image sample is scaled to obtain the image sample.

6. A liveness detection device, characterized in that, include: The acquisition module is used to acquire the image of the target user in the target modality. The recognition module is used to input the image to be recognized into a first detection model to obtain the liveness detection result of the target user output by the first detection model. The first detection model is obtained by constraining the initial first detection model based on a first category token determined by a second detection model. The second detection model is trained based on multimodal image samples, and the target modality is a modality in the multimodal model. The first category token is a parameter used by the second detection model to perform liveness detection on the user corresponding to the image sample. The first detection model was trained in the following manner: The multimodal image samples are input into the initial first detection model to obtain at least two second tokens corresponding to each modality; Extract the third token features of at least two second tokens corresponding to each modality; Based on the third token features of different modalities, at least two first concatenated token features are obtained, and each first concatenated token feature includes at least one third token feature corresponding to a modality; For each first concatenated token feature, a second preset category token is added to the first concatenated token feature to obtain a fourth token feature; The fourth token feature is updated to obtain the updated second preset category token; Based on formula Constrain the initial first detection model; in, This represents the first category of tokens; This represents the set of each of the updated second preset category tokens; express The updated second preset category token mentioned above; Based on the constraint results and the updated second preset category token, the network parameters of the initial first detection model are updated to obtain the first detection model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the liveness detection method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the liveness detection method as described in any one of claims 1 to 5.

Images