Liveness detection methods, training methods for liveness detection models, and corresponding devices

Abstract

This application discloses a liveness detection method, a training method for a liveness detection model, and a corresponding apparatus. The main technical solution includes: acquiring a first image and a second image of a target object using a first camera and a second camera; obtaining depth map information based on the disparity between the first image and the second image; extracting image features and depth map features from the first image and the depth map information respectively using a liveness detection model; fusing the image features and depth map features to obtain multimodal fusion features; and using the multimodal fusion features to obtain a detection result indicating whether the target object is live. This application can effectively improve the accuracy of liveness detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a liveness detection method, a training method for a liveness detection model, and a corresponding device. Background Technology

[0002] In recent years, liveness detection has been widely used as a supplement to recognition tasks in various scenarios, such as smart door locks, facial recognition unlocking, facial recognition payment, and facial recognition login for applications. Given that these scenarios often involve important aspects of users' lives, privacy, and finances, accurately implementing liveness detection has become a key challenge for major service providers. Summary of the Invention

[0003] In view of this, this application provides a liveness detection method, a training method for a liveness detection model, and a corresponding device to improve the accuracy of liveness detection.

[0004] This application provides the following solution:

[0005] In a first aspect, a method for detecting liveness is provided, the method comprising:

[0006] Acquire first and second images of the object to be detected captured by the first and second cameras, respectively.

[0007] Depth map information is obtained based on the disparity between the first image and the second image;

[0008] After extracting features from the first image and the depth map information using a liveness detection model to obtain image features and depth map features respectively, the image features and depth map features are fused to obtain multimodal fusion features. The detection result of whether the object to be detected is a live object is obtained using the multimodal fusion features. The liveness detection model is a machine learning model.

[0009] According to one achievable method in an embodiment of this application, obtaining depth map information based on the disparity of the first image and the second image includes:

[0010] Epipolar correction is performed on the first image and the second image using the intrinsic parameter matrix, distortion coefficient matrix and rotation matrix calibrated for the first camera and the second camera respectively.

[0011] Using the first image after epipolar correction as a reference, the parallax between the first and second images at each location is determined;

[0012] The depth map information is obtained by using the parallax of the first and second images at each location, as well as the optical center distance between the first and second cameras, to determine the depth information at each location.

[0013] According to one achievable method in an embodiment of this application, the liveness detection model includes: a feature extraction layer, a feature fusion layer, and a decoding layer;

[0014] The feature extraction layer performs feature extraction on the first image and the depth map information respectively to obtain image features and depth map features;

[0015] The feature fusion layer performs a second merging process on the image features and depth map features to obtain a second fused feature; the second fused feature is then subjected to a first attention process and a second attention process to obtain an image attention matrix and a depth map attention matrix; the image attention matrix and the depth map attention matrix are fused to obtain a multimodal attention matrix; and the multimodal attention matrix is ​​used to perform a third merging process on the image features and the depth map features to obtain a multimodal fused feature.

[0016] The decoding layer uses the multimodal fusion features to obtain the detection result of whether the object to be detected is a living body.

[0017] According to one achievable method in an embodiment of this application, the feature fusion layer performs a second merging process on the image features and depth map features to obtain a second fused feature, including:

[0018] The feature fusion layer performs a first merging process on the image features and depth map features to obtain a first fused feature;

[0019] The first fused feature is processed by a fully connected layer to obtain a multimodal context matrix;

[0020] The image features and the depth map features are subjected to a first enhancement process using the multimodal context matrix to obtain first enhanced image features and first enhanced depth map features;

[0021] The first enhanced image features and the first enhanced depth map features are subjected to a second merging process to obtain the second fused features.

[0022] According to one achievable method in an embodiment of this application, the image features include image features at N resolution scales, and the depth map features include depth map features at N resolution scales, where N is a positive integer greater than 1;

[0023] The process of fusing the image features and depth map features to obtain multimodal fusion features includes fusing image features and depth map features at each resolution scale to obtain multimodal fusion features at N resolution scales.

[0024] Secondly, a user authentication method is provided, which includes:

[0025] The liveness detection method described in the first aspect above is used to perform liveness detection on the object to be detected, and a detection result is obtained to determine whether the object to be detected is a living organism; and

[0026] Perform biometric identification on the object to be detected to obtain a result indicating whether the biometric identification has passed.

[0027] If the object to be detected is a living person and is identified by biometrics, then the user authentication is considered successful.

[0028] Thirdly, a training method for a liveness detection model is provided, the method comprising:

[0029] Acquire training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live; wherein, the image samples include multiple image pairs, each image pair includes a first image and a second image captured by a first camera and a second camera for the same object, and the depth map samples include depth map information obtained based on the disparity between the first image and the second image in each image pair;

[0030] The liveness detection model is trained by taking the first image and depth map samples from the image samples as input and the labels indicating whether the objects in the image samples are live as the target output. The liveness detection model extracts features from the first image and depth map samples respectively to obtain image features and depth map features, then fuses the image features and depth map features to obtain multimodal fusion features, and uses the multimodal fusion features to obtain the detection result of whether the objects in the first image are live. The liveness detection model is a machine learning model.

[0031] According to one achievable method in an embodiment of this application, the liveness detection model includes: a feature extraction layer, a feature fusion layer, and a decoding layer;

[0032] The feature extraction layer performs feature extraction on the first image and the depth map sample respectively to obtain image features and depth map features;

[0033] The feature fusion layer performs a second merging process on the image features and depth map features to obtain a second fused feature; the second fused feature is then subjected to a first attention process and a second attention process to obtain an image attention matrix and a depth map attention matrix; the image attention matrix and the depth map attention matrix are fused to obtain a multimodal attention matrix; and the multimodal attention matrix is ​​used to perform a third merging process on the image features and the depth map features to obtain the multimodal fused feature.

[0034] The decoding layer uses the multimodal fusion features to obtain the detection result of whether the object in the first image is a live object.

[0035] According to one achievable method in an embodiment of this application, the feature fusion layer performs a second merging process on the image features and depth map features to obtain a second fused feature, including:

[0036] The feature fusion layer performs a first merging process on the image features and depth map features to obtain a first fused feature;

[0037] The first fused feature is processed by a fully connected layer to obtain a multimodal context matrix;

[0038] The image features and the depth map features are subjected to a first enhancement process using the multimodal context matrix to obtain first enhanced image features and first enhanced depth map features;

[0039] The first enhanced image features and the first enhanced depth map features are subjected to a second merging process to obtain the second fused features.

[0040] According to one achievable method in an embodiment of this application, the feature fusion layer further utilizes the multimodal fusion features to perform a second enhancement process on the image features and the depth map features to obtain second enhanced image features and second enhanced depth map features;

[0041] The decoding layer includes an image encoder, a depth encoder, and a fusion encoder;

[0042] The image encoder uses the second enhanced image features to perform binary classification and obtain a first detection result as to whether the object in the first image is a living body.

[0043] The depth encoder uses the second enhanced depth map features to perform binary classification and obtain a second detection result as to whether the object in the first image is a living body.

[0044] The fusion encoder uses the multimodal fusion features to obtain a third detection result as to whether the object in the first image is a live object;

[0045] The training objective is to minimize the value of the total loss function, which is determined by a first loss function value, a second loss function value, and a third loss function value. The first loss function value is determined by the difference between the first detection result and the label indicating whether an object in the image sample is a live object. The second loss function value is determined by the difference between the second detection result and the label indicating whether an object in the image sample is a live object. The third loss function value is determined by the difference between the third detection result and the label indicating whether an object in the image sample is a live object.

[0046] After training is completed, the decoding layer retains only the fusion encoder.

[0047] According to one achievable method in an embodiment of this application, the image features include image features at N resolution scales, and the depth map features include depth map features at N resolution scales, where N is a positive integer greater than 1;

[0048] The process of fusing the image features and depth map features to obtain multimodal fusion features includes fusing image features and depth map features at each resolution scale to obtain multimodal fusion features at N resolution scales.

[0049] Fourthly, a liveness detection device is provided, the device comprising:

[0050] The image acquisition unit is configured to acquire a first image and a second image captured by a first camera and a second camera for the object to be detected.

[0051] The depth acquisition unit is configured to acquire depth map information based on the disparity between the first image and the second image;

[0052] The liveness detection unit is configured to extract image features and depth map features from the first image and the depth map information respectively through a liveness detection model, then fuse the image features and depth map features to obtain multimodal fusion features, and use the multimodal fusion features to obtain the detection result of whether the object to be detected is a live object, wherein the liveness detection model is a machine learning model.

[0053] Fifthly, a training device for a liveness detection model is provided, the device comprising:

[0054] The sample acquisition unit is configured to acquire training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live. The image samples include multiple image pairs, each image pair including one of a first image and a second image acquired by a first camera and a second camera for the same object. The depth map samples include depth map information obtained based on the disparity between the first image and the second image in each image pair.

[0055] The model training unit is configured to take a first image and a depth map sample from the image samples as input, and take the label indicating whether an object in the image samples is a live object as the target output to train a liveness detection model; wherein, the liveness detection model extracts features from the first image and the depth map sample respectively to obtain image features and depth map features, then fuses the image features and depth map features to obtain multimodal fusion features, and uses the multimodal fusion features to obtain the detection result of whether the object in the first image is a live object, wherein the liveness detection model is a machine learning model.

[0056] According to a sixth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0057] According to a seventh aspect, an electronic device is provided, characterized in that it comprises:

[0058] One or more processors; and

[0059] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0060] According to the specific embodiments provided in this application, this application can achieve the following technical effects:

[0061] 1) This application uses two cameras to acquire images, and then uses the parallax between the images to obtain depth map information. The depth map information is used as a supplement to the image and input into the liveness detection model to perform multimodal feature fusion for liveness detection. Compared with the method of using images for single-modal detection, this improves the accuracy of liveness detection.

[0062] 2) The technical solution provided in this application only requires the use of a camera for image acquisition, without the need to deploy expensive sensors such as structured light sensors and motion sensors, which can effectively reduce the size of the device and reduce the cost, making it convenient to deploy on terminal devices.

[0063] 3) In this application, the calibration results of the camera are used to perform epipolar correction on the first image and the second image. Based on the epipolar corrected image, the disparity is calculated and depth map information is further obtained, so that the obtained depth map information is more accurate, and the liveness detection based on it is also more accurate.

[0064] 4) In this application, when extracting features from the first image and depth map information, features at N resolution scales are extracted. This "multi-level" feature extraction method enables the multi-mode fusion features obtained by subsequent fusion to better capture effective depth map features and image features, thereby improving the model detection effect.

[0065] 5) During the training of the liveness detection model, the second enhanced image features, the second enhanced depth map features, and the multimodal fusion features are input into the image encoder, the depth map encoder, and the fusion encoder, respectively, to obtain three detection results. The loss function values ​​obtained from these three detection results are then used to train the model. After training, the encoding layer retains only the fusion encoder, minimizing model parameters and computational load while achieving high recognition accuracy, thus shortening prediction time.

[0066] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is an illustrative system architecture diagram that can be applied to the embodiments of this application;

[0069] Figure 2 This is a flowchart of a liveness detection method provided in an embodiment of this application;

[0070] Figure 3 A flowchart illustrating a method for obtaining depth map information provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram illustrating the principle of the liveness detection model provided in the embodiments of this application;

[0072] Figure 5 This is a schematic diagram of the feature fusion layer provided in the embodiments of this application;

[0073] Figure 6 A flowchart illustrating the method for establishing a liveness detection model provided in this application embodiment;

[0074] Figure 7 A schematic diagram illustrating the principle of training a liveness detection model provided in an embodiment of this application;

[0075] Figure 8A schematic block diagram of a liveness detection device according to one embodiment is shown;

[0076] Figure 9 A schematic block diagram of an apparatus for establishing a liveness detection model according to one embodiment is shown;

[0077] Figure 10 This is an architectural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0079] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0080] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0081] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0082] Traditional liveness detection technologies mostly require user cooperation. For example, when collecting user image information, the user needs to perform actions such as opening their mouth, blinking, and turning their head according to instructions. This method requires the user to perform an extra series of actions, takes a long time to detect, and has a poor overall user experience. With the rapid development of deep learning technology, most liveness detection technologies based on traditional methods have been replaced by deep learning models. Currently, the main implementation methods are as follows:

[0083] The first method involves using the single-modal information, such as images captured by a single camera, as input to a liveness detection model. The liveness detection model then analyzes the images to determine whether the object to be detected in the image is a live object.

[0084] The second approach uses images captured by a single camera and information from other sensors as input to a liveness detection model. This multimodal input is then analyzed to determine whether the target object is alive. While this multimodal method offers higher accuracy compared to single-modal methods, it often requires the deployment of expensive additional sensors, such as structured light sensors and motion sensors. These sensors increase device size and cost, making them less suitable for deployment on end devices.

[0085] In view of this, this application provides a novel approach to achieve liveness detection. To facilitate understanding of the embodiments of this application, a brief description of the system architecture applicable to this application is first provided. Figure 1 This is an illustrative system architecture that can be applied to the embodiments of this application. For example... Figure 1 As shown, the system architecture may include terminal devices 101 and 102, a network 103, and a server 104. Network 103 provides a communication link between terminal devices 101, 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0086] Users can interact with server 104 via network 103 using terminal devices 101 and 102. Various applications can be installed on terminal devices 101 and 102, and one or more of these applications need to work with liveness detection technology to complete user authentication.

[0087] Taking terminal device 101 as an example, a binocular camera is deployed on this device to collect user images. As one possible implementation, the images collected by the binocular camera on terminal device 101 can be sent to server 104. A liveness detection model is pre-trained on server 104. After receiving the user images sent by terminal device 101, server 104 uses the liveness detection model to perform liveness detection. The liveness detection device provided in this embodiment can be located in server 104, and it can be implemented as multiple software programs or software modules (e.g., to provide distributed services), or as a single software program or software module; no specific limitation is made here.

[0088] As another possible approach, server 104 trains a liveness detection model and distributes it to terminal device 101 for deployment. After the binocular camera deployed on terminal device 101 captures the user's image, it is input into the liveness detection model for liveness detection. In this case, the liveness detection device provided in this embodiment is located in terminal device 101, and it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0089] Terminal devices 101 and 102 may include, but are not limited to, smart mobile terminals, smart home devices, network devices, wearable devices, smart medical devices, and PCs (Personal Computers). Smart mobile devices may include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and connected cars. Smart home devices may include smart appliances such as smart TVs, smart air conditioners, and smart refrigerators; they may also include smart door locks and smart cameras. Network devices may include switches, wireless access points, and servers. Wearable devices may include smartwatches, smart glasses, smart bracelets, virtual reality devices, augmented reality devices, and mixed reality devices (devices that support both virtual and augmented reality). Smart medical devices may include smart thermometers, smart blood pressure monitors, and smart blood glucose meters.

[0090] Server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability.

[0091] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0092] Figure 2 This is a flowchart of the liveness detection method provided in the embodiments of this application, such as... Figure 2 As shown, the method may include the following steps:

[0093] Step 202: Acquire the first image and the second image captured by the first camera and the second camera for the object to be detected.

[0094] Step 204: Obtain depth map information based on the disparity between the first image and the second image.

[0095] Step 206: After extracting features from the first image and depth map information using the liveness detection model to obtain image features and depth map features respectively, the image features and depth map features are fused to obtain multimodal fusion features. The detection result of whether the object to be detected is a live object is obtained using the multimodal fusion features.

[0096] As can be seen from the above process, this application uses two cameras to acquire images, then uses the parallax between the images to obtain depth map information, and uses the depth map information as a supplement to the images to input into the liveness detection model for multimodal feature fusion liveness detection. Compared with the method of using images for single-modal detection, this improves the accuracy of liveness detection.

[0097] In addition, this technical solution only requires the use of a camera for image acquisition, without the need to deploy expensive sensors such as structured light sensors and motion sensors, which can effectively reduce the size of the device and reduce costs.

[0098] The following is a detailed description of each of the above steps.

[0099] First, the above step 202, namely "acquiring the first image and the second image captured by the first camera and the second camera for the object to be detected", will be described in detail with reference to the embodiments.

[0100] One feasible approach is to deploy two independent cameras, positioned relatively fixedly, to capture images of the same object to be detected.

[0101] As another feasible approach, a binocular camera can be used to acquire images of the object to be inspected.

[0102] In this embodiment, the first and second cameras can be positioned appropriately according to the actual application scenario of liveness detection. For example, they can be placed on a smart door lock to acquire user images for liveness detection to assist user authentication, and unlock the door after successful authentication. Another example is a dual-lens camera on the front-facing camera of a mobile phone to acquire user images for liveness detection when the screen is locked, and unlock the screen after successful authentication. Yet another example is a dual-lens camera on a self-service payment device to acquire user images for liveness detection during facial recognition payment, and complete payment after successful authentication. Other scenarios are not listed here.

[0103] The following describes step 204, namely "obtaining depth map information based on the disparity of the first image and the second image", in detail with reference to the embodiments.

[0104] As one possible approach, this step can be implemented using... Figure 3 The process shown is implemented as follows: Figure 3 As shown, it may specifically include the following steps:

[0105] Step 302: Using the intrinsic parameter matrix, distortion coefficient matrix and rotation matrix calibrated for the first camera and the second camera respectively, perform epipolar correction on the first image and the second image.

[0106] Due to limitations in camera manufacturing processes, it is impossible to guarantee that the first and second images captured by the two cameras are perfectly horizontal, which affects the accuracy of parallax calculation. Therefore, it is necessary to calibrate the two cameras and then perform epipolar correction on the obtained first and second images.

[0107] To determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in an image, a geometric model of the camera imaging must be established. These geometric model parameters are the camera parameters, and the process of determining these parameters is the calibration process. Camera parameter calibration is a relatively mature technique, and methods such as the Zhang Zhengyou calibration method can be used; these will not be detailed here.

[0108] After camera calibration, we can obtain the 3×3 intrinsic parameter matrices N1 and N2 of the first and second cameras, the 1×4 distortion coefficient matrices D1 and D2, the 3×3 rotation matrices R1 and R2, and the extrinsic parameters between the two cameras, such as the translation vector T.

[0109] Epipolar correction can employ any existing epipolar correction method, with the goal of ensuring that the optical axes of the two cameras are parallel after correction, and that the projection height of the same target point on the left and right camera planes is consistent, i.e., horizontal. Examples include the Fusiello algorithm and the Bouguet algorithm.

[0110] First, distortion correction is performed, including: using the internal parameters of each camera to transform the first and second images into the world coordinate system; then, distortion correction is performed on the transformed first and second images using distortion coefficient matrices D1 and D2 respectively, to obtain the first and second images after distortion correction.

[0111] Then, epipolar correction is performed on the first and second images after distortion correction, using methods such as the Bouguet algorithm as follows:

[0112] To achieve row alignment, the first and second cameras need to rotate around the optical axis according to R1 and R2 respectively. Taking the first camera as an example, this step requires constructing a transformation matrix R1. rec This causes the first camera to rotate around the optical axis. This can be achieved with respect to R1. rec Perform coordinate decomposition:

[0113] R1 rec =[e1 e2 e3] (1)

[0114]

[0115] e3 = e1 × e2 (4)

[0116] Where |||| denotes the modulus of a vector, [] T T represents transpose, × represents the cross product of vectors, and T represents the transpose. x T y and T z This represents the components of T in the X, Y, and Z axes.

[0117] Then, the rotation matrix and the transformation matrix are multiplied to obtain the overall rotation matrix R1' of the first camera:

[0118] R1'=R1 rec *R1 (5)

[0119] The overall rotation matrix R2' of the second camera is obtained in a similar manner.

[0120] Then, the first image and the second image are rotated using the overall rotation matrix R1' of the first camera and the overall rotation matrix R2' of the second camera, respectively, to obtain coplanar and row-aligned images, namely the first image and the second image after epipolar correction.

[0121] Step 304: Using the first image after epipolar correction as a reference, determine the parallax between the first image and the second image at each location.

[0122] After epipolar correction, the corresponding pixels of the first and second images are perfectly horizontal. Using the first image as a reference, the following steps are performed for each position in the first image: An image patch is taken at the current position in the first image, and starting at the same position in the second image, an image patch of the same size is slid along a preset direction. The similarity between each slidable image patch and the image patch in the first image is calculated. If the similarity is maximized after sliding disp pixels, then the disparity at the current position in the first image is disp. The first image can be either an image captured by the first camera or the second camera targeting the same object. For a binocular camera, if the first image is a left view, the preset direction can be from left to right; if the first image is a right view, the preset direction can be from right to left.

[0123] Step 306: Using the parallax of the first and second images at each location and the optical center distance between the first and second cameras, the depth information at each location is determined to obtain depth map information.

[0124] The depth information (Depth) at each location can be obtained using the following formula:

[0125]

[0126] Where fx is the focal length of the camera on the X-axis, which is parallel to the horizontal direction, and b is the distance between the optical centers of the first and second cameras.

[0127] After obtaining the depth information at each location, depth map information can be derived from this depth information. The depth map information can include depth information and its corresponding location information, or a sequence or matrix of depth information arranged according to location.

[0128] Analysis of numerous depth maps corresponding to live and prosthetic subjects revealed a significant difference in their depth map representations. Depth maps of prosthetic subjects exhibited either a relatively uniform depth or a chaotic appearance, while depth maps of live subjects displayed hierarchy and the object's outline. Therefore, this embodiment of the application uses depth map information as a modal input to the liveness detection model.

[0129] The following describes in detail step 206, namely, "after extracting features from the first image and depth map information using the liveness detection model to obtain image features and depth map features, the image features and depth map features are fused to obtain multimodal fusion features, and the detection result of whether the object to be detected is a live object is obtained using the multimodal fusion features," with reference to the embodiments.

[0130] In this step, the liveness detection model is input with multimodal information, that is, depth map information is used as a supplement to the first image and input into the liveness detection model for multimodal feature fusion liveness detection. The liveness detection model provided in this application embodiment can employ a machine learning model, such as... Figure 4 As shown, it includes a feature extraction layer, a feature fusion layer, and a decoding layer.

[0131] The feature extraction layer is used to extract features from the first image and the depth map information respectively, so as to obtain image features and depth map features.

[0132] The feature extraction layer can include two feature extraction networks. These two networks have the same structure but do not share network parameters. One feature extraction network extracts features from the first image to obtain image features. The other feature extraction network extracts features from the depth map to obtain depth map features.

[0133] Each feature extraction layer mainly consists of two parts:

[0134] The first part of the network preprocesses the input (i.e., the first image or depth map information). This preprocessing mainly involves simple abstraction of the input and extraction of preliminary features. One possible implementation is a convolutional module, a batch normalization module, an activation module, and a max-pooling module. The activation module can employ activation functions such as ReLU.

[0135] The second part of the network further extracts features from the initial features, obtaining image features (Img) and depth map features (Nlcd). One possible implementation could include, in sequence, a convolutional module with a kernel of 1, a batch normalization module, an activation module, a convolutional module with a kernel of 3, a batch normalization module, an activation module, a convolutional module with a kernel of 1, a batch normalization module, and an activation module, where the activation module can employ an activation function such as ReLU.

[0136] In a more preferred implementation, the second part of the network is an FPN (Feature Pyramid Network). After feature extraction, image features and depth map features corresponding to N resolution scales can be obtained, where N is a positive integer greater than 1. Taking N=3 as an example, this part can be obtained by stacking 3 lightweight blocks to extract features at 3 different resolution scales. As one possible implementation, each lightweight block can sequentially contain a convolutional module with a kernel of 1, a batch normalization module, an activation module, a convolutional module with a kernel of 3, a batch normalization module, an activation module, a convolutional module with a kernel of 1, a batch normalization module, and an activation module, where the activation module can use an activation function such as ReLU. The obtained image features at the 3 resolution scales can be represented as Img1, Img2, and Img3, and the obtained depth map features at the 3 resolution scales can be represented as Nlcd1, Nlcd2, and Nlcd3.

[0137] The following is combined Figure 5 The implementation process of the feature fusion layer is described in detail.

[0138] If the feature extraction layer targets image features and depth map features at only one resolution scale, the feature fusion layer fuses these two features to obtain multimodal fused features. If the feature extraction layer targets image features and depth map features at N resolution scales, the feature extraction layer fuses the image features and depth map features at each resolution scale separately to obtain N multimodal fused features.

[0139] The feature fusion layer first performs a second merging process on the image features and depth map features to obtain the second fused feature. One possible approach is to directly merge the image features and depth map features to obtain the aforementioned second fused feature.

[0140] However, as an alternative approach, the feature fusion layer described above performs a second merging process on image features and depth map features to obtain the second fused feature, which can specifically include the following steps:

[0141] like Figure 5 As shown, the feature fusion layer can first perform a first merging process on the image features (denoted as Img) and the depth map features (denoted as Nlcd) to obtain the first fused feature I, which can be represented as:

[0142] I = F gp (Img||Nlcd) (7)

[0143] Among them, F gp () indicates global average pooling, and || indicates concatenation.

[0144] Then, the first fused feature I is processed by a fully connected layer to obtain the multimodal context matrix. In this embodiment, the first fully connected layer can be processed separately to obtain the multimodal context matrix W for the image features. img A second fully connected processing step is performed to obtain the multimodal context matrix W for the depth map features. nlcd .

[0145] W img =σ(F mlp1 (I)) (8)

[0146] W nlcd =σ(F mlp2 (I)) (9)

[0147] F mlp1 () and F mlp2 () represents two fully connected layers with the same structure but no shared parameters, and σ() represents the activation function.

[0148] The above fully connected processing can also be applied to the first fused feature I to obtain a unified multimodal context matrix for image features and depth map features.

[0149] Then, the image feature Img and the depth map feature Nlcd are subjected to a first enhancement process using the multimodal context matrix, respectively, to obtain the first enhanced image feature Img. rec and the first enhanced depth map feature Nlcd rec This can be represented as:

[0150] Img rec =Img*W img +Img

[0151] Nlcd rec =Nlcd*Wnlcd +Nlcd

[0152] Finally, the first enhanced image feature Img was analyzed. rec and the first enhanced depth map feature Nlcd rec Perform a second merging process to obtain the second fusion feature I. rec This can be represented as:

[0153] I rec = (Img) rec ||Nlcd rec (10)

[0154] The second fusion feature I is obtained by using the above process. rec Then, the feature fusion layer can process the second fused feature I. rec The image attention matrix Att is obtained by performing the first attention process and the second attention process respectively. img Att of the depth map attention matrix nlcd This can be represented as:

[0155] Att img =F conv1 (I rec (11)

[0156] Att nlcd =F conv2 (I rec (12)

[0157] F conv1 () and F conv2 () indicates the processing of two convolutional layers with the same structure but not shared parameters.

[0158] Then, the feature fusion layer will apply the image attention matrix Att img Att of the depth map attention matrix nlcd The multimodal attention matrix Att is obtained by fusion. It can be represented as:

[0159] Att = softmax(Att) img ||Att nlcd (13)

[0160] softmax() is the softmax function.

[0161] The feature fusion layer then uses the multimodal attention matrix Att to perform a third merging process on the image feature Img and the depth map feature Nlcd, respectively, to obtain the multimodal fused feature M. This can be represented as:

[0162] M=Img*Att[:,0:1]+Nlcd*Att[:,1:2] (14)

[0163] Att[:,0:1] represents the method of slicing Att, which means taking all data in the first dimension of Att, taking the 0th data in the second dimension, and taking the 1st data in the third dimension. Att[:,1:2] means taking all data in the first dimension of Att, taking the 1st data in the second dimension, and taking the 2nd data in the third dimension.

[0164] See also Figure 4 The decoding layer uses multimodal fusion features to obtain the detection result of whether the object to be detected is a live object.

[0165] As one possible approach, the decoding layer can perform binary classification using only multimodal fusion features to obtain a detection result on whether the object to be detected is a live object.

[0166] If N multimodal fusion features are obtained based on image features and depth map features at N resolution scales, then all N multimodal fusion features are input into the decoding layer, and the decoding layer uses these N multimodal fusion features to perform binary classification.

[0167] As another feasible approach, the feature fusion layer can further utilize multimodal fusion features M to perform a second enhancement process on image features and depth features, resulting in a second enhanced image feature Img. new Second Enhanced Depth Map Features Nlcd new This can be represented as:

[0168] Img new = (Img + M) / 2 (15)

[0169] Nlcd new =(Nlcd+M) / 2 (16)

[0170] The decoding layer can perform binary classification using multimodal fusion features, second enhanced image features, and second enhanced depth map features, respectively, and then combine the results of each binary classification to obtain the detection result of whether the object to be detected is a live object. For example, the results of each binary classification can be used for voting or comprehensive scoring, and the detection result of whether the object to be detected is a live object can be determined based on the voting result or the comprehensive score result.

[0171] Similarly, if there are N image features and depth map features at N resolution scales, then N second enhanced image features and second depth map image features are obtained accordingly. Then, the N second enhanced image features are used for binary classification, and the N second depth map features are used for binary classification.

[0172] After testing, it was found that the detection results obtained by binary classification using only multimodal fusion features in the decoding layer can achieve a 90% pass rate for live test samples and a 99.969% interception rate for spurious test samples. Moreover, this method has fewer model parameters, less computation, and can reduce inference time. Therefore, the method of obtaining detection results by binary classification using only multimodal fusion features is preferred.

[0173] The above describes the liveness detection process using a liveness detection model, which can be executed online. The liveness detection model can be pre-built offline. Figure 6 A flowchart illustrating the method for establishing a liveness detection model provided in this application embodiment is shown below. Figure 6 As shown, the method may include the following steps:

[0174] Step 602: Obtain training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live. The image samples include multiple image pairs, and each image pair includes a first image and a second image of the same object captured by the first camera and the second camera. The depth map samples include depth map information obtained based on the disparity between the first image and the second image in each image pair.

[0175] In this embodiment, a first camera and a second camera can be used to capture images of the same object, resulting in a first image and a second image. The same object is either a known living body or a known prosthesis. Then, using the intrinsic parameter matrix, distortion coefficient matrix, and rotation matrix calibrated for the first and second cameras respectively, epipolar correction is performed on the first and second images. The epipolar-corrected first image is then used as a reference to determine the disparity between the first and second images at various locations. Finally, using the disparity between the first and second images at various locations and the optical center distance between the first and second cameras, depth information is determined at each location to obtain depth map information. For specific determination methods, please refer to [link to relevant documentation]. Figure 3 The method shown in the example will not be described in detail here. This method can generate a large number of samples and labels as training data.

[0176] Step 604: Take the first image and depth map sample in the image sample as input, and take the label of whether the object in the image sample is a live object as the target output to train the liveness detection model; wherein, the liveness detection model extracts features from the first image and depth map sample respectively to obtain image features and depth map features, then fuses the image features and depth map features to obtain multimodal fusion features, and uses the multimodal fusion features to obtain the detection result of whether the object in the first image is a live object.

[0177] The structure of the liveness detection model can be as follows: Figure 7As shown, it includes: a feature extraction layer, a feature fusion layer, and a decoding layer.

[0178] The feature extraction layer extracts features from the first image and the depth map sample respectively, and obtains image features and depth map features.

[0179] The feature fusion layer performs a second merging process on image features and depth map features to obtain a second fused feature; the second fused feature is then subjected to a first attention process and a second attention process to obtain an image attention matrix and a depth map attention matrix; the image attention matrix and the depth map attention matrix are fused to obtain a multimodal attention matrix; the multimodal attention matrix is ​​used to perform a second enhancement process on image features and depth map features, and the second enhanced image features and the second enhanced depth map features are then subjected to a third merging process to obtain the multimodal fused feature.

[0180] The decoding layer uses multimodal fusion features to obtain the detection result of whether the object in the first image is a live object.

[0181] As one possible approach, when the feature fusion layer performs a second merging process on image features and depth map features to obtain a second fused feature, it can perform a first merging process on image features and depth map features to obtain a first fused feature; perform a fully connected process on the first fused feature to obtain a multimodal context matrix; use the multimodal context matrix to perform a first enhancement process on image features and depth map features respectively to obtain a first enhanced image feature and a first enhanced depth map feature; and perform a second merging process on the first enhanced image feature and the first enhanced depth map feature to obtain the second fused feature.

[0182] Furthermore, the feature fusion layer can utilize multimodal fusion features to perform a second enhancement process on image features and depth features, resulting in second enhanced image features and second enhanced depth map features.

[0183] During model training, such as Figure 7 As shown, the decoding layer includes an image encoder, a depth encoder, and a fusion encoder.

[0184] The image encoder utilizes the second enhanced image feature Img new Perform binary classification to obtain the first detection result D of whether the object in the first image is a live object. img This can be represented as:

[0185] D img =Decode1(Img new (17)

[0186] Decode1() is the processing function used by the image encoder.

[0187] The depth encoder utilizes second enhanced depth map features Nlcd new Perform binary classification to obtain a second detection result D: whether the object in the first image is a live object. nlcd This can be represented as:

[0188] D nlcd =Decode2(Nlcd new (18)

[0189] Decode2() is the processing function used by the depth encoder.

[0190] The fusion encoder uses multimodal fusion features M to perform binary classification, obtaining a third detection result D to determine whether the object in the first image is a live object. m This can be represented as:

[0191] D m =Decode3(D m (19)

[0192] Decode3() is the processing function used by the fusion encoder.

[0193] The training objective is to minimize the value of the total loss function L, which is determined by the values ​​of the first, second, and third loss functions, and can be expressed as:

[0194] L = w1*L1 + w2*L2 + w3*L3

[0195] Among them, w1, w2 and w3 are weighting coefficients, which can be set to w3>w1>w2 according to their importance.

[0196] The first loss function L1 is determined by the difference between the first detection result and the label y indicating whether an object in the image sample is a live object. The second loss function L2 is determined by the difference between the second detection result and the label y indicating whether an object in the image sample is a live object. The third loss function L3 is determined by the difference between the third detection result and the label y indicating whether an object in the image sample is a live object. These can be expressed as follows:

[0197] L1 = CE(D img ,y) (20)

[0198] L2=CE(D nlcd ,y) (21)

[0199] L3=CE(D m ,y) (22)

[0200] CE() represents the cross-entropy loss function.

[0201] In each iteration, the model parameters of the liveness detection model are updated using methods such as gradient descent, based on the value of the total loss function, until a preset training termination condition is met. This termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0202] As an feasible approach, the decoding layer can retain the fusion encoder, image encoder, and depth encoder after training.

[0203] After testing, it was found that the detection results obtained by the decoding layer using only multimodal fusion features for binary classification could achieve a 90% pass rate for live test samples and a 99.969% interception rate for spurious test samples. Furthermore, this method has fewer model parameters, lower computational cost, and can shorten prediction time. Therefore, as a preferred implementation, after training, the decoding layer can retain only the fusion encoder and delete the image encoder and depth encoder.

[0204] The liveness detection method provided in this application can be widely applied to user authentication in various scenarios. Besides performing liveness detection on the target object as described in the above embodiments, it also performs biometric identification on the target object to obtain a result indicating whether the biometric identification has been successful. If the target object is live and passes biometric identification, then user authentication is confirmed to be successful. Several application scenarios applicable to this application embodiment are listed below, which may include, but are not limited to:

[0205] Scenario 1: A binocular camera is installed on a smart door lock. When a person opens the door, the method provided in this application provides to acquire a first image and a second image captured by the binocular camera. The parallax of the two images is then obtained, and depth map information is further acquired. The first image and depth map information are then input into a liveness detection model to obtain a detection result indicating whether the object in the first image is a live person. This detection result can be used in conjunction with biometric recognition technologies such as face recognition, fingerprint recognition, and iris recognition in the smart door lock to perform user authentication. For example, if face recognition identifies the user as authenticating and the detection result is also live, the user is confirmed to be authenticating and the door lock is opened. If either face recognition or liveness detection fails, the door lock cannot be opened, thus preventing unauthorized users from opening the door by holding a photo of the authenticating user.

[0206] Scenario 2: The phone's front-facing camera is set to a dual-lens camera. After acquiring a face image through the front-facing camera, face recognition is performed. Simultaneously, using the method provided in this application embodiment, a first image and a second image captured by the dual-lens camera are obtained. The disparity between the two images is then calculated, and depth map information is further acquired. The first image and depth map information are then input into a liveness detection model to obtain a detection result indicating whether the object in the first image is a live subject. The phone screen can only unlock if face recognition passes and a live subject is detected. If either face recognition or liveness detection fails, the screen cannot be unlocked, thus preventing unauthorized users from unlocking the phone screen by holding a photo of the authenticated user.

[0207] Scenario 3: A binocular camera is installed on the self-service payment device. When the payment function is triggered, facial recognition is performed. Simultaneously, the method provided in this application embodiment acquires a first image and a second image captured by the binocular camera. The disparity between the two images is then obtained, and depth map information is further acquired. The first image and depth map information are then input into a liveness detection model to obtain a detection result indicating whether the object in the first image is a live object. Payment can only be completed if facial recognition passes and a live object is detected. If either facial recognition or liveness detection fails, payment cannot be processed. This prevents unauthorized users from using photos of verified users to make payments.

[0208] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0209] According to another embodiment, a liveness detection device is provided. Figure 8 A schematic block diagram of a liveness detection device according to one embodiment is shown. Figure 8 As shown, the device 800 includes: an image acquisition unit 801, a depth acquisition unit 802, and a liveness detection unit 803. The main functions of each component are as follows:

[0210] The image acquisition unit 801 is configured to acquire a first image and a second image captured by a first camera and a second camera for the object to be detected.

[0211] The depth acquisition unit 802 is configured to acquire depth map information based on the disparity between the first image and the second image.

[0212] The liveness detection unit 803 is configured to extract image features and depth map features from the first image and depth map information respectively through a liveness detection model, then fuse the image features and depth map features to obtain multimodal fusion features, and use the multimodal fusion features to obtain the detection result of whether the object to be detected is a live object. The liveness detection model is a machine learning model.

[0213] As one possible approach, the first and second cameras mentioned above can be binocular cameras.

[0214] As one possible implementation method, the depth acquisition unit 802 can be specifically configured to: perform epipolar correction on the first image and the second image using the intrinsic parameter matrix, distortion coefficient matrix and rotation matrix calibrated for the first camera and the second camera respectively; use the epipolar corrected first image as a reference to determine the disparity between the first image and the second image at each position; and use the disparity between the first image and the second image at each position and the optical center distance between the first camera and the second camera to determine the depth information at each position to obtain depth map information.

[0215] The liveness detection model includes a feature extraction layer, a feature fusion layer, and a decoding layer.

[0216] The feature extraction layer extracts features from the first image and the depth map information respectively to obtain image features and depth map features;

[0217] The feature fusion layer performs a second merging process on image features and depth map features to obtain a second fused feature; the second fused feature is then subjected to a first attention process and a second attention process to obtain an image attention matrix and a depth map attention matrix; the image attention matrix and the depth map attention matrix are fused to obtain a multimodal attention matrix; the multimodal attention matrix is ​​then used to perform a third merging process on image features and depth map features to obtain a multimodal fused feature.

[0218] The decoding layer uses multimodal fusion features to obtain the detection result of whether the object to be detected is a live object.

[0219] As one possible approach, when the feature fusion layer performs a second merging process on image features and depth map features to obtain the second fused feature, it can specifically perform the following:

[0220] The first merging process is performed on the image features and the depth map features to obtain the first fused features;

[0221] The first fused feature is processed by a fully connected layer to obtain a multimodal context matrix;

[0222] The image features and depth map features are enhanced using a multimodal context matrix to obtain the first enhanced image features and the first enhanced depth map features.

[0223] The first enhanced image features and the first enhanced depth map features are subjected to a second merging process to obtain the second fused features.

[0224] As one possible approach, image features can include image features at N resolution scales, and depth map features can include depth map features at N resolution scales, where N is a positive integer greater than 1.

[0225] Accordingly, fusing image features and depth map features to obtain multimodal fusion features includes fusing image features and depth map features at each resolution scale to obtain multimodal fusion features at N resolution scales.

[0226] For the principles of the liveness detection model, please refer to the method embodiments regarding... Figure 4 and Figure 5 The relevant records will not be elaborated here.

[0227] According to another embodiment, a training apparatus for a liveness detection model is provided. Figure 9 A schematic block diagram of a training apparatus for a liveness detection model according to one embodiment is shown. Figure 9 As shown, the device 900 includes a sample acquisition unit 901 and a model training unit 902. The main functions of each component are as follows:

[0228] The sample acquisition unit 901 is configured to acquire training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live. The image samples include multiple image pairs, each image pair including a first image and a second image acquired by a first camera and a second camera for the same object. The depth map samples include depth map information acquired based on the disparity between the first image and the second image in each image pair.

[0229] The model training unit 902 is configured to take the first image and depth map samples from the image samples as input, and the label of whether the object in the image sample is a live object as the target output to train the liveness detection model. The liveness detection model extracts features from the first image and depth map samples respectively to obtain image features and depth map features, then fuses the image features and depth map features to obtain multimodal fusion features, and uses the multimodal fusion features to obtain the detection result of whether the object in the first image is a live object.

[0230] Specifically, a liveness detection model may include a feature extraction layer, a feature fusion layer, and a decoding layer.

[0231] The feature extraction layer extracts features from the first image and the depth map sample respectively, and obtains image features and depth map features.

[0232] The feature fusion layer performs a second merging process on image features and depth map features to obtain a second fused feature; the second fused feature is then subjected to a first attention process and a second attention process to obtain an image attention matrix and a depth map attention matrix; the image attention matrix and the depth map attention matrix are fused to obtain a multimodal attention matrix; the multimodal attention matrix is ​​then used to perform a third merging process on image features and depth map features to obtain a multimodal fused feature.

[0233] The decoding layer uses multimodal fusion features to obtain the detection result of whether the object in the first image is a live object.

[0234] As one possible approach, when the feature fusion layer performs a second merging process on image features and depth map features to obtain the second fused feature, it can perform the following:

[0235] The feature fusion layer performs a first merging process on image features and depth map features to obtain the first fused feature;

[0236] The first fused feature is processed by a fully connected layer to obtain a multimodal context matrix;

[0237] The image features and depth map features are enhanced using a multimodal context matrix to obtain the first enhanced image features and the first enhanced depth map features.

[0238] The first enhanced image features and the first enhanced depth map features are subjected to a second merging process to obtain the second fused features.

[0239] Furthermore, the feature fusion layer can utilize multimodal fusion features to perform a second enhancement process on image features and depth map features, resulting in second enhanced image features and second enhanced depth map features.

[0240] The decoding layer includes an image encoder, a depth encoder, and a fusion encoder.

[0241] The image encoder uses the second enhanced image features to perform binary classification and obtain a first detection result as to whether the object in the first image is a living body.

[0242] The depth encoder uses the second enhanced depth map features to perform binary classification, obtaining a second detection result as to whether the object in the first image is a live object.

[0243] The fusion encoder uses multimodal fusion features to obtain a third detection result: whether the object in the first image is a live object.

[0244] The training objective is to minimize the value of the total loss function, which is determined by the first loss function value, the second loss function value, and the third loss function value. The first loss function value is determined by the difference between the first detection result and the label indicating whether the object in the image sample is alive. The second loss function value is determined by the difference between the second detection result and the label indicating whether the object in the image sample is alive. The third loss function value is determined by the difference between the third detection result and the label indicating whether the object in the image sample is alive.

[0245] After training, the decoding layer can retain only the fusion encoder. Alternatively, it can retain the image encoder, depth encoder, and fusion encoder. However, considering factors such as model performance, number of parameters, and computational cost, retaining only the fusion encoder is preferred.

[0246] As one possible approach, the image features include image features at N resolution scales, and the depth map features include depth map features at N resolution scales, where N is a positive integer greater than 1.

[0247] The process of fusing image features and depth map features to obtain multimodal fusion features includes fusing image features and depth map features at each resolution scale to obtain multimodal fusion features at N resolution scales.

[0248] For details on the training principles of the liveness detection model, please refer to the method implementation examples. Figure 7 The relevant records will not be elaborated here.

[0249] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0250] And an electronic device, comprising:

[0251] One or more processors; and

[0252] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0253] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0254] in, Figure 10An exemplary architecture of an electronic device is shown, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020 can communicate with each other via a communication bus 1030.

[0255] The processor 1010 can be implemented using a general-purpose CPU (central processing unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0256] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system 1021 for controlling the operation of the electronic device 1000, and the Basic Input Output System (BIOS) 1022 for controlling the low-level operations of the electronic device 1000. Additionally, it can store a web browser 1023, a data storage management system 1024, and a liveness detection device / liveness detection model training device 1025, etc. The aforementioned liveness detection device / liveness detection model training device 1025 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1020 and executed by the processor 1010.

[0257] Input / output interface 1013 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0258] Network interface 1014 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired connections, such as USB (Universal Serial Bus) or Ethernet cables. It can also communicate wirelessly, such as via mobile networks, Wi-Fi, or Bluetooth.

[0259] Bus 1030 includes a pathway for transmitting information between various components of the device (e.g., processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020).

[0260] It should be noted that although the above-described device only shows the processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, memory 1020, bus 1030, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0261] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0262] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0263] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of detecting living matter, characterized by, The method includes: Acquire first and second images of the object to be detected captured by the first and second cameras, respectively. Depth map information is obtained based on the disparity between the first image and the second image; After extracting features from the first image and the depth map information using a liveness detection model to obtain image features and depth map features respectively, the image features and depth map features are fused to obtain multimodal fusion features. The detection result of whether the object to be detected is a live object is obtained using the multimodal fusion features. The liveness detection model is a machine learning model. The liveness detection model includes: a feature extraction layer, a feature fusion layer, and a decoding layer; The method involves extracting features from the first image and the depth map information using a liveness detection model to obtain image features and depth map features, including: extracting features from the first image and the depth map information using the feature extraction layer to obtain the image features and the depth map features. The process of fusing the image features and depth map features to obtain a multimodal fusion feature, and using the multimodal fusion feature to obtain a detection result of whether the object to be detected is a live subject, includes: performing a second merging process on the image features and depth map features through the feature fusion layer to obtain a second fusion feature, wherein the second merging process includes: direct merging process; performing a first attention process and a second attention process on the second fusion feature to obtain an image attention matrix and a depth map attention matrix; fusing the image attention matrix and the depth map attention matrix to obtain a multimodal attention matrix; performing a third merging process on the image features and depth map features using the multimodal attention matrix to obtain the multimodal fusion feature; and using the multimodal fusion feature through the decoding layer to obtain a detection result of whether the object to be detected is a live subject.

2. The method of claim 1, wherein, Obtaining depth map information based on the disparity of the first and second images includes: Epipolar correction is performed on the first image and the second image using the intrinsic parameter matrix, distortion coefficient matrix and rotation matrix calibrated for the first camera and the second camera respectively. Using the first image after epipolar correction as a reference, the parallax between the first and second images at each location is determined; The depth map information is obtained by using the parallax between the first and second images at each location, as well as the optical center distance between the first and second cameras, to determine the depth information at each location.

3. The method of claim 1, wherein, The second fused feature is obtained by performing a second merging process on the image features and the depth map features through the feature fusion layer, including: The first fused feature is obtained by performing global average pooling and concatenation on the image features and the depth map features through the feature fusion layer. The first fused feature is processed by a fully connected layer to obtain a multimodal context matrix; The image features and the depth map features are subjected to a first enhancement process using the multimodal context matrix to obtain first enhanced image features and first enhanced depth map features; The first enhanced image feature and the first enhanced depth map feature are stitched together to obtain the second fused feature.

4. The method according to any one of claims 1 to 3, characterized in that, The image features include image features at N resolution scales, and the depth map features include depth map features at N resolution scales, where N is a positive integer greater than 1; The multimodal fusion features obtained by fusing the image features and depth map features include: Image features and depth map features at each resolution scale are fused separately to obtain multimodal fusion features at N resolution scales.

5. A user authentication method, characterized in that, The method includes: The liveness detection method as described in any one of claims 1 to 4 is used to perform liveness detection on the object to be detected, and a detection result is obtained as to whether the object to be detected is a living organism; and Perform biometric identification on the object to be detected to obtain a result indicating whether the biometric identification has passed. If the object to be detected is a living person and is identified by biometrics, then the user authentication is considered successful.

6. A training method for a liveness detection model, characterized in that, The method includes: Acquire training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live; wherein, the image samples include multiple image pairs, each image pair includes a first image and a second image captured by a first camera and a second camera for the same object, and the depth map samples include depth map information obtained based on the parallax of the first image and the second image in each image pair; The liveness detection model is trained by taking the first image and the depth map sample as inputs and the labels indicating whether the objects in the image samples are live. The liveness detection model extracts features from the first image and the depth map sample to obtain image features and depth map features respectively, then fuses the image features and the depth map features to obtain multimodal fusion features. The detection result of whether the object in the first image is live is obtained using the multimodal fusion features. The liveness detection model is a machine learning model. The liveness detection model includes: a feature extraction layer, a feature fusion layer, and a decoding layer; The liveness detection model extracts features from the first image and the depth map sample to obtain image features and depth map features, including: extracting features from the first image and the depth map sample through the feature extraction layer to obtain the image features and the depth map features; The process of fusing the image features and the depth map features to obtain a multimodal fusion feature, and using the multimodal fusion feature to obtain a detection result of whether an object in the first image is a live object, includes: performing a second merging process on the image features and the depth map features through the feature fusion layer to obtain a second fusion feature, wherein the second merging process includes: direct merging process; performing a first attention process and a second attention process on the second fusion feature to obtain an image attention matrix and a depth map attention matrix; fusing the image attention matrix and the depth map attention matrix to obtain a multimodal attention matrix; performing a third merging process on the image features and the depth map features using the multimodal attention matrix to obtain the multimodal fusion feature; and using the multimodal fusion feature through the decoding layer to obtain a detection result of whether the object is a live object.

7. The method according to claim 6, characterized in that, The image features and the depth map features are further enhanced by the feature fusion layer using the multimodal fusion features to obtain second enhanced image features and second enhanced depth map features. The decoding layer includes an image encoder, a depth encoder, and a fusion encoder; The image encoder uses the second enhanced image features to perform binary classification and obtain a first detection result as to whether the object in the first image is a living body. The depth encoder uses the second enhanced depth map features to perform binary classification and obtain a second detection result as to whether the object in the first image is a living body. The fusion encoder uses the multimodal fusion features to obtain a third detection result as to whether the object in the first image is a live object; The training objective is to minimize the value of the total loss function, which is determined by a first loss function value, a second loss function value, and a third loss function value. The first loss function value is determined by the difference between the first detection result and the label indicating whether an object in the image sample is a live object. The second loss function value is determined by the difference between the second detection result and the label indicating whether an object in the image sample is a live object. The third loss function value is determined by the difference between the third detection result and the label indicating whether an object in the image sample is a live object. After training is completed, the decoding layer retains only the fusion encoder.

8. The method according to any one of claims 6 to 7, characterized in that, The image features include image features at N resolution scales, and the depth map features include depth map features at N resolution scales, where N is a positive integer greater than 1; The process of fusing the image features and depth map features to obtain multimodal fusion features includes fusing image features and depth map features at each resolution scale to obtain multimodal fusion features at N resolution scales.

9. A liveness detection device, characterized in that, The device includes: The image acquisition unit is configured to acquire a first image and a second image captured by a first camera and a second camera for the object to be detected. The depth acquisition unit is configured to acquire depth map information based on the disparity between the first image and the second image; The liveness detection unit is configured to extract image features and depth map features from the first image and the depth map information respectively through a liveness detection model, then fuse the image features and the depth map features to obtain a multimodal fusion feature, and use the multimodal fusion feature to obtain a detection result of whether the object to be detected is a live object, wherein the liveness detection model is a machine learning model; The liveness detection model includes a feature extraction layer, a feature fusion layer, and a decoding layer. The liveness detection unit is configured to perform the following steps to extract features from the first image and the depth map information using the liveness detection model to obtain image features and depth map features: extracting features from the first image and the depth map information using the feature extraction layer to obtain the image features and the depth map features. The liveness detection unit is further configured to perform the following steps to fuse the image features and depth map features to obtain a multimodal fusion feature, and use the multimodal fusion feature to obtain a detection result of whether the object to be detected is a liveness: The image features and depth map features are subjected to a second merging process by the feature fusion layer to obtain a second fusion feature, wherein the second merging process includes: direct merging; performing a first attention process and a second attention process on the second fusion feature to obtain an image attention matrix and a depth map attention matrix; fusing the image attention matrix and the depth map attention matrix to obtain a multimodal attention matrix; using the multimodal attention matrix to perform a third merging process on the image features and depth map features to obtain the multimodal fusion feature; and using the multimodal fusion feature by the decoding layer to obtain a detection result of whether the object to be detected is a liveness.

10. A training device for a liveness detection model, characterized in that, The device includes: The sample acquisition unit is configured to acquire training data, which includes image samples, depth map samples, and labels indicating whether objects in the image samples are live. The image samples include multiple image pairs, each image pair including a first image and a second image acquired by a first camera and a second camera for the same object. The depth map samples include depth map information obtained based on the disparity between the first image and the second image in each image pair. The model training unit is configured to take a first image and a depth map sample from the image samples as input, and take the label indicating whether an object in the image samples is a live object as the target output to train a liveness detection model; wherein, the liveness detection model extracts features from the first image and the depth map sample respectively to obtain image features and depth map features, then fuses the image features and the depth map features to obtain multimodal fusion features, and uses the multimodal fusion features to obtain the detection result of whether the object in the first image is a live object, wherein the liveness detection model is a machine learning model; The liveness detection model includes: a feature extraction layer, a feature fusion layer, and a decoding layer; The liveness detection model is used to extract features from the first image and the depth map sample to obtain image features and depth map features, respectively. This includes: extracting features from the first image and the depth map sample respectively through the feature extraction layer to obtain the image features and the depth map features. The process of fusing the image features and the depth map features to obtain a multimodal fusion feature, and using the multimodal fusion feature to obtain a detection result of whether an object in the first image is a live object, includes: performing a second merging process on the image features and the depth map features through the feature fusion layer to obtain a second fusion feature, wherein the second merging process includes: direct merging process; performing a first attention process and a second attention process on the second fusion feature to obtain an image attention matrix and a depth map attention matrix; fusing the image attention matrix and the depth map attention matrix to obtain a multimodal attention matrix; performing a third merging process on the image features and the depth map features using the multimodal attention matrix to obtain the multimodal fusion feature; and using the multimodal fusion feature through the decoding layer to obtain a detection result of whether the object is a live object.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 8.

Images