Liveness detection method, device, electronic device, storage medium and program product

Abstract

The embodiments of the present application provide a method, device, electronic device, storage medium, and program product for liveness detection, which relate to the fields of image processing and application security technology. The method comprises: obtaining a multimodal facial image; performing feature extraction and fusion on the multimodal facial image to obtain a multimodal fusion feature for characterizing the feature correlation between the modalities; and determining the liveness detection result of the multimodal facial image based on the multimodal fusion feature. The embodiments of the present application are used to solve the problem in the prior art of low detection accuracy due to independent detection of images under each modality while ignoring the correlation between the modalities. The implementation of the present application improves the accuracy of liveness detection.

Description

Technical Field

[0001] The present application relates to the field of image processing and application security technology. Specifically, the present application relates to a liveness detection method, device, electronic device, storage medium and program product. Background Art

[0002] Liveness detection is essential to prevent malicious actors from maliciously exploiting forged biometrics for identity authentication, such as stolen photos, videos recorded online, or fabricated masks. Current liveness detection solutions include silent liveness detection and motion liveness detection. Unlike motion liveness detection, silent liveness detection requires no user interaction and can perform liveness authentication without detection, resulting in faster speed and a better user experience.

[0003] However, current silent liveness detection technology often uses different liveness detection models for multimodal images. Since each modality is detected independently, it is difficult to capture the information differences between the modalities, and the correlation between the modalities cannot be considered, which greatly affects the accuracy of liveness detection. Summary of the Invention

[0004] The embodiments of the present application provide a liveness detection method, apparatus, electronic device, storage medium, and program product, which are used to solve the problem in the prior art of ignoring the correlation between the modalities due to independent detection of images under each modality.

[0005] According to one aspect of an embodiment of the present application, a liveness detection method is provided, comprising:

[0006] Acquire multimodal face images;

[0007] Extracting and fusing features of the multimodal facial image to obtain a multimodal fusion feature for characterizing feature correlation between modalities;

[0008] Based on the multimodal fusion features, a liveness detection result of the multimodal face image is determined.

[0009] In one possible implementation, extracting features from the multimodal facial images and fusing them to obtain multimodal fusion features for characterizing feature correlations between modalities includes:

[0010] The image features of the multimodal facial image in each preset modality are extracted, and for the image features in each modality, the image features are fused with the image features in other modalities to output multimodal fusion features for characterizing the correlation between the image features in the modality and the image features in other modalities.

[0011] In one possible implementation, determining a liveness detection result of the multimodal face image based on the multimodal fusion feature includes:

[0012] The multimodal fusion features under each modality are fused, and a liveness detection result of the multimodal face image is determined according to the fusion result.

[0013] In a possible implementation, the multimodal facial image is subjected to feature extraction and subsequent fusion by a first feature extraction network in the feature extraction network;

[0014] The first feature extraction network includes a global feature extraction network and / or a local feature extraction network, and the multimodal fusion feature includes at least one of the output of the global feature extraction network, the output of the local feature extraction network, and a fusion feature organized by the output of the global feature extraction network and the output of the local feature extraction network.

[0015] In one possible implementation, the local feature extraction network includes a cascaded first linear layer, a feature fusion layer based on a self-attention mechanism, and a second linear layer;

[0016] The following steps are performed by the local feature extraction network to obtain the output of the local feature extraction network:

[0017] Performing a linear transformation on the multimodal face image through the first linear layer to obtain a local feature sequence for representing local features of the i-th target region in the image in each mode, where i ≥ 1;

[0018] Performing feature fusion on the local feature sequence through the feature fusion layer to obtain fusion features for the i-th target area under different modalities;

[0019] The fusion feature is linearly transformed by the second linear layer to obtain a multimodal local fusion feature for characterizing the correlation between each modality and the local features of other modalities, and the multimodal local fusion feature is used as the output of the local feature extraction network.

[0020] In one possible implementation, the global feature extraction network includes a first residual network and a second residual network in cascade order, the first residual network includes an attention mechanism, and the second residual network includes a multi-layer perceptron;

[0021] The following steps are performed by the global feature extraction network to obtain the output of the global feature extraction network:

[0022] Performing feature extraction on the multimodal facial image using the first residual network, performing residual analysis on the output result and the input multimodal facial image, and outputting a first global feature under each modality;

[0023] The first global feature under each modality is extracted through the second residual network, and residual analysis is performed on the output result and the input first global feature to obtain the second global feature under each modality, and use it as the output of the global feature extraction network.

[0024] In a possible implementation, the multimodal facial image is subjected to feature extraction and subsequent fusion by a second feature extraction network in the feature extraction network;

[0025] The step of extracting and fusing the multimodal facial images to obtain multimodal fusion features for characterizing feature correlations between modalities further includes:

[0026] The multimodal face image is input into the second feature extraction network to extract the image features under each modality, and the image features under each modality are fused to output a multimodal fusion feature for characterizing the correlation between the image features of each modality.

[0027] In one possible implementation, acquiring a multimodal facial image includes:

[0028] Acquire a multimodal image sent by a terminal device at the same time, where the multimodal image includes at least two of an RGB image, an infrared image, and a depth image;

[0029] Performing face recognition on the multimodal image to obtain a corresponding multimodal face image, wherein the multimodal face image includes at least two of RGB face images, infrared face images, and depth face images.

[0030] According to another aspect of an embodiment of the present application, a living body detection device is provided, comprising:

[0031] An image acquisition module, used to acquire multimodal face images;

[0032] A multimodal feature processing module, configured to extract and fuse features from the multimodal facial image to obtain multimodal fusion features for characterizing feature correlations between modalities;

[0033] A liveness detection module is used to determine a liveness detection result of the multimodal face image based on the multimodal fusion feature.

[0034] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the liveness detection method described in the above embodiment.

[0035] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the liveness detection method described in the above embodiment are implemented.

[0036] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the living body detection method described in the above embodiment when executed by a processor.

[0037] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0038] The liveness detection method provided in the present application obtains multimodal facial images, then extracts and fuses the features of the multimodal facial images to obtain multimodal fusion features for characterizing the feature correlation between the modalities. Based on the multimodal fusion features, the liveness detection result of the multimodal facial image is determined. This method can solve the problem in the prior art of ignoring the correlation between the modalities due to independent detection of images under each modality. It can take into account the differences and correlations between the modal data during the detection process, so as to fully utilize the image feature data in multiple dimensions for liveness detection and improve the accuracy of liveness detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0040] Figure 1 A schematic diagram of the computer system architecture provided in an embodiment of the present application;

[0041] Figure 2 A schematic structural diagram of a liveness detection system provided by an exemplary embodiment of the present application;

[0042] Figure 3 A schematic diagram of a flow chart of a liveness detection method provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of an MML-Vit network provided as an exemplary embodiment of the present application;

[0044] Figure 5 A schematic structural diagram of an MML-Vit Block model provided as another exemplary embodiment of the present application;

[0045] Figure 6 A schematic structural diagram of a living body detection device provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0048] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0049] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0050] Figure 1 Schematic diagram of a computer system architecture provided in an embodiment of the present application. The computer system 100 includes multiple terminal devices 101 and a server 102, wherein each terminal device 101 is connected to the server 102 via a communication network, and the terminal devices 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0051] The terminal device 101 can be any terminal device that has an application installed or can run a program, such as a smart camera device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart wearable device, an in-vehicle device, etc., and the embodiments of this application are not limited to this. Regarding the hardware structure, the terminal device 101 includes a camera, a display screen, a memory, a processor, and an input device, but is not limited thereto. Exemplarily, the application is a terminal-side application of a multimedia platform.

[0052] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. In addition, in this application, the server 102 can also be other processing devices with data processing functions. The server carries a multimedia platform, and the above-mentioned server provides background services for applications developed and running in multiple terminal devices.

[0053] Among them, the terminal device 101 is used to obtain multimodal images at the same time. In this application, the terminal device can be a camera device with the function of collecting images under multiple modalities, or it can be multiple camera devices with the function of collecting images under a single modality. This application does not limit this. The server 102 is an electronic device that provides computing power. It is connected to the terminal device 101 and is used to receive the multimodal images sent by the terminal device 101, and perform face recognition on the multimodal images to obtain a multimodal face image, thereby performing feature extraction and feature fusion on the multimodal face image to obtain a multimodal fusion feature for characterizing the feature correlation between the modalities, so that the liveness detection result of the multimodal face image is determined based on the multimodal fusion feature. In this way, the feature correlation between the images under each modality is taken into account, thereby improving the accuracy of liveness detection. Optionally, the above-mentioned operation steps performed by the server 102 can also be performed by the terminal device 101.

[0054] In a specific embodiment, Figure 2A structural diagram of a living body detection system provided for an exemplary embodiment of the present application, multimodal images include but are not limited to RGB images, infrared images and depth images, so for collecting multimodal images, the terminal device 101 includes at least two of a visible light camera device (such as an RGBD camera), a near-infrared camera device (such as an infrared camera) and a depth camera device (such as a structured light depth camera, a binocular stereo vision camera, a TOF camera). Optionally, in addition to the above-mentioned example terminal devices, the terminal device 101 of the present application also includes other camera devices capable of collecting RGB images, infrared images and depth images, as well as camera devices for collecting imaging in other modes besides the above-mentioned RGB mode, infrared mode and depth mode, and the present application does not limit this. More specifically, as Figure 2 As shown, a visible light camera 1011 is connected to the server 102 to capture RGB images and send them to the server; a near-infrared camera 1012 is connected to the server 102 to capture infrared images and send them to the server; and a depth camera 1013 is connected to the server 102 to capture depth images and send them to the server. In this way, at least two of the RGB, infrared, and depth images are used as input, and liveness detection is performed using the information differences and correlations between the images in each modality. Specifically, the server 102 performs face detection and expansion on the received multimodal image to obtain an RGB face image, an infrared face image, and a depth face image, thereby performing feature extraction and feature fusion on the RGB face image, the infrared face image, and the depth face image, which is conducive to obtaining the differences and correlations between the face images in the RGB modality, the infrared modality, and the depth modality, so as to achieve liveness detection of the target detection object based on the multimodal fusion features generated by multimodal face image processing.

[0055] Figure 3 A flow chart of a liveness detection method provided in the embodiment of the present application. A liveness detection method is provided in the embodiment of the present application, which is applied to Figure 1 、 Figure 2 Taking the server 102 as an example, the method includes steps S301 to S303.

[0056] S301: Acquire a multimodal face image.

[0057] In some embodiments, acquiring a multimodal facial image includes:

[0058] Acquire a multimodal image sent by a terminal device at the same time, where the multimodal image includes at least two of an RGB image, an infrared image, and a depth image;

[0059] Performing face recognition on the multimodal image to obtain a corresponding multimodal face image, wherein the multimodal face image includes at least two of RGB face images, infrared face images, and depth face images.

[0060] In various scenarios such as access control systems, payment applications, and social applications, user identity verification is often required. Common attack methods include: 2D-based fake face data, such as using printed paper, coated paper, photo paper, computer screens, tablet screens, mobile phone screens, etc. to perform portrait attacks; 3D-based fake face data, such as using head molds, silicone masks, and face masks with holes cut out of paper photos to perform portrait attacks.

[0061] Among them, RGB image refers to a color image obtained by shooting the target detection object with a camera, and each pixel value of the RGB image is represented by the color components of the three channels R (red), G (green), and B (blue). Infrared image refers to an infrared image formed by different infrared rays obtained by measuring the infrared difference between the target detection object itself and the background with a detector. Depth image refers to an image with the distance value from the camera to each point in the scene where the target detection object is located as the pixel value, and the depth image can reflect the geometric shape of the target surface. Therefore, the present application performs liveness detection based on the facial images of the target detection object in multiple modalities, can make full use of the facial feature information in multiple modalities, perform liveness detection from multiple dimensions, intercept the above-mentioned attack images, and improve the detection accuracy.

[0062] In this embodiment, the server performs facial recognition and expansion on the RGB, infrared, and depth images sent by the terminal device, respectively, to obtain RGB, infrared, and depth facial images of preset pixel sizes. Alternatively, facial recognition refers to the use of computers to process, analyze, and understand images to identify various target objects. It is a practical application of artificial intelligence algorithms such as deep learning algorithms and computer vision algorithms.

[0063] S302: Extract features from the multimodal facial image and fuse them to obtain multimodal fusion features for characterizing feature correlation between modalities.

[0064] It should be noted that, taking images under RGB modality, infrared modality and depth modality as examples, feature extraction is performed on RGB face images, infrared face images and depth face images, thereby obtaining image features under each modality, that is, image features corresponding to RGB face images, infrared face images and depth face images, respectively. Further, the image features under RGB modality, infrared modality and depth modality are feature fused to obtain multimodal fusion features that can be used to characterize the correlation and difference between image features under RGB modality, infrared modality and depth modality, thereby improving the accuracy of subsequent liveness detection. In this way, the present application performs feature extraction and feature fusion on multimodal face images, avoiding the lack of information correlation between modalities due to independent detection of images under each modality, and the difficulty in capturing information differences between modalities, thereby improving the expressiveness and accuracy of the model, and improving the correlation between data of each modality.

[0065] S303: Determine a liveness detection result of the multimodal face image based on the multimodal fusion feature.

[0066] The liveness detection method provided in the embodiments of the present application obtains a multimodal facial image, then extracts and fuses the features of the multimodal facial image to obtain a multimodal fusion feature for characterizing the feature correlation between the modalities. Based on the multimodal fusion feature, the liveness detection result of the multimodal facial image is determined. This method can solve the problem in the prior art of ignoring the correlation between the modalities due to independent detection of images under each modality. It can take into account the differences and correlations between the modal data during the detection process, so as to fully utilize the image feature data in multiple dimensions for liveness detection, thereby improving the accuracy of liveness detection.

[0067] In some embodiments, step S302 includes:

[0068] The image features of the multimodal facial image in each preset modality are extracted, and for the image features in each modality, the image features are fused with the image features in other modalities to output multimodal fusion features for characterizing the correlation between the image features in the modality and the image features in other modalities.

[0069] In this embodiment, for facial images under each modality (i.e., at least two of RGB facial images, infrared facial images, and depth facial images), image features under each modality are extracted. Furthermore, feature fusion is performed on the image features under each modality to obtain a multimodal fusion feature that reflects the correlation between each modality and the image features under other modalities. Each modality has a corresponding multimodal fusion feature, including multimodal fusion features under RGB modality, infrared modality, and depth modality. It should be noted that the multimodal fusion feature is used to characterize the fusion feature of each modality under the mutual influence and mutual correlation with the image features of other modalities. For example, for the RGB modality, the image features of the RGB modality are fused with the feature images under the infrared modality and the depth modality, taking into account the feature correlation between the RGB modality and the infrared modality and the depth modality, to obtain the multimodal fusion feature under the RGB modality. Therefore, the multimodal fusion feature used for liveness detection in this embodiment takes into account the mutual correlation between modalities, thereby improving the accuracy of image features and the precision of liveness detection.

[0070] Based on the above embodiment, in some embodiments, step S303 includes:

[0071] The multimodal fusion features under each modality are fused, and a liveness detection result of the multimodal face image is determined according to the fusion result.

[0072] In this embodiment, multiple multimodal fusion features for different modalities are further fused. Optionally, in the feature fusion process of this embodiment, multiple multimodal fusion features are fused into at least one fusion feature result to characterize image feature information that has a decisive and critical role in liveness detection under all modalities, so as to perform liveness detection on multimodal face images based on the fusion feature result. Optionally, in the feature fusion process, multiple multimodal fusion features for different modalities are further fused with each other to obtain a fusion feature result under the corresponding modality, making the feature more accurate and being able to use the fusion feature result under any modality to quickly and accurately identify the liveness detection result. Therefore, this embodiment further fuses the multimodal fusion features under different modalities, thereby improving the information richness and effectiveness of the image features used for liveness detection, and helping to improve the accuracy and efficiency of liveness detection.

[0073] In some embodiments, the multimodal facial image is subjected to feature extraction and subsequent fusion by a first feature extraction network in the feature extraction network;

[0074] The first feature extraction network includes a global feature extraction network and / or a local feature extraction network, and the multimodal fusion feature includes at least one of the output of the global feature extraction network, the output of the local feature extraction network, and a fusion feature organized by the output of the global feature extraction network and the output of the local feature extraction network.

[0075] In this embodiment, by inputting multimodal images into the same feature extraction network (i.e., the first feature extraction network) for feature extraction and feature fusion, the situation in which the correlation between the modalities is ignored due to the use of independent models for image detection under each modality is avoided. This allows the differences and correlations between the modal data to be taken into account during the detection process, thereby improving the accuracy of liveness detection.

[0076] For example, Figure 4 The schematic diagram of the structure of the MML-Vit network provided for an exemplary embodiment of the present application is a diagram of the structure of the MML-Vit network. The images at the same moment are acquired by a visible light camera, a near infrared camera, and a depth camera respectively. The acquired images are subjected to face detection and expansion using the retinaface face detection algorithm to obtain a multimodal face image of size 224*224. The RGB face image, infrared face image, and depth face image are evenly divided into 32*32 sizes, and a total of 3 groups of image feature blocks are obtained. The number of feature blocks in each group is 7*7, which are used as the input of the feature extraction network. Figure 4 As shown, in this exemplary embodiment, the feature extraction network adopts the MML-Vit network, and the above three groups of image feature blocks are simultaneously input into the MML-Vit (Multi-Model Local Vision Transformer) network to output the liveness detection result. The MML-Vit network includes an RGB branch, an IR branch, and a Depth branch, which input each group of image feature blocks into their respective corresponding branches. Each branch of the MML-Vit network includes a linear layer (such as Figure 4 Linear project layer in MML-VitBlock model, multi-layer perceptron (such as Figure 4 MLP header layer in) and Sigmoid layer (such as Figure 4Cat&Sigmoid layer in the model), linearly transform the image feature blocks of each branch in the linear layer, and input the linearly changed image feature blocks into the MML-Vit Block model to perform feature extraction, fusion and other processing on each group of image feature blocks to output multimodal fusion features for characterizing the feature correlation between modalities, and further extract and select features from the output of the MML-Vit Block model through a multilayer perceptron, so as to perform feature fusion on the output of the multilayer perceptron of each branch, that is, fuse the image feature data under the RGB modality, infrared modality and depth modality, and input the feature fusion result into the Sigmoid layer to realize nonlinear transformation of the feature space to improve the fitting of the model, so as to perform classification based on the output of the Sigmoid layer to obtain the liveness detection result.

[0077] Optionally, the MML-Vit Block model is organized by L (L≥2) layers of feature extraction subnetworks, so that multi-layer feature extraction is performed on each group of image feature blocks to obtain image features under each modality.

[0078] Based on the above embodiment, in one embodiment, the local feature extraction network includes a cascaded first linear layer, a feature fusion layer based on a self-attention mechanism, and a second linear layer;

[0079] The following steps are performed by the local feature extraction network to obtain the output of the local feature extraction network:

[0080] Performing a linear transformation on the multimodal face image through the first linear layer to obtain a local feature sequence for representing local features of the i-th target region in the image in each mode, where i ≥ 1;

[0081] Performing feature fusion on the local feature sequence through the feature fusion layer to obtain fusion features for the i-th target area under different modalities;

[0082] The fusion feature is linearly transformed by the second linear layer to obtain a multimodal local fusion feature for characterizing the correlation between each modality and the local features of other modalities, and the multimodal local fusion feature is used as the output of the local feature extraction network.

[0083] For the target area in the image, in an optional embodiment, this embodiment is provided with a plurality of detection points (i.e., target areas), such as the eye area, the mouth area, the eyebrow area, and the contour area, wherein the number of detection points is not limited. Thus, according to the position information of the detection points, the detection points in the image are identified using key point tracking technology or target recognition technology. In another optional embodiment, the spots and corner points of the image under each modality are identified, so that the spots and corner points are used as detection points for characterizing local feature points. Among them, spots generally refer to areas that have color and grayscale differences from the surrounding area, and corner points are the corners of an object on one side of the image or the intersection between lines. For this purpose, LoG and SIFT algorithms are used to detect spots, and Harris algorithm and FAST algorithm are used to detect corner points.

[0084] In this embodiment, a local feature extraction network is used to extract local features for different target regions in an image under different modalities, and from these local features, a multimodal local fusion feature is obtained for the same target region under different modalities. The multimodal local fusion feature characterizes the correlation between the image features of each modality and the other modalities for the same target region, as well as the correlation between the fusion features of i target regions under different modalities. Therefore, during the liveness detection process, the multimodal local fusion feature used to characterize the local features is used as the multimodal fusion feature for liveness detection. This improves the accuracy of liveness detection by taking into account the mutual correlation between the local regions of the image under different modalities and the correlation between the multimodal features of the local regions.

[0085] Specifically, a linear transformation is performed in the first linear layer to obtain local features for the same target area under each modality, and the local features representing the same target area under different modalities are organized into a local feature sequence. For example, the RGB face image, infrared face image and depth face image are used as the input of the first linear layer of the local feature extraction network, which are represented as and After the linear transformation of the first linear layer, the local feature sequences representing each target area are obtained, which are Furthermore, in order to obtain the correlation of local features of the same target area under different modalities, the above local feature sequences are fused separately in the feature fusion layer based on the self-attention mechanism to obtain the fusion features of the same target area under RGB modality, infrared modality and depth modality. Then, linear transformation is performed in the second linear layer to obtain multimodal local fusion features that represent the correlation between local features of each modality and other modalities, which are denoted as and And use it as the output of the local feature extraction network.

[0086] Based on the above embodiment, in one embodiment, the global feature extraction network includes a first residual network and a second residual network in cascade order;

[0087] The following steps are performed by the global feature extraction network to obtain the output of the global feature extraction network:

[0088] Performing feature extraction on the multimodal facial image using the first residual network, performing residual analysis on the output result and the input multimodal facial image, and outputting a first global feature under each modality;

[0089] The first global feature under each modality is extracted through the second residual network, and residual analysis is performed on the output result and the input first global feature to obtain the second global feature under each modality, and use it as the output of the global feature extraction network.

[0090] In this embodiment, global features include but are not limited to color features, texture features, contour features and spatial features. Specifically, the first residual network includes an attention mechanism, and the second residual network includes a multi-layer perceptron. This application uses a transformer encoder to extract features from the input RGB face image, infrared face image and depth face image respectively to obtain global features under RGB mode, infrared mode and depth mode, which are respectively recorded as and where l∈{1,2,…,L}.

[0091] For example, Figure 4 The MML-Vit Block model shown includes a global feature extraction network and a local feature extraction network. Figure 5 A schematic diagram of the structure of the MML-Vit Block model provided for another exemplary embodiment of the present application is shown. The structure on the left side of the figure is a global feature extraction network, and the structure on the right side of the figure is a local feature extraction network. The output of the global feature extraction network (global features under RGB / infrared / depth modalities) is fused with the output of the local feature extraction network, and the fused features obtained under each modality are used as the output result of the MML-Vit Block model. Multi-layer feature extraction, fusion, and other processing are performed on multimodal facial images based on the global feature extraction network and the local feature extraction network, respectively. The global feature extraction network and the local feature extraction network have the same number of layers.

[0092] For example, Figure 5As shown in Figure 1, the RGB face image, infrared face image and depth face image are input into the global feature extraction network respectively, and then input into the first residual network. They are normalized and standardized in the Norm layer of the first residual network to accelerate convergence. Then, the output result of the Norm layer is input into the attention mechanism ( Figure 5 The multi-head Attention in the algorithm is used to calculate the correlation between features, and then the residual analysis is performed on the output of the attention mechanism and the corresponding input multimodal face image (i.e., the input of the first residual network), so that the first residual network outputs the first global feature. Furthermore, the first global feature under the RGB mode, infrared mode, and depth mode is input to the second residual network, and normalized and standardized in the Norm layer of the second residual network to accelerate convergence. Then, the output of the Norm layer is input to the multilayer perceptron ( Figure 5 The MLP in the multi-layer perceptron is used to perform residual analysis on the output result of the multi-layer perceptron and the corresponding input image global features (i.e., the input of the second residual network), so that the second residual network outputs the second global features of the image in RGB mode, infrared mode, and depth mode, and uses the second global features as the output of the global feature extraction network.

[0093] The RGB face image, infrared face image and depth face image (represented as and ) are respectively input into the local feature extraction network to perform linear transformation in the first linear layer to obtain local features for the same target area under different modalities, and organize the local features of the images under different modalities representing the same target area into a local feature sequence Furthermore, the feature fusion layer based on the self-attention mechanism is used to fuse the above local feature sequences respectively to obtain the fusion features for the same target area in RGB mode, infrared mode and depth mode. Then, a linear transformation is performed in the second linear layer to obtain the multimodal local fusion features that represent the correlation between each modality and the local features of other modalities, which are denoted as and It is used as the output of the local feature extraction network.

[0094] Further, if Figure 5 As shown in , the output of the global feature extraction network and the output of the local feature extraction network are fused according to different modes, that is, and As the multimodal fusion feature under each modality. In this way, in this embodiment, the image features include a multimodal local fusion feature for characterizing local features and a second global feature for characterizing global features, which can better characterize the global and local features of images under different modalities, and take into account the global and local information differences and correlations between different modalities in liveness detection, enriching the effective information content of image features.

[0095] In some embodiments, the first feature extraction network includes a global feature extraction network, which can be used to extract global features of images of different modalities, and for each global feature of the image under the modality, it is respectively fused with the global features of the image under other modalities to obtain a multimodal global fusion feature for characterizing the correlation between the global features of the image of the modality and other modalities. Therefore, the multimodal global fusion feature output by the global feature extraction network is used as a multimodal fusion feature for liveness detection, taking into account the mutual correlation between the global features of the image in different modalities, thereby improving the accuracy of liveness detection. Therefore, the structure of the global feature extraction network is not limited in this application.

[0096] In some embodiments, the first feature extraction network includes a global feature extraction network and a local feature extraction network. Among them, the local feature extraction network is used to obtain local features for different target areas under RGB modality, infrared modality and depth modality. The structure of the local feature extraction network is not limited in this application. The global feature extraction network is used to obtain multimodal global fusion features for characterizing the correlation between the global image features of each modality and other modalities. Therefore, the multimodal global fusion features output by the global feature extraction network are fused with the local features output by the local feature extraction network, and the fused fusion features are used as multimodal fusion features for liveness detection. Taking into account the mutual correlation between the global features of the image in different modalities and the local features of the image in different modalities, the accuracy of liveness detection is improved.

[0097] In some embodiments, the first feature extraction network includes a global feature extraction network and a local feature extraction network. The local feature extraction network is used to extract local features for different target areas in the image under different modalities, and obtain multimodal local fusion features that characterize the correlation between the image features of each modality and other modalities for the same target area. In addition, the global feature extraction network is used to obtain multimodal global fusion features for characterizing the correlation between the global image features of each modality and other modalities. Therefore, the multimodal global fusion features output by the global feature extraction network are fused with the multimodal local fusion features output by the local feature extraction network, and the fused fusion features are used as multimodal fusion features for liveness detection. Taking into account the mutual correlation between the global features of the image in different modalities and the mutual correlation between the local features of the image in different modalities, the accuracy of liveness detection is improved.

[0098] In some embodiments, the multimodal facial image is subjected to feature extraction and subsequent fusion by a second feature extraction network in the feature extraction network;

[0099] The step of extracting and fusing the multimodal facial images to obtain multimodal fusion features for characterizing feature correlations between modalities further includes:

[0100] The multimodal face image is input into the second feature extraction network to extract the image features under each modality, and the image features under each modality are fused to output a multimodal fusion feature for characterizing the correlation between the image features of each modality.

[0101] In this embodiment, facial images under each modality (i.e., at least two of RGB facial images, infrared facial images, and depth facial images) are input into a second feature extraction network to extract image features under each modality. Image features under multiple different modalities are fused into at least one multimodal fusion feature, which is used to characterize image feature information that plays a decisive and critical role in liveness detection under all modalities, taking into account the information differences and feature correlations between different modalities, so as to perform liveness detection on multimodal facial images based on the fusion feature results. For example, Figure 4As shown, the second feature extraction network is used as the MML-Vit Block model for illustration. RGB face images, infrared face images, and depth face images are input into the MML-Vit Block model for feature extraction. Image features in RGB, infrared, and depth modes are output. Each image feature is fused after passing through a multi-layer perceptron to obtain a multimodal fusion feature for characterizing the feature association between different modalities. The multimodal fusion feature is then input into the Sigmoid layer for classification to obtain a liveness detection result. In this way, this embodiment inputs multimodal images into the same feature extraction network for feature extraction and feature fusion. Taking into account the differences and correlations between the modal data, the image feature data in multiple dimensions is fully utilized for liveness detection, thereby improving the accuracy of liveness detection.

[0102] Based on the above embodiment, optionally, the second feature extraction network includes a global feature network and / or a local feature network. The local feature network is used to obtain local features for different target areas under different modalities, and the global feature network is used to obtain global features under different modalities, so that the output of the global feature network, the output of the local feature network, and one of the features organized by the output of the global feature network and the output of the local feature network are used as image features under different modalities, thereby fused into a multimodal fusion feature that characterizes the feature correlation between different modalities, thereby taking into account the mutual correlation between the global features and local features of the image in different modalities.

[0103] Figure 6 This is a schematic diagram of the structure of a liveness detection device provided in an embodiment of the present application. The liveness detection device 400 includes:

[0104] Image acquisition module 401, used to acquire multimodal face images;

[0105] A multimodal feature processing module 402 is configured to extract and fuse features of the multimodal facial image to obtain multimodal fusion features for characterizing feature correlations between modalities.

[0106] The liveness detection module 403 is configured to determine a liveness detection result of the multimodal face image based on the multimodal fusion feature.

[0107] In some embodiments, the multimodal feature processing module 402 includes:

[0108] The first feature processing unit is used to extract image features of the multimodal facial image in each preset modality, and for the image features in each modality, perform feature fusion on the image features with the image features in other modalities to output multimodal fusion features for characterizing the correlation between the image features in the modality and the image features in other modalities.

[0109] In some embodiments, the liveness detection module 403 includes:

[0110] The multimodal fusion unit is used to fuse the multimodal fusion features under each modality and determine the liveness detection result of the multimodal face image according to the fusion result.

[0111] In some embodiments, the apparatus 400 further includes:

[0112] A first feature extraction network processing unit is used to perform feature extraction and subsequent fusion on the multimodal facial image through a first feature extraction network in the feature extraction network; the first feature extraction network includes a global feature extraction network and / or a local feature extraction network, and the multimodal fusion feature includes at least one of the output of the global feature extraction network, the output of the local feature extraction network, and a fusion feature organized by the output of the global feature extraction network and the output of the local feature extraction network.

[0113] In some embodiments, the local feature extraction network includes a cascaded sorted first linear layer, a feature fusion layer based on a self-attention mechanism, and a second linear layer;

[0114] The first feature extraction network processing unit includes:

[0115] a local feature extraction unit, configured to perform a linear transformation on the multimodal face image through the first linear layer to obtain a local feature sequence for representing local features of an i-th target region in the image under each mode, where i ≥ 1;

[0116] A local feature fusion unit, configured to perform feature fusion on the local feature sequence through the feature fusion layer to obtain fusion features for the i-th target region under different modalities;

[0117] The multimodal local feature fusion unit is used to perform a linear transformation on the fused features through the second linear layer to obtain a multimodal local fusion feature for characterizing the correlation between the local features of each modality and other modalities, and use it as the output of the local feature extraction network.

[0118] In some embodiments, the global feature extraction network includes a cascaded sorted first residual network and a second residual network;

[0119] The first feature extraction network processing unit further includes:

[0120] a first global feature extraction unit, configured to perform feature extraction on the multimodal facial image using the first residual network, perform residual analysis on the output result and the input multimodal facial image, and output a first global feature under each modality;

[0121] The second global feature extraction unit is used to extract the first global feature under each modality through the second residual network, and perform residual analysis on the output result and the input first global feature to obtain the second global feature under each modality, and use it as the output of the global feature extraction network.

[0122] In some embodiments, the multimodal facial image is subjected to feature extraction and subsequent fusion by a second feature extraction network in the feature extraction network, and the multimodal feature processing module 402 further includes:

[0123] The second feature processing unit is used to input the multimodal face image into the second feature extraction network, extract the image features under each modality, and perform feature fusion on the image features under each modality to output a multimodal fusion feature for characterizing the correlation between the image features of each modality.

[0124] In some embodiments, the image acquisition module 401 includes:

[0125] a multimodal image acquisition unit, configured to acquire multimodal images sent by a terminal device at the same time, wherein the multimodal images include at least two of RGB images, infrared images, and depth images;

[0126] A face recognition unit is used to perform face recognition on the multimodal image to obtain a corresponding multimodal face image, where the multimodal face image includes at least two of RGB face images, infrared face images, and depth face images.

[0127] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0128] An embodiment of the present application provides an electronic device including a memory, a processor, and a computer program stored on the memory. The processor executes the above-mentioned computer program to implement the steps of the liveness detection method. Compared with the related art, it can achieve: solving the problem of ignoring the correlation between the modalities due to independent detection of images under each modality in the prior art, so that the differences and correlations between the modal data can be taken into account during the detection process, so as to fully utilize the image feature data in multiple dimensions for liveness detection, thereby improving the accuracy of liveness detection.

[0129] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7The electronic device 500 shown includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may further include a transceiver 504, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present application.

[0130] The processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0131] The bus 502 may include a path for transmitting information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0132] The memory 503 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0133] The memory 503 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the steps shown in the above method embodiments.

[0134] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0135] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0136] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0137] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0138] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A method for detecting a living body, characterized in that: include: Acquire multimodal face images; Extracting and fusing features of the multimodal facial image to obtain a multimodal fusion feature for characterizing feature correlation between modalities; Determining a liveness detection result of the multimodal face image based on the multimodal fusion features; The step of extracting features from the multimodal facial images and fusing them includes: Performing feature extraction and subsequent fusion on the multimodal face image through a first feature extraction network in the feature extraction network; The first feature extraction network includes a global feature extraction network and / or a local feature extraction network, and the multimodal fusion feature includes at least one of an output of the global feature extraction network, an output of the local feature extraction network, and a fusion feature formed by organizing the output of the global feature extraction network and the output of the local feature extraction network; The global feature extraction network includes a first residual network and a second residual network in cascade order; The following steps are performed by the global feature extraction network to obtain the output of the global feature extraction network: Performing feature extraction on the multimodal facial image using the first residual network, performing residual analysis on the output result and the input multimodal facial image, and outputting a first global feature under each modality; The first global feature under each modality is extracted through the second residual network, and residual analysis is performed on the output result and the input first global feature to obtain the second global feature under each modality, and use it as the output of the global feature extraction network.

2. The method according to claim 1, characterized in that The step of extracting and fusing the multimodal facial images to obtain multimodal fusion features for characterizing feature correlations between modalities includes: The image features of the multimodal facial image in each preset modality are extracted, and for the image features in each modality, the image features are fused with the image features in other modalities to output multimodal fusion features for characterizing the correlation between the image features in the modality and the image features in other modalities.

3. The method according to claim 2, characterized in that Determining a liveness detection result of the multimodal face image based on the multimodal fusion feature includes: The multimodal fusion features under each modality are fused, and a liveness detection result of the multimodal face image is determined according to the fusion result.

4. The method according to claim 1, wherein The local feature extraction network includes a first linear layer of cascade sorting, a feature fusion layer based on a self-attention mechanism, and a second linear layer; The following steps are performed by the local feature extraction network to obtain the output of the local feature extraction network: The multimodal face image is linearly transformed by the first linear layer to obtain a local feature sequence for characterizing the local features of the i-th target area in the image in each mode, wherein: ; Performing feature fusion on the local feature sequence through the feature fusion layer to obtain fusion features for the i-th target area under different modalities; The fusion feature is linearly transformed by the second linear layer to obtain a multimodal local fusion feature for characterizing the correlation between each modality and the local features of other modalities, and the multimodal local fusion feature is used as the output of the local feature extraction network.

5. The method according to claim 1, wherein Performing feature extraction and subsequent fusion on the multimodal face image through a second feature extraction network in the feature extraction network; The step of extracting and fusing the multimodal facial images to obtain multimodal fusion features for characterizing feature correlations between modalities further includes: The multimodal face image is input into the second feature extraction network to extract the image features under each modality, and the image features under each modality are fused to output a multimodal fusion feature for characterizing the correlation between the image features of each modality.

6. The method according to claim 1, characterized in that The acquiring of a multimodal facial image comprises: Acquire a multimodal image sent by a terminal device at the same time, where the multimodal image includes at least two of an RGB image, an infrared image, and a depth image; Performing face recognition on the multimodal image to obtain a corresponding multimodal face image, wherein the multimodal face image includes at least two of RGB face images, infrared face images, and depth face images.

7. A living body detection device, characterized in that: include: An image acquisition module, used to acquire multimodal face images; A multimodal feature processing module, configured to extract and fuse features from the multimodal facial image to obtain multimodal fusion features for characterizing feature correlations between modalities; a liveness detection module, configured to determine a liveness detection result of the multimodal face image based on the multimodal fusion feature; The step of extracting features from the multimodal facial images and fusing them includes: Performing feature extraction and subsequent fusion on the multimodal face image through a first feature extraction network in the feature extraction network; The first feature extraction network includes a global feature extraction network and / or a local feature extraction network, and the multimodal fusion feature includes at least one of an output of the global feature extraction network, an output of the local feature extraction network, and a fusion feature formed by organizing the output of the global feature extraction network and the output of the local feature extraction network; The global feature extraction network includes a first residual network and a second residual network in cascade order; The following steps are performed by the global feature extraction network to obtain the output of the global feature extraction network: Performing feature extraction on the multimodal facial image using the first residual network, performing residual analysis on the output result and the input multimodal facial image, and outputting a first global feature under each modality; The first global feature under each modality is extracted through the second residual network, and residual analysis is performed on the output result and the input first global feature to obtain the second global feature under each modality, and use it as the output of the global feature extraction network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the liveness detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the liveness detection method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the liveness detection method according to any one of claims 1 to 6 are implemented.

Images