Live detection methods, systems, electronic devices, and computer-readable storage media

Abstract

Embodiments of the present application relate to the technical field of machine vision, and disclose a living body detection method and system, an electronic device and a computer readable storage medium. The method comprises: acquiring an infrared image and a speckle image of a target object, and performing frequency domain conversion on the speckle image to obtain a frequency domain image; inputting the infrared image, the speckle image and the frequency domain image into a pre-trained living body detection model to acquire a fusion detection score and a frequency domain detection score of the target object output by the living body detection model; determining a living body score of the target object according to the fusion detection score and the frequency domain detection score, and determining that the target object is a living body in the case that the living body score is greater than a first preset threshold. The living body detection method provided by the embodiments of the present application can realize high-precision living body detection without using high-cost sensors, the data of different modalities are complementary, the parameter quantity and the calculation quantity are effectively reduced, and the real-time requirement of embedded terminals for living body detection is well met.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a liveness detection method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] Facial recognition technology has been widely used in people's daily lives and work. However, facial recognition systems are vulnerable to attacks from various types of spoofs, such as printed images, makeup, masks, and 3D models. To ensure the security of facial recognition systems, facial liveness detection technology has received widespread attention from academia and industry. With the rapid development of deep learning technology, facial liveness detection technology has also been improved.

[0003] Currently, commonly used liveness detection algorithms in the industry include unimodal and multimodal algorithms. Unimodal liveness detection algorithms struggle to meet the accuracy requirements of high-precision scenarios. While multimodal liveness detection algorithms offer higher accuracy, their feature extraction branches operate independently, resulting in a large number of parameters and computational overhead, making them unsuitable for embedded systems and real-time scenarios. Furthermore, these liveness detection algorithms rely on detailed facial information, requiring the use of costly and high-precision sensors. Summary of the Invention

[0004] The purpose of this application is to provide a liveness detection method, system, electronic device, and computer-readable storage medium that can achieve high-precision liveness detection without the use of high-cost sensors. The data from different modalities are complementary, effectively reducing the number of parameters and computational load, and well meeting the real-time requirements of embedded terminals for liveness detection.

[0005] To address the aforementioned technical problems, embodiments of this application provide a liveness detection method, comprising the following steps: acquiring an infrared image and a speckle image of a target object, and performing frequency domain transformation on the speckle image to obtain a frequency domain image; inputting the infrared image, the speckle image, and the frequency domain image into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model; wherein, the liveness detection model includes a first sub-model that outputs the fusion detection score based on the infrared image and the speckle image, and a second sub-model that outputs the frequency domain detection score based on the frequency domain image; determining a liveness score of the target object based on the fusion detection score and the frequency domain detection score, and determining the target object as a live object if the liveness score is greater than a first preset threshold.

[0006] This application also provides a liveness detection system, including: a camera module, a frequency domain calculation module, a storage module, an execution module, a scoring module, and an output module; the camera module is used to acquire an infrared image and a speckle image of a target object; the frequency domain calculation module is used to perform frequency domain transformation on the speckle image to obtain a frequency domain image; the storage module is used to store a pre-trained liveness detection model, the liveness detection model including a first sub-model and a second sub-model; the execution module is used to input the infrared image, the speckle image, and the frequency domain image into the liveness detection model, and obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model, the first sub-model is used to output the fusion detection score based on the infrared image and the speckle image, and the second sub-model is used to output the frequency domain detection score based on the frequency domain image; the scoring module is used to determine the liveness score of the target object based on the fusion detection score and the frequency domain detection score; the output module is used to determine whether the liveness score is greater than a first preset threshold, and if the liveness score is greater than the first preset threshold, determine that the target object is a live object.

[0007] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described liveness detection method.

[0008] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described liveness detection method.

[0009] The liveness detection method, system, electronic device, and computer-readable storage medium provided in the embodiments of this application, when performing liveness detection, first acquire the infrared image and speckle image of the target object, and perform frequency domain conversion on the speckle image to obtain the frequency domain image corresponding to the speckle image. Then, the infrared image, speckle image, and frequency domain image are all input into a pre-trained liveness detection model. The liveness detection model includes a first sub-model and a second sub-model. The first sub-model outputs a fusion detection score of the target object based on the input infrared image and speckle image, and the second sub-model outputs a frequency domain detection score of the target object based on the input frequency domain image. Finally, the liveness score of the target object is determined based on the fusion detection score and the frequency domain detection score of the target object. If the liveness score of the target object is greater than a first preset threshold, the target object is determined to be a live object. Considering that industry-standard liveness detection technologies require the use of high-cost, high-precision sensors, or rely on infrared, color, or depth maps of the target object for detection, resulting in high equipment and computational costs, the embodiments of this application use infrared, speckle, and frequency domain maps with smaller data volumes that can reflect the details of the target object for liveness detection. The data from different modalities can complement each other, ensuring the accuracy of liveness detection without the need for high-precision, high-cost sensors. This also effectively reduces the number of parameters and computational load in the detection process, thus well meeting the real-time requirements of embedded terminals for liveness detection.

[0010] Furthermore, the step of inputting the infrared image, speckle image, and frequency domain image into a pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model includes: extracting features from the infrared image and speckle image using the first sub-model to obtain an infrared feature map and a speckle feature map; obtaining a fusion classification result based on the infrared feature map and the speckle feature map, and determining the fusion detection score of the target object based on the fusion classification result; extracting features from the frequency domain image using the second sub-model to obtain a frequency domain feature map; obtaining a frequency domain classification result based on the frequency domain feature map, and determining the frequency domain detection score of the target object based on the frequency domain classification result. This application uses infrared images, speckle images, and frequency domain images as multimodal inputs to the liveness detection model. The features of images from different modalities are complementary. Infrared images focus on the reflectivity and detail features of different materials, speckle images focus on the morphology of speckles on different materials, and frequency domain images characterize speckle features from a frequency perspective. This scientifically and reasonably evaluates the liveness of the target object, effectively improving the accuracy of liveness detection.

[0011] Furthermore, the liveness detection model is trained through the following steps: A batch of sample images is input into the liveness detection model to be trained. The sample images include corresponding sample infrared images, sample speckle images, and sample frequency domain images. The sample infrared feature map and sample speckle feature map output by the first sub-model of the liveness detection model to be trained, and the sample frequency domain feature map output by the second sub-model are obtained. Sample infrared classification results and sample speckle classification results are obtained based on the sample infrared feature map and the sample speckle feature map, respectively, and sample fusion classification results are obtained based on the sample infrared feature map and the sample speckle feature map. Sample frequency domain classification results are obtained based on the sample frequency domain feature map. A loss function is constructed based on the sample infrared classification result, the sample speckle classification result, the sample fusion classification result, and the sample frequency domain classification result. The liveness detection model to be trained is trained according to the loss function until convergence, resulting in a trained liveness detection model. When training the model, infrared classification results, speckle classification results, fusion classification results, and frequency domain classification results are all used as constraints. This can prevent the fused features from collapsing into a single mode, thereby training a liveness detection model with higher detection accuracy.

[0012] In addition, the loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term. The infrared loss term is a cross-entropy loss term constructed based on the infrared classification results of the samples. The speckle loss term is a cross-entropy loss term constructed based on the speckle classification results of the samples. The frequency domain loss term is a cross-entropy loss term constructed based on the frequency domain classification results of the samples. The gradient harmonic loss term is constructed through the following steps: calculating the gradient magnitude of each sample image and determining the number of sample images whose gradient magnitudes are within a preset range; determining the gradient density of the batch of sample images based on the number of sample images whose gradient magnitudes are within the preset range and a preset interval length; and constructing the gradient harmonic loss term based on the gradient density, the sample fusion classification results, the labels of the sample images, and the number of sample images in the batch. Considering that some low-quality outliers may affect the detection performance of the final trained liveness detection model, when constructing the loss function, a gradient harmonic loss term can be constructed based on the characteristics of the sample fusion feature map and the sample fusion classification results. This can significantly reduce the impact of outliers on model training and further improve the accuracy of the trained liveness detection model.

[0013] Furthermore, the constructed loss function is expressed by the following formula: L total =aL ir +bL speck +cL GHM +dL FreqWhere, a is the preset first loss weight coefficient, b is the preset second loss weight coefficient, c is the preset third loss weight coefficient, d is the preset fourth loss weight coefficient, and L ir For the infrared loss term, L speck For the speckle loss term, L GHM Let L be the gradient harmonic loss term. Freq This refers to the frequency domain loss term.

[0014] In addition, the first sub-model includes a first input interface, a second input interface, a grouped convolutional backbone network with group 2 connected to the first input interface and the second input interface, and a first output head, a second output head, and a third output head connected to the backbone network. The first input interface is used to receive the input infrared image, the second interface is used to receive the input speckle image, the backbone network is used to extract features from the infrared image and the speckle image to obtain an infrared feature map and a speckle feature map, the first output head is used to output an infrared classification result based on the infrared feature map, the second output head is used to output a speckle classification result based on the speckle feature map, and the third output head is used to output a fusion classification result based on the infrared feature map and the speckle feature map.

[0015] Additionally, the second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to the feature extraction module and the 1×1 convolutional layer respectively. The third input interface is used to receive the input frequency domain map. The first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module are used to extract features from the frequency domain map to obtain a frequency domain feature map. The fourth output head is used to output a frequency domain classification result based on the frequency domain feature map.

[0016] Furthermore, the step of performing frequency domain transformation on the speckle map to obtain a frequency domain map includes: compressing the speckle map according to the JPEG algorithm; and performing a 2D-DCT transformation on the compressed speckle map to calculate the frequency domain map. Embedded terminals have limited computing power and may not be able to run the FTT algorithm quickly and accurately. Therefore, this application first uses the JPEG algorithm to compress the larger speckle map, and then performs a less computationally intensive 2D-DCT transformation to quickly and accurately obtain the frequency domain map corresponding to the speckle map.

[0017] Furthermore, after acquiring the infrared image and speckle image of the target object, and before performing frequency domain transformation on the speckle image to obtain a frequency domain image, the method further includes: detecting facial key points in the acquired infrared image; performing an affine transformation on the infrared image to a preset size based on the facial key points; performing face detection on the acquired speckle image to obtain a face bounding box; cropping the speckle image based on the face bounding box; and scaling the cropped speckle image to the preset size; the frequency domain transformation on the speckle image to obtain a frequency domain image specifically involves performing a frequency domain transformation on the scaled speckle image to obtain a frequency domain image. Since this application is applicable to embedded terminals with limited computing power, preprocessing the infrared image and speckle image is sufficient to retain only the necessary information for liveness detection, which helps improve the speed of liveness detection.

[0018] Furthermore, the determination of the liveness score of the target object based on the fused detection score and the frequency domain detection score is achieved through the following formula: S total =λS GHM +μS freq Where λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, and S GHM S is the score for the fusion detection. freq S is the score for the frequency domain detection. total A liveness score is given to the target object. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0020] Figure 1 This is a flowchart of a liveness detection method provided in one embodiment of this application;

[0021] Figure 2 This is a comparison image of speckle patterns on a living organism and speckle patterns on a prosthesis provided in one embodiment of this application;

[0022] Figure 3 In one embodiment of this application, an infrared image, speckle image, and frequency domain image are input into a pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model.

[0023] Figure 4 This is a schematic diagram of the network architecture of a liveness detection model provided in one embodiment of this application;

[0024] Figure 5 This is a flowchart of training a liveness detection model in one embodiment of this application;

[0025] Figure 6 This is a flowchart of constructing the gradient harmonic loss term in one embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a liveness detection system provided in another embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0029] With the rapid development of deep learning technology, the accuracy of face liveness detection technology has been greatly improved. However, the current liveness detection technology still has the following drawbacks.

[0030] The industry's liveness detection technology relies heavily on detailed facial information, thus requiring sensors with good imaging capabilities and higher costs for capturing images. For some embedded terminals that use sensors with lower costs and lower imaging quality, the accuracy of face liveness detection remains relatively low.

[0031] Commonly used liveness detection technologies in the industry include single-modal liveness detection technology and multi-modal liveness detection technology. The accuracy of single-modal liveness detection technology is difficult to meet the requirements of high-precision liveness detection, and therefore cannot meet the needs of embedded terminals.

[0032] Currently, multimodal liveness detection technologies mostly employ independent, multi-branch approaches for feature extraction, using color images, infrared images, and depth maps as inputs. However, for embedded terminals with limited computing power, the process of recovering the depth map from the speckle image consumes a significant amount of computation, leading to a decrease in liveness detection speed. Furthermore, while color images contain a wealth of information, they are easily affected by lighting conditions, making liveness detection using color images less robust and unable to meet the real-time and high-precision requirements of embedded terminals.

[0033] To address the technical problems of low accuracy, high cost, and poor real-time performance in the aforementioned liveness detection technologies, one embodiment of this application proposes a liveness detection method applied to an electronic device. The electronic device can be a terminal or a server; in this embodiment and the following embodiments, a server is used as an example for illustration. The implementation details of the liveness detection method of this embodiment are described below. These details are provided for ease of understanding and are not essential for implementing this solution.

[0034] The specific process of the liveness detection method in this embodiment can be as follows: Figure 1 As shown, it includes:

[0035] Step 101: Obtain the infrared image and speckle image of the target object, and perform frequency domain transformation on the speckle image to obtain the frequency domain image.

[0036] In the specific implementation, the server is set in an embedded terminal, which is equipped with a camera module. The camera module can take pictures of the target object to obtain the infrared image and speckle image of the target object, and perform frequency domain conversion on the speckle image to obtain the corresponding frequency domain image.

[0037] In some examples, the camera module includes a structured light projector and an infrared lens. The infrared lens can capture an infrared image of the target object, the structured light projector can project a speckle structured light pattern onto the target object, and the infrared lens captures the projection of the speckle structured light pattern onto the target object to obtain a speckle image of the target object.

[0038] In some cases, after the server obtains the infrared image and speckle image of the target object, it can convert the information in the speckle image to the frequency domain according to a preset frequency domain conversion algorithm, thereby obtaining the frequency domain image corresponding to the speckle image.

[0039] In some cases, after obtaining the speckle map of the target object, the server can perform a frequency domain transformation on the speckle map using the FFT (Fast Fourier Transform) algorithm to obtain the corresponding frequency domain map. Although the FFT algorithm is relatively slow, its transformation accuracy and precision are very high.

[0040] In some cases, after the server obtains the speckle map of the target object, it can first compress the speckle map according to the JPEG algorithm (Joint Photographic Experts Group algorithm) to obtain the compressed speckle map, and then perform 2D-DCT (2D Discrete Cosine Transform) transformation on the compressed speckle map to calculate the frequency domain map corresponding to the speckle map.

[0041] Step 102: Input the infrared image, speckle image, and frequency domain image into the pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model.

[0042] Specifically, after the server obtains the infrared image, speckle image, and frequency domain image of the target object, it can input the infrared image, speckle image, and frequency domain image into the pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. The pre-trained liveness detection model includes a first sub-model that outputs the fusion detection score of the target object based on the infrared image and speckle image, and a second sub-model that outputs the frequency domain detection score of the target object based on the frequency domain image.

[0043] In some cases, the pre-trained liveness detection model is stored in the internal memory of the embedded terminal. After the server obtains the infrared image, speckle image, and frequency domain image of the target object, it can call the liveness detection model from the memory to perform liveness detection on the target object.

[0044] In some cases, the pre-trained liveness detection model is stored in the cloud, and the server can send the infrared image, speckle image, and frequency domain image of the target object to the cloud for the liveness detection model to use.

[0045] In its implementation, the pre-trained liveness detection model has three inputs and two outputs. The three inputs receive the infrared image, speckle image, and frequency domain image, respectively. The two outputs output the fused detection score and the frequency domain detection score of the target object, respectively. Specifically, the first sub-model has two inputs and one output, receiving the infrared image and speckle image and outputting the fused detection score of the target object. The second sub-model has one input and one output, receiving the frequency domain image and outputting the frequency domain detection score of the target object.

[0046] It is understandable that there are significant characteristic differences between speckle patterns on living organisms and speckle patterns on prostheses, such as... Figure 2 As shown, the speckle pattern on the human face skin is larger and blurry, while the speckle pattern on the corresponding position of the prosthesis is smaller and clearer. Using speckle patterns can reduce the reliance on high-quality infrared images. At the same time, the edge position of the speckle pattern of the prosthesis is clearer and the frequency range of energy distribution is wider, while the speckle pattern in the human face area is more blurry and the energy is mainly concentrated in the low frequency part. Therefore, the frequency domain map corresponding to the speckle pattern can fully reflect the characteristics of the living body and the prosthesis.

[0047] Step 103: Determine the liveness score of the target object based on the fusion detection score and the frequency domain detection score, and determine the target object as a live object if the liveness score of the target object is greater than the first preset threshold.

[0048] Specifically, after the server obtains the fusion detection score and frequency domain detection score of the target object, it can calculate the liveness score of the target object based on the fusion detection score and frequency domain detection score, and determine whether the liveness score of the target object is greater than a first preset threshold. If the liveness score of the target object is greater than the first preset threshold, the target object is determined to be a live object; if the liveness score of the target object is greater than the first preset threshold, the target object is determined to be a spurious object.

[0049] In some cases, the server can use the sum of the target object's fusion detection score and frequency domain detection score as the target object's liveness score.

[0050] In some examples, the server determines the liveness score of the target object based on the fusion detection score and the frequency domain detection score, using the following formula: S total =λS GHM +μS freq In the formula, λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, and S GHM S is used to score the fusion detection of the target object. freq S is used to score the frequency domain detection of the target object. total For the liveness scoring of the target object, the preset first scoring weight coefficient and the preset second scoring weight coefficient can be set by those skilled in the art according to actual needs, and the embodiments of this application do not specifically limit this.

[0051] In this embodiment, during liveness detection, the infrared image and speckle image of the target object are first acquired, and the speckle image is converted to the frequency domain to obtain the corresponding frequency domain image. Then, the infrared image, speckle image, and frequency domain image are all input into a pre-trained liveness detection model. The liveness detection model includes a first sub-model and a second sub-model. The first sub-model outputs a fusion detection score of the target object based on the input infrared image and speckle image, and the second sub-model outputs a frequency domain detection score of the target object based on the input frequency domain image. Finally, the liveness score of the target object is determined based on the fusion detection score and the frequency domain detection score of the target object. If the liveness score of the target object is greater than a first preset threshold, the target object is determined to be a live object. Considering that industry-standard liveness detection technologies require the use of high-cost, high-precision sensors, or rely on infrared, color, or depth maps of the target object for detection, resulting in high equipment and computational costs, the embodiments of this application use infrared, speckle, and frequency domain maps with smaller data volumes that can reflect the details of the target object for liveness detection. The data from different modalities can complement each other, ensuring the accuracy of liveness detection without the need for high-precision, high-cost sensors. This also effectively reduces the number of parameters and computational load in the detection process, thus well meeting the real-time requirements of embedded terminals for liveness detection.

[0052] In one embodiment, after acquiring the infrared image and speckle image of the target object, before performing frequency domain transformation on the speckle image to obtain the frequency domain image, the server can detect facial key points in the acquired infrared image, perform an affine transformation on the infrared image to a preset size based on the facial key points, then perform face detection on the acquired speckle image to obtain the face bounding box of the speckle image, crop the speckle image based on the face bounding box, and scale the cropped speckle image to a preset size. The frequency domain transformation of the speckle image to obtain the frequency domain image specifically involves performing a frequency domain transformation on the scaled speckle image. When the server inputs the infrared image, speckle image, and frequency domain image into a pre-trained liveness detection model, it specifically inputs the affine transformed infrared image, the scaled speckle image, and the corresponding frequency domain image of the scaled speckle image into the pre-trained liveness detection model. Since the liveness detection method in this embodiment is applicable to embedded terminals with weak computing power, the infrared image and speckle image can be preprocessed to retain the necessary information for liveness detection, which helps to improve the speed of liveness detection.

[0053] In some examples, the default size is 224px × 224px.

[0054] In one embodiment, the infrared image, speckle image, and frequency domain image are input into a pre-trained liveness detection model to obtain the fused detection score and frequency domain detection score of the target object output by the liveness detection model. This can be achieved through methods such as... Figure 3 The steps shown are implemented as follows:

[0055] Step 201: Extract features from the infrared image and speckle image using the first sub-model to obtain the infrared feature image and speckle feature image.

[0056] In the specific implementation, the server inputs the infrared image, speckle image, and frequency domain image into the liveness detection model. The infrared image and speckle image are actually input into the first sub-model. The first sub-model extracts features from the infrared image and speckle image to obtain the infrared feature image and speckle feature image.

[0057] In some cases, the infrared feature map and the speckle feature map are stitched together.

[0058] Step 202: Obtain the fusion classification result based on the infrared feature map and the speckle feature map, and determine the fusion detection score of the target object based on the fusion classification result.

[0059] In the specific implementation, after the first sub-model extracts the infrared feature map and speckle feature map, it obtains the fusion classification result based on the infrared feature map and speckle feature map and outputs it from the output head of the extraction network. Then, based on the fusion classification result, it determines the fusion detection score of the target object and outputs it through the output layer of the first sub-model.

[0060] In some examples, the first sub-model includes a first input interface, a second input interface, a grouped convolutional backbone network with group 2 connected to the first and second input interfaces, and a first output head, a second output head, and a third output head connected to the backbone network. The first input interface receives the input infrared image, the second interface receives the input speckle image, the backbone network extracts features from the infrared image and the speckle image to obtain infrared feature maps and speckle feature maps, the first output head outputs the infrared classification result based on the infrared feature map, the second output head outputs the speckle classification result based on the speckle feature map, and the third output head outputs the fused classification result based on the infrared feature map and the speckle feature map. During model use, an output layer is connected after the third output head, which determines the fused detection score of the target object based on the fused classification result. The first and second output heads are not connected to an output layer; that is, the infrared classification result and the speckle classification result are only used during model training.

[0061] In some examples, the backbone network with group=2 convolution extracts a 7×7×96 infrared feature map and a speckle feature map that are connected together. Before entering the first and second output heads, the 7×7×96 infrared feature map and speckle feature map that are connected together need to be split into a 7×7×48 infrared feature map and a 7×7×48 speckle feature map.

[0062] Step 203: Extract features from the frequency domain map using the second sub-model to obtain the frequency domain feature map.

[0063] In the specific implementation, the server inputs the infrared image, speckle image, and frequency domain image into the liveness detection model. The frequency domain image is actually input into the second sub-model, which extracts features from the frequency domain image to obtain the frequency domain feature map.

[0064] Step 204: Obtain the frequency domain classification result based on the frequency domain feature map, and determine the frequency domain detection score of the target object based on the frequency domain classification result.

[0065] In the specific implementation, after the second sub-model extracts the frequency domain feature map, it obtains the frequency domain classification result based on the frequency domain feature map and outputs it from the output head of the extraction network. Then, based on the frequency domain classification result, it determines the frequency domain detection score of the target object and outputs it through the output layer of the second sub-model.

[0066] In some examples, the second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to both the feature extraction module and the 1×1 convolutional layer. The third input interface receives the input frequency domain map. The first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module extract features from the frequency domain map to obtain a frequency domain feature map. The fourth output head outputs a frequency domain classification result based on the frequency domain feature map. During model usage, an output layer is connected after the fourth output head. This output layer determines the frequency domain detection score of the target object based on the frequency domain classification result.

[0067] In some examples, the input frequency domain plot is a 28×28×64 frequency domain plot.

[0068] In some examples, the network architecture of a liveness detection model is as follows: Figure 4 As shown.

[0069] In some examples, steps 201 and 202 are executed sequentially, and steps 203 and 204 are executed sequentially, but steps 201 and 203 can be executed in parallel.

[0070] In this embodiment, infrared images, speckle images, and frequency domain images are used as multimodal inputs to the liveness detection model. The features of the images from different modalities are complementary. Infrared images focus on the reflectivity and detail features of different materials, speckle images focus on the morphology of speckles on different materials, and frequency domain images characterize the speckle features from the perspective of frequency. This scientifically and reasonably evaluates the liveness of the target object and effectively improves the accuracy of liveness detection.

[0071] In one embodiment, the liveness detection model can be achieved through, for example... Figure 5 The training process shown includes the following steps:

[0072] Step 301: Input a batch of sample images into the liveness detection model to be trained. The sample images include the corresponding sample infrared image, sample speckle image and sample frequency domain image. Obtain the sample infrared feature map and sample speckle feature map output by the first sub-model of the liveness detection model to be trained, as well as the sample frequency domain feature map output by the second sub-model.

[0073] Specifically, when training the liveness detection model, the server first acquires training data, i.e., sample images. A set of sample images contains three images: a sample infrared image, a sample speckle image, and a sample frequency domain image. The objects (faces) in the sample infrared image and the sample speckle image are the same. The sample frequency domain image can be obtained by frequency domain transformation based on the sample speckle image. The sample infrared image, sample speckle image, and sample frequency domain image are all labeled to characterize whether the object in the sample image is a live object or a fake object. The server inputs a batch of sample images into the liveness detection model to be trained, and can obtain the sample infrared feature map and sample speckle feature map output by the first sub-model of the liveness detection model to be trained, as well as the sample frequency domain feature map output by the second sub-model. The batch size can be set by those skilled in the art according to actual needs.

[0074] Step 302: Obtain the sample infrared classification result and sample speckle classification result based on the sample infrared feature map and sample speckle feature map, respectively, and obtain the sample fusion classification result based on the sample infrared feature map and sample speckle feature map.

[0075] Specifically, after the server obtains the sample infrared feature map and sample speckle feature map output by the first sub-model of the liveness detection model to be trained, it can obtain the sample infrared classification result and sample speckle classification result respectively based on the sample infrared feature map and sample speckle feature map, and obtain the sample fusion classification result based on the sample infrared feature map and sample speckle feature map.

[0076] In some examples, the feature extraction network of the first sub-model has three output heads, which output the infrared classification result of the sample, the speckle classification result of the sample, and the fusion classification result of the sample, respectively.

[0077] Step 303: Obtain the sample frequency domain classification result based on the sample frequency domain feature map.

[0078] Specifically, after the server obtains the sample frequency domain feature map output by the second sub-model of the liveness detection model to be trained, it can obtain the sample frequency domain classification result based on the sample frequency domain feature map.

[0079] Step 304: Construct a loss function based on the sample infrared classification results, sample speckle classification results, sample fusion classification results, and sample frequency domain classification results.

[0080] Step 305: Train the liveness detection model to be trained according to the loss function until convergence, and obtain the trained liveness detection model.

[0081] Specifically, after obtaining the sample infrared classification results, sample speckle classification results, sample fusion classification results, and sample frequency domain classification results, the server can construct a loss function based on these results. The server then trains the liveness detection model to be trained using the constructed loss function until convergence, resulting in a trained liveness detection model. The convergence condition can be set by those skilled in the art according to actual needs. Using the infrared classification results, speckle classification results, fusion classification results, and frequency domain classification results as constraints during model training can prevent the fused features from collapsing into a single modality, thus training a liveness detection model with higher detection accuracy.

[0082] In some examples, the server can construct a loss function based on cross-entropy loss, that is, construct an infrared loss term based on the sample infrared classification results, a speckle loss term based on the sample speckle classification results, a fusion loss term based on the sample fusion classification results, and a frequency domain loss term based on the sample frequency domain classification results, and use the sum of these four loss terms as the loss function.

[0083] In some examples, the loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term. The infrared loss term is a cross-entropy loss term constructed based on the infrared classification results of the samples; the speckle loss term is a cross-entropy loss term constructed based on the speckle classification results of the samples; the frequency domain loss term is a cross-entropy loss term constructed based on the frequency domain classification results of the samples; and the gradient harmonic loss term can be constructed as follows: Figure 6 The steps shown are as follows:

[0084] Step 401: Calculate the gradient magnitude of each sample image and determine the number of sample images whose gradient magnitude is within a preset range.

[0085] Specifically, the server can calculate the gradient magnitude of each sample image separately, determine whether the gradient magnitude of each sample image is within a preset range, and determine the number of sample images whose gradient magnitude is within the preset range.

[0086] Step 402: Determine the gradient density of the batch of sample images based on the number of sample images whose gradient magnitude is within a preset range and the preset interval length.

[0087] Specifically, after determining the number of sample images whose gradient magnitude is within a preset range, the server can determine the gradient density of the batch of sample images based on the number of sample images whose gradient magnitude is within the preset range and the preset interval length.

[0088] In some examples, the gradient density of the batch of sample images is determined based on the number of sample images whose gradient magnitudes are within a preset range and a preset interval length, which can be achieved using the following formula:

[0089]

[0090] In the formula, g k Let N be the gradient magnitude of the k-th sample image in the batch, N be the total number of sample images in the batch, ε be a preset constant, and δ be the gradient magnitude of the k-th sample image in the batch. ε (g k (g) represents the gradient magnitude distribution in N sample images. The number of sample images, l ε (g) is the preset interval length, GD(g) k ) represents the gradient density of the k-th sample image in the batch.

[0091] Step 403: Based on the gradient density, sample fusion classification results, sample image labels, and the number of sample images in the batch, a gradient harmonic loss term is constructed.

[0092] Specifically, after the server determines the gradient density of the sample images in the batch, it can construct a gradient harmonic loss term based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch. Considering that some low-quality outliers may affect the detection performance of the final trained liveness detection model, when constructing the loss function, a gradient harmonic loss term can be constructed based on the characteristics of the sample fusion feature map and the sample fusion classification result. This significantly reduces the impact of outliers on model training and further improves the accuracy of the trained liveness detection model.

[0093] In some examples, the gradient harmonic loss term is constructed based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch, and is implemented by the following formula:

[0094]

[0095] In the formula, p k This refers to the fusion classification result of the k-th sample image in the batch. The label for the k-th sample image in the batch, For p k and The constructed cross-entropy loss function, L GHM This is the gradient harmonic loss term constructed.

[0096] In some examples, the loss function obtained by the server construction is: L total =aL ir +bLspeck +cL GHM +dL Freq Where, a is the preset first loss weight coefficient, b is the preset second loss weight coefficient, c is the preset third loss weight coefficient, d is the preset fourth loss weight coefficient, and L ir For the infrared loss term, L speck For speckle loss, L GHM L is the gradient harmonic loss term. Freq This is the frequency domain loss term.

[0097] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0098] Another embodiment of this application relates to a liveness detection system. The implementation details of this liveness detection system are described below. The following details are for ease of understanding and are not essential for implementing this solution. A schematic diagram of the liveness detection system in this embodiment can be seen as follows: Figure 7 As shown, the system includes a camera module 501, a frequency domain calculation module 502, a storage module 503, an execution module 504, a scoring module 505, and an output module 506. The camera module 501 is connected to both the frequency domain calculation module 502 and the execution module 504. The execution module 504 is also connected to the storage module 503 and the scoring module 505, and the scoring module 505 is also connected to the output module 506.

[0099] Camera module 501 is used to acquire infrared images and speckle images of the target object.

[0100] The frequency domain calculation module 502 is used to perform frequency domain transformation on the speckle map to obtain the frequency domain map.

[0101] The storage module 503 is used to store a pre-trained liveness detection model, which includes a first sub-model and a second sub-model.

[0102] The execution module 504 is used to input the infrared image, speckle image and frequency domain image into the liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. The first sub-model is used to output the fusion detection score of the target object based on the infrared image and speckle image, and the second sub-model is used to output the frequency domain detection score of the target object based on the frequency domain image.

[0103] The scoring module 505 is used to determine the liveness score of the target object based on the fusion detection score and the frequency domain detection score.

[0104] The output module 506 is used to determine whether the liveness score is greater than the first preset threshold, and if the liveness score is greater than the first preset threshold, the target object is determined to be a live body.

[0105] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0106] Another embodiment of this application relates to an electronic device, such as... Figure 8 As shown, it includes: at least one processor 601; and a memory 602 communicatively connected to the at least one processor 601; wherein the memory 602 stores instructions executable by the at least one processor 601, the instructions being executed by the at least one processor 601 to enable the at least one processor 601 to perform the liveness detection methods in the above embodiments.

[0107] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0108] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0109] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0110] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for detecting liveness, characterized in that, include: The infrared image and speckle image of the target object are obtained, and the speckle image is transformed in the frequency domain to obtain the frequency domain image; The infrared image, the speckle image, and the frequency domain image are input into a pre-trained liveness detection model to obtain the fusion detection score and the frequency domain detection score of the target object output by the liveness detection model. The liveness detection model includes: a first sub-model that outputs the fusion detection score based on the infrared image and the speckle image through a grouped convolutional backbone network; and a second sub-model that outputs the frequency domain detection score based on the frequency domain image through sequentially connected convolutional layers, a channel attention mechanism module, and a feature extraction module. The liveness detection model is trained based on a loss function, which is constructed through the following steps: The loss function is constructed based on the sample infrared classification results, sample speckle classification results, sample fusion classification results, and sample frequency domain classification results. The sample infrared classification results, sample speckle classification results, and sample fusion classification results are obtained from the sample infrared map and sample speckle map in a batch of sample images, and the sample frequency domain classification results are obtained from the sample frequency domain feature map in the batch of sample images. The loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term; The gradient harmonic loss term is constructed through the following steps: calculating the gradient magnitude of each sample image and determining the number of sample images whose gradient magnitudes fall within a preset range; determining the gradient density of the batch of sample images based on the number of sample images whose gradient magnitudes fall within the preset range and a preset interval length; and constructing the gradient harmonic loss term based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch. The liveness score of the target object is determined based on the fusion detection score and the frequency domain detection score, and the target object is determined to be a live object if the liveness score is greater than a first preset threshold.

2. The live detection method according to claim 1, characterized in that, The step of inputting the infrared image, the speckle image, and the frequency domain image into a pre-trained liveness detection model to obtain the fused detection score and frequency domain detection score of the target object output by the liveness detection model includes: The infrared image and the speckle image are feature extracted using the first sub-model to obtain an infrared feature image and a speckle feature image. The fusion classification result is obtained based on the infrared feature map and the speckle feature map, and the fusion detection score of the target object is determined based on the fusion classification result; The frequency domain map is obtained by extracting features from the frequency domain map using the second sub-model; The frequency domain classification result is obtained based on the frequency domain feature map, and the frequency domain detection score of the target object is determined based on the frequency domain classification result.

3. The live detection method according to claim 2, characterized in that, The liveness detection model is trained through the following steps: A batch of sample images is input into the liveness detection model to be trained. The sample images include corresponding sample infrared images, sample speckle images and sample frequency domain images. The sample infrared feature images and sample speckle feature images output by the first sub-model of the liveness detection model to be trained, as well as the sample frequency domain feature images output by the second sub-model, are obtained. The infrared classification result and speckle classification result of the sample are obtained based on the infrared feature map and speckle feature map of the sample, respectively, and the fusion classification result of the sample is obtained based on the infrared feature map and speckle feature map of the sample. The sample frequency domain classification result is obtained based on the sample frequency domain feature map; The loss function is constructed based on the sample infrared classification results, the sample speckle classification results, the sample fusion classification results, and the sample frequency domain classification results. The liveness detection model to be trained is trained according to the loss function until convergence, thus obtaining the trained liveness detection model.

4. The live detection method according to claim 3, characterized in that, The infrared loss term is a cross-entropy loss term constructed based on the infrared classification results of the samples, the speckle loss term is a cross-entropy loss term constructed based on the speckle classification results of the samples, and the frequency domain loss term is a cross-entropy loss term constructed based on the frequency domain classification results of the samples.

5. The liveness detection method according to claim 4, characterized in that, The gradient density of the batch of sample images is determined based on the number of sample images whose gradient magnitudes are within a preset range and the preset interval length, using the following formula: in, Let N be the gradient magnitude of the k-th sample image in the batch, and N be the total number of sample images in the batch. As a preset constant, The gradient magnitude distribution in the N sample images is as follows: The number of sample images, The preset interval length, The gradient density of the k-th sample image in the batch; The gradient harmonic loss term is constructed based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch, and is implemented through the following formula: in, This refers to the fusion classification result of the k-th sample image in the batch. The label for the k-th sample image in the batch, Based on the and stated The constructed cross-entropy loss function, Let be the gradient harmonic loss term.

6. The liveness detection method according to claim 5, characterized in that, The constructed loss function is expressed by the following formula: L total =aL ir +bL speck +cL GHM +dL Freq Where a is a preset first loss weight coefficient, b is a preset second loss weight coefficient, c is a preset third loss weight coefficient, d is a preset fourth loss weight coefficient, and L ir For the infrared loss term, L speck For the speckle loss term, L GHM Let L be the gradient harmonic loss term. Freq This refers to the frequency domain loss term.

7. The live detection method according to any one of claims 3 to 6, characterized in that, The first sub-model includes a first input interface, a second input interface, a grouped convolutional backbone network with group 2 connected to the first and second input interfaces, and a first output head, a second output head, and a third output head connected to the backbone network. The first input interface is used to receive the input infrared image, and the second input interface is used to receive the input speckle image. The backbone network is used to extract features from the infrared image and the speckle image to obtain an infrared feature map and a speckle feature map. The first output head is used to output an infrared classification result based on the infrared feature map, the second output head is used to output a speckle classification result based on the speckle feature map, and the third output head is used to output a fusion classification result based on the infrared feature map and the speckle feature map.

8. The live detection method according to any one of claims 3 to 6, characterized in that, The second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to the feature extraction module and the 1×1 convolutional layer respectively. The third input interface is used to receive the input frequency domain map. The first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module are used to extract features from the frequency domain map to obtain a frequency domain feature map. The fourth output head is used to output a frequency domain classification result based on the frequency domain feature map.

9. The live detection method according to any one of claims 1 to 6, characterized in that, The step of performing frequency domain transformation on the speckle map to obtain a frequency domain map includes: The speckle image is compressed according to the JPEG algorithm; The compressed speckle map is converted to 2D-DCT to obtain the frequency domain map.

10. The live detection method according to any one of claims 1 to 6, characterized in that, After acquiring the infrared image and speckle image of the target object, and before performing frequency domain transformation on the speckle image to obtain a frequency domain image, the method further includes: The facial key points in the acquired infrared image are detected, and the infrared image is affinely transformed to a preset size based on the facial key points. Face detection is performed on the obtained speckle map to obtain the face bounding box of the speckle map; The speckle image is cropped according to the face frame, and the cropped speckle image is scaled to the preset size. The step of performing frequency domain transformation on the speckle map to obtain a frequency domain map specifically involves performing frequency domain transformation on the scaled speckle map to obtain a frequency domain map.

11. The live detection method according to any one of claims 1 to 6, characterized in that, The liveness score of the target object is determined based on the fused detection score and the frequency domain detection score, using the following formula: S total =λS GHM +μS freq Where λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, and S GHM S is the score for the fusion detection. freq S is the score for the frequency domain detection. total A liveness score is given to the target object.

12. A liveness detection system, characterized in that, It includes a camera module, a frequency domain calculation module, a storage module, an execution module, a scoring module, and an output module; The camera module is used to acquire infrared images and speckle images of the target object; The frequency domain calculation module is used to perform frequency domain transformation on the speckle map to obtain a frequency domain map; The storage module is used to store a pre-trained liveness detection model, which includes a first sub-model and a second sub-model. The execution module is used to input the infrared image, the speckle image, and the frequency domain image into the liveness detection model, and obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. The first sub-model is used to output the fusion detection score based on the infrared image and the speckle image through a grouped convolutional backbone network. The second sub-model is used to output the frequency domain detection score based on the frequency domain image through a sequentially connected convolutional layer, a channel attention mechanism module, and a feature extraction module. The liveness detection model is trained based on a loss function, which is constructed through the following steps: constructing the loss function based on the sample infrared classification result, the sample speckle classification result, the sample fusion classification result, and the sample frequency domain classification result. The sample speckle classification result and the sample fusion classification result are obtained from the sample infrared image and sample speckle image in a batch of sample images. The sample frequency domain classification result is obtained from the sample frequency domain feature image in the batch of sample images. The loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term. The gradient harmonic loss term is constructed through the following steps: calculating the gradient magnitude of each sample image and determining the number of sample images whose gradient magnitude is within a preset range; determining the gradient density of the batch of sample images based on the number of sample images whose gradient magnitude is within the preset range and a preset interval length; and constructing the gradient harmonic loss term based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch. The scoring module is used to determine the liveness score of the target object based on the fusion detection score and the frequency domain detection score; The output module is used to determine whether the liveness score is greater than a first preset threshold, and if the liveness score is greater than the first preset threshold, the target object is determined to be a live body.

13. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the liveness detection method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the liveness detection method according to any one of claims 1 to 11.

Images