Living body detection method and system, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention relates to the technical field of machine vision, and discloses a living body detection method and system, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining an infrared image and a speckle image of a target object, and carrying out the frequency domain conversion of the speckle image, and obtaining a frequency domain image; inputting the infrared image, the speckle image and the frequency domain image into a pre-trained living body detection model, and obtaining a fusion detection score and a frequency domain detection score of the target object output by the living body detection model; and according to the fusion detection score and the frequency domain detection score, determining a living body score of the target object, and when the living body score is greater than a first preset threshold, determining that the target object is a living body. According to the living body detection method provided by the embodiment of the invention, high-precision living body detection can be realized without using a high-cost sensor, data of different modalities are complementary, the parameter quantity and the calculation quantity are effectively reduced, and the requirement of an embedded terminal on the real-time performance of living body detection is well met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine vision technology, and in particular to a living body detection method, system, electronic device, and computer-readable storage medium. Background Art

[0002] Facial recognition technology has been widely used in our daily lives, but it is vulnerable to attacks from various types of prosthetics, such as prints, makeup, masks, and 3D models. To ensure the security of facial recognition systems, facial liveness detection technology has received widespread attention from academia and industry. With the rapid development of deep learning technology, facial liveness detection technology has also been improved.

[0003] Currently, common liveness detection algorithms in the industry include single-modal and multi-modal liveness detection algorithms. Single-modal liveness detection algorithms are not accurate enough to meet the detection requirements of high-precision scenarios. While multi-modal liveness detection algorithms offer higher accuracy, their feature extraction branches are independent, resulting in a high number of parameters and computational complexity, making them difficult to meet the detection requirements of embedded systems and scenarios with high real-time requirements. Furthermore, these liveness detection algorithms rely on detailed facial information, which requires high-cost, high-precision sensors. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a liveness detection method, system, electronic device and computer-readable storage medium, which can achieve high-precision liveness detection without using high-cost sensors. The data of different modalities complement each other, effectively reducing the number of parameters and calculations, and well meeting the real-time liveness detection requirements of embedded terminals.

[0005] To solve the above technical problems, an embodiment of the present application provides a liveness detection method, comprising the following steps: obtaining an infrared image and a speckle pattern of a target object, and performing frequency domain conversion on the speckle pattern to obtain a frequency domain pattern; inputting the infrared image, the speckle pattern, and the frequency domain pattern into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model; wherein the liveness detection model includes a first sub-model that outputs the fusion detection score based on the infrared image and the speckle pattern, and a second sub-model that outputs the frequency domain detection score based on the frequency domain pattern; determining a liveness score of the target object based on the fusion detection score and the frequency domain detection score, and determining that the target object is live if the liveness score is greater than a first preset threshold.

[0006] An embodiment of the present application further provides a liveness detection system, comprising: a camera module, a frequency domain calculation module, a storage module, an execution module, a scoring module, and an output module; the camera module is configured to obtain an infrared image and a speckle pattern of a target object; the frequency domain calculation module is configured to perform frequency domain conversion on the speckle pattern to obtain a frequency domain pattern; the storage module is configured to store a pre-trained liveness detection model, wherein the liveness detection model includes a first sub-model and a second sub-model; the execution module is configured to input the infrared image, the speckle pattern, and the frequency domain pattern into the liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model, wherein the first sub-model is configured to output the fusion detection score based on the infrared image and the speckle pattern, and the second sub-model is configured to output the frequency domain detection score based on the frequency domain pattern; the scoring module is configured to determine a liveness score of the target object based on the fusion detection score and the frequency domain detection score; and the output module is configured to determine whether the liveness score is greater than a first preset threshold, and if the liveness score is greater than the first preset threshold, determine that the target object is live.

[0007] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned liveness detection method.

[0008] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned living body detection method when executed by a processor.

[0009] The liveness detection method, system, electronic device, and computer-readable storage medium provided by the embodiments of the present application, when performing liveness detection, first obtain an infrared image and a speckle pattern of a target object, and perform frequency domain conversion on the speckle pattern to obtain a frequency domain map corresponding to the speckle pattern. The infrared image, speckle pattern, and frequency domain map are then input into a pre-trained liveness detection model. The liveness detection model includes a first sub-model and a second sub-model. The first sub-model outputs a fusion detection score of the target object based on the input infrared image and speckle pattern, and the second sub-model outputs a frequency domain detection score of the target object based on the input frequency domain map. Finally, a liveness score of the target object is determined based on the fusion detection score and the frequency domain detection score of the target object. If the liveness score of the target object is greater than a first preset threshold, the target object is determined to be live. Considering that the common liveness detection technology in the industry requires the use of high-cost and high-precision sensors, or is based on infrared images, color images, and depth images of the target object for detection, the equipment cost and computing cost are relatively high. The embodiments of the present application use infrared images, speckle images, and frequency domain images with small data volume and that can reflect the details of the target object for liveness detection. The data of different modalities can complement each other. While ensuring the accuracy of liveness detection, there is no need to use high-precision and high-cost sensors. At the same time, the number of parameters and the amount of calculation in the detection process are effectively reduced, which well meets the real-time requirements of embedded terminals for liveness detection.

[0010] In addition, the method of inputting the infrared image, the speckle pattern, and the frequency domain image into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model includes: extracting features from the infrared image and the speckle pattern using the first sub-model to obtain an infrared feature map and a speckle feature map; obtaining a fusion classification result based on the infrared feature map and the speckle feature map, and determining the fusion detection score of the target object based on the fusion classification result; extracting features from the frequency domain image using the second sub-model to obtain a frequency domain feature map; obtaining a frequency domain classification result based on the frequency domain feature map, and determining the frequency domain detection score of the target object based on the frequency domain classification result. This application uses infrared images, speckle patterns, and frequency domain images as multimodal inputs to the liveness detection model. The features of images of different modalities complement each other. The infrared image focuses on the reflective properties and detailed features of different materials, the speckle pattern focuses on the morphology of speckle on different materials, and the frequency domain image characterizes the characteristics of speckle from the perspective of frequency. This method scientifically and rationally evaluates the liveness of the target object and effectively improves the accuracy of liveness detection.

[0011] In addition, the liveness detection model is trained by the following steps: inputting a batch of sample images into the liveness detection model to be trained, the sample images including corresponding sample infrared images, sample speckle images, and sample frequency domain images, obtaining sample infrared feature images and sample speckle feature images output by the first sub-model of the liveness detection model to be trained, and sample frequency domain feature images output by the second sub-model; obtaining sample infrared classification results and sample speckle classification results based on the sample infrared feature images and the sample speckle feature images, respectively, and obtaining sample fusion classification results based on the sample infrared feature images and the sample speckle feature images; obtaining sample frequency domain classification results based on the sample frequency domain feature images; constructing a loss function based on the sample infrared classification results, the sample speckle classification results, the sample fusion classification results, and the sample frequency domain classification results; and training the liveness detection model to be trained according to the loss function until convergence to obtain a trained liveness detection model. When training the model, the infrared classification results, speckle classification results, fusion classification results, and frequency domain classification results are all used as constraints to avoid the collapse of the fused features into a single mode, thereby training a liveness detection model with higher detection accuracy.

[0012] In addition, the loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term. The infrared loss term is a cross-entropy loss term constructed based on the sample infrared classification result, the speckle loss term is a cross-entropy loss term constructed based on the sample speckle classification result, and the frequency domain loss term is a cross-entropy loss term constructed based on the sample frequency domain classification result. The gradient harmonic loss term is constructed by the following steps: calculating the gradient modulus of each sample image respectively, and determining the number of sample images whose gradient modulus is within a preset range; determining the gradient density of the sample images of the batch based on the number of sample images whose gradient modulus is within the preset range and a preset interval length; and constructing the gradient harmonic loss term based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch. Considering that some low-quality outliers may affect the detection effect of the finally trained liveness detection model, when constructing the loss function, we can construct a gradient harmonic loss term based on the characteristics of the sample fusion feature map and the sample fusion classification results, which can greatly reduce the impact of outliers on model training and further improve the accuracy of the trained liveness detection model.

[0013] In addition, the constructed loss function is expressed by the following formula: total =aL ir +bL speck +cL GHM +dL Freq, where a is the preset first loss weight coefficient, b is the preset second loss weight coefficient, c is the preset third loss weight coefficient, d is the preset fourth loss weight coefficient, L ir is the infrared loss term, L speck is the speckle loss term, L GHM is the gradient reconciliation loss term, L Freq is the frequency domain loss term.

[0014] In addition, the first sub-model includes a first input interface, a second input interface, a backbone network with a group convolution group of 2 connected to the first input interface and the second input interface, and a first output head, a second output head, and a third output head connected to the backbone network. The first input interface is used to receive the input infrared image, and the second interface is used to receive the input speckle pattern. The backbone network is used to extract features of the infrared image and features of the speckle pattern to obtain an infrared feature map and a speckle feature map. The first output head is used to output an infrared classification result based on the infrared feature map, the second output head is used to output a speckle classification result based on the speckle feature map, and the third output head is used to output a fusion classification result based on the infrared feature map and the speckle feature map.

[0015] In addition, the second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to the feature extraction module and the 1×1 convolutional layer respectively. The third input interface is used to receive the input frequency domain map, the first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module are used to extract features of the frequency domain map to obtain a frequency domain feature map, and the fourth output head is used to output a frequency domain classification result based on the frequency domain feature map.

[0016] Furthermore, performing frequency domain conversion on the speckle pattern to obtain a frequency domain map includes: compressing the speckle pattern according to the JPEG algorithm; and performing 2D-DCT conversion on the compressed speckle pattern to calculate the frequency domain map. Embedded terminals have limited computing power and may not be able to quickly and accurately run the FTT algorithm. Therefore, this application first compresses larger speckle patterns using the JPEG algorithm, and then performs the relatively computationally intensive 2D-DCT conversion, thereby quickly and accurately obtaining the frequency domain map corresponding to the speckle pattern.

[0017] In addition, after acquiring the infrared image and speckle pattern of the target object, and before performing frequency domain conversion on the speckle pattern to obtain a frequency domain map, the method further includes: detecting facial key points in the acquired infrared image, and performing an affine transformation on the infrared image to a preset size based on the facial key points; performing face detection on the acquired speckle pattern to obtain a face frame for the speckle pattern; cropping the speckle pattern based on the face frame, and scaling the cropped speckle pattern to the preset size; and performing frequency domain conversion on the speckle pattern to obtain a frequency domain map, specifically: performing frequency domain conversion on the scaled speckle pattern to obtain a frequency domain map. Because the present application is applicable to embedded terminals with relatively low computing power, the infrared image and speckle pattern can be preprocessed to retain the necessary information for liveness detection, thereby helping to improve the speed of liveness detection.

[0018] In addition, the liveness score of the target object is determined according to the fusion detection score and the frequency domain detection score, which is achieved by the following formula: total =λS GHM +μS freq , where λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, S GHM Score the fusion detection, S freq Score the frequency domain detection, S total A living body score is provided for the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0020] Figure 1 This is a flowchart of a liveness detection method provided by one embodiment of the present application;

[0021] Figure 2 This is a comparison diagram of speckle on a living body and speckle on a prosthesis provided in an embodiment of the present application;

[0022] Figure 3 This is a flowchart of an embodiment of the present application, in which an infrared image, a speckle image, and a frequency domain image are input into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of a target object output by the liveness detection model;

[0023] Figure 4 This is a schematic diagram of the network architecture of a liveness detection model provided in one embodiment of the present application;

[0024] Figure 5 This is a flowchart for training a liveness detection model in one embodiment of the present application;

[0025] Figure 6 This is a flowchart for constructing a gradient reconciliation loss term in one embodiment of the present application;

[0026] Figure 7 is a schematic diagram of a liveness detection system provided by another embodiment of the present application;

[0027] Figure 8 It is a structural diagram of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0029] With the rapid development of deep learning technology, the accuracy of face liveness detection technology has been greatly improved. However, the current liveness detection technology still has the following drawbacks.

[0030] The industry's liveness detection technology is highly dependent on detailed facial information, and therefore requires sensors with better imaging effects and higher costs to capture. For some embedded terminals that use sensors with lower costs and lower imaging quality, the accuracy of facial liveness detection is still relatively low.

[0031] Commonly used liveness detection technologies in the industry include single-modality liveness detection technology and multi-modality liveness detection technology. The accuracy of single-modality liveness detection technology is difficult to meet the requirements of high-precision liveness detection, and naturally cannot meet the usage needs of embedded terminals.

[0032] Currently, multimodal liveness detection technologies mostly use an independent multi-branch approach to extract features, using color, infrared, and depth images as input. However, for embedded terminals with limited computing power, the process of converting speckle patterns into depth images consumes a significant amount of computing power, which slows down liveness detection. Furthermore, while color images contain more information, they are easily affected by ambient lighting. Using color images for liveness detection is not robust enough to meet the real-time and high-precision requirements of embedded terminals.

[0033] To address the technical issues of low accuracy, high cost, and poor real-time performance in the aforementioned liveness detection technologies, one embodiment of the present application proposes a liveness detection method for use in an electronic device, where the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The following describes the implementation details of the liveness detection method of this embodiment. The following content is provided solely for ease of understanding and is not essential for implementing this solution.

[0034] The specific process of the living body detection method of this embodiment can be as follows: Figure 1 Shown, including:

[0035] Step 101: Acquire an infrared image and a speckle pattern of a target object, and perform frequency domain conversion on the speckle pattern to obtain a frequency domain pattern.

[0036] In a specific implementation, the server is set in an embedded terminal equipped with a camera module. The camera module can shoot the target object to obtain the infrared image and speckle pattern of the target object, and perform frequency domain conversion on the speckle pattern to obtain the frequency domain map corresponding to the speckle pattern.

[0037] In some examples, the camera module includes a structured light projector and an infrared lens. The infrared lens can photograph a target object to obtain an infrared image of the target object. The structured light projector can project a speckle structured light pattern onto the target object. The infrared lens photographs the projection of the speckle structured light pattern projected onto the target object to obtain a speckle pattern of the target object.

[0038] In some examples, after obtaining the infrared image and speckle pattern of the target object, the server can convert the information in the speckle pattern into the frequency domain according to a preset frequency domain conversion algorithm, thereby obtaining a frequency domain map corresponding to the speckle pattern.

[0039] In some examples, after obtaining the speckle pattern of the target object, the server can perform a frequency domain transform on the speckle pattern based on the FFT algorithm (Fast Fourier Transformation) to obtain the corresponding frequency domain image of the speckle pattern. Although the FFT algorithm is slow to execute, it has very high precision and accuracy.

[0040] In some examples, after obtaining the speckle pattern of the target object, the server can first compress the speckle pattern according to the JPEG algorithm (Joint Photographic Experts Group algorithm) to obtain a compressed speckle pattern, and then perform a 2D-DCT (two-dimensional discrete cosine transform) on the compressed speckle pattern to calculate the frequency domain map corresponding to the speckle pattern.

[0041] Step 102: Input the infrared image, speckle image, and frequency domain image into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model.

[0042] Specifically, after obtaining the infrared image, speckle pattern, and frequency domain map of the target object, the server can input the infrared image, speckle pattern, and frequency domain map into a pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. The pre-trained liveness detection model includes a first sub-model that outputs the fusion detection score of the target object based on the infrared image and speckle pattern, and a second sub-model that outputs the frequency domain detection score of the target object based on the frequency domain map.

[0043] In some examples, a pre-trained liveness detection model is stored in a memory inside the embedded terminal. After the server obtains the infrared image, speckle image, and frequency domain image of the target object, it can call the liveness detection model from the memory to perform liveness detection on the target object.

[0044] In some examples, a pre-trained liveness detection model is stored in the cloud, and the server can send the infrared image, speckle image, and frequency domain image of the target object to the cloud for use by the liveness detection model.

[0045] In the specific implementation, the pre-trained liveness detection model has three inputs and two outputs. The three inputs receive the infrared image, speckle pattern, and frequency domain image, respectively, while the two outputs output the fusion detection score and frequency domain detection score of the target object, respectively. Specifically, the first sub-model has two inputs and one output, receiving the infrared image and speckle pattern and outputting the fusion detection score of the target object. The second sub-model has one input and one output, receiving the frequency domain image and outputting the frequency domain detection score of the target object.

[0046] It is understandable that there are obvious characteristic differences between speckles on living bodies and those on prostheses, such as Figure 2 As shown in the figure, the speckle on the human face is large and blurred, while the speckle on the corresponding position of the prosthetic prop is small and clear. The use of speckle patterns can reduce the dependence on high-quality infrared images. At the same time, the edge position of the prosthetic speckle pattern is clearer and the frequency range of energy distribution is wider, while the speckle in the human face area is more blurred and the energy is mainly concentrated in the low-frequency part. Therefore, the frequency domain diagram corresponding to the speckle pattern can fully reflect the characteristics of the living body and the prosthesis.

[0047] Step 103 : determining a liveness score of the target object according to the fusion detection score and the frequency domain detection score, and determining that the target object is alive when the liveness score of the target object is greater than a first preset threshold.

[0048] Specifically, after the server obtains the fusion detection score and frequency domain detection score of the target object, it can calculate the liveness score of the target object based on the fusion detection score and the frequency domain detection score, and determine whether the liveness score of the target object is greater than a first preset threshold. If the liveness score of the target object is greater than the first preset threshold, the target object is determined to be alive; if the liveness score of the target object is greater than the first preset threshold, the target object is determined to be a prosthesis.

[0049] In some examples, the server may use the sum of the fusion detection score and the frequency domain detection score of the target object as the liveness score of the target object.

[0050] In some examples, the server determines the liveness score of the target object based on the fusion detection score and the frequency domain detection score of the target object, which is implemented by the following formula: S total =λS GHM +μS freq , where λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, S GHM Score the fusion detection of the target object, S freq Score the frequency domain detection of the target object, S total For scoring the live body of the target object, the preset first scoring weight coefficient and the preset second scoring weight coefficient can be set by those skilled in the art according to actual needs, and the embodiments of the present application do not specifically limit this.

[0051] In this embodiment, when performing liveness detection, an infrared image and a speckle pattern of the target object are first obtained, and the speckle pattern is converted into the frequency domain to obtain a frequency domain map corresponding to the speckle pattern. The infrared image, speckle pattern, and frequency domain map are then input into a pre-trained liveness detection model. The liveness detection model includes a first sub-model and a second sub-model. The first sub-model outputs a fusion detection score of the target object based on the input infrared image and speckle pattern, and the second sub-model outputs a frequency domain detection score of the target object based on the input frequency domain map. Finally, a liveness score of the target object is determined based on the fusion detection score and the frequency domain detection score of the target object. If the liveness score of the target object is greater than a first preset threshold, the target object is determined to be live. Considering that the common liveness detection technology in the industry requires the use of high-cost and high-precision sensors, or is based on infrared images, color images, and depth images of the target object for detection, the equipment cost and computing cost are relatively high. The embodiments of the present application use infrared images, speckle images, and frequency domain images with small data volume and that can reflect the details of the target object for liveness detection. The data of different modalities can complement each other. While ensuring the accuracy of liveness detection, there is no need to use high-precision and high-cost sensors. At the same time, the number of parameters and the amount of calculation in the detection process are effectively reduced, which well meets the real-time requirements of embedded terminals for liveness detection.

[0052] In one embodiment, after acquiring an infrared image and speckle pattern of a target object, the server may detect facial key points in the acquired infrared image and, before performing frequency domain conversion on the speckle pattern to obtain a frequency domain map, affine transform the infrared image to a preset size based on the facial key points in the infrared image. Face detection is then performed on the acquired speckle pattern to obtain a face frame for the speckle pattern. The speckle pattern is then cropped based on the face frame and scaled to a preset size. Frequency domain conversion is then performed on the speckle pattern to obtain a frequency domain map, specifically by performing frequency domain conversion on the scaled speckle pattern to obtain a frequency domain map. When the server inputs the infrared image, speckle pattern, and frequency domain map into a pre-trained liveness detection model, the server specifically inputs the affine transformed infrared image, the scaled speckle pattern, and the frequency domain map corresponding to the scaled speckle pattern into the pre-trained liveness detection model. Since the liveness detection method of this embodiment is applicable to embedded terminals with weak computing power, the infrared image and speckle pattern can be preprocessed to retain the necessary information for liveness detection, which helps to improve the speed of liveness detection.

[0053] In some examples, the preset size is 224px×224px.

[0054] In one embodiment, the infrared image, speckle image and frequency domain image are input into the pre-trained liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. Figure 3 The steps shown in the figure are as follows:

[0055] Step 201 : extracting features from the infrared image and the speckle pattern using the first sub-model to obtain an infrared feature map and a speckle feature map.

[0056] In a specific implementation, the server inputs the infrared image, speckle pattern and frequency domain image into the living body detection model, wherein the infrared image and speckle pattern are actually input into the first sub-model, and the first sub-model extracts features from the infrared image and speckle pattern to obtain the infrared feature map and the speckle feature map.

[0057] In some examples, the infrared signature and the speckle signature are stitched together.

[0058] Step 202 : obtaining a fusion classification result based on the infrared feature map and the speckle feature map, and determining a fusion detection score of the target object based on the fusion classification result.

[0059] In the specific implementation, after the first sub-model extracts the infrared feature map and the speckle feature map, it obtains the fusion classification result based on the infrared feature map and the speckle feature map and outputs it from the output head of the extraction network. Then, the fusion detection score of the target object is determined based on the fusion classification result and output through the output layer of the first sub-model.

[0060] In some examples, the first sub-model includes a first input interface, a second input interface, a backbone network with a group convolution of two connected to the first and second input interfaces, and first, second, and third output heads connected to the backbone network. The first input interface is used to receive an infrared image as input, and the second interface is used to receive a speckle pattern as input. The backbone network is used to extract features from the infrared image and the speckle pattern to obtain an infrared feature map and a speckle feature map. The first output head is used to output an infrared classification result based on the infrared feature map, the second output head is used to output a speckle classification result based on the speckle feature map, and the third output head is used to output a fused classification result based on the infrared feature map and the speckle feature map. When the model is in use, an output layer is connected after the third output head, and the output layer is used to determine a fused detection score for the target object based on the fused classification result. The first and second output heads are not connected to the output layer, meaning that the infrared and speckle classification results are only used during model training.

[0061] In some examples, the backbone network with group convolution group=2 extracts a connected 7×7×96 infrared feature map and speckle feature map. Before entering the first output head and the second output head, the connected 7×7×96 infrared feature map and speckle feature map need to be split into a 7×7×48 infrared feature map and a 7×7×48 speckle feature map.

[0062] Step 203: extract features from the frequency domain graph using the second sub-model to obtain a frequency domain feature graph.

[0063] In a specific implementation, the server inputs the infrared image, speckle image and frequency domain image into the living body detection model, wherein the frequency domain image is actually input into the second sub-model, and the second sub-model extracts features from the frequency domain image to obtain a frequency domain feature image.

[0064] Step 204 : Obtain a frequency domain classification result according to the frequency domain feature map, and determine a frequency domain detection score of the target object according to the frequency domain classification result.

[0065] In the specific implementation, after the second sub-model extracts the frequency domain feature map, it obtains the frequency domain classification result based on the frequency domain feature map and outputs it from the output head of the extraction network. Then, the frequency domain detection score of the target object is determined based on the frequency domain classification result and output through the output layer of the second sub-model.

[0066] In some examples, the second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to the feature extraction module and the 1×1 convolutional layer, respectively. The third input interface is used to receive an input frequency domain map. The first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module are used to extract features from the frequency domain map to obtain a frequency domain feature map. The fourth output head is used to output a frequency domain classification result based on the frequency domain feature map. When the model is in use, an output layer is also connected after the fourth output head, and the output layer is used to determine a frequency domain detection score for the target object based on the frequency domain classification result.

[0067] In some examples, the input frequency domain image is a 28×28×64 frequency domain image.

[0068] In some examples, the network architecture of the liveness detection model is as follows Figure 4 shown.

[0069] In some examples, step 201 and step 202 are executed sequentially, and step 203 and step 204 are executed sequentially, but step 201 and step 203 can be executed in parallel.

[0070] This embodiment uses infrared images, speckle patterns, and frequency domain images as multimodal inputs for the liveness detection model. The features of images from different modalities complement each other. Infrared images focus on the reflective properties and detailed features of different materials, speckle patterns focus on the morphology of speckle on different materials, and frequency domain images characterize speckle characteristics from a frequency perspective. This allows for a scientific and reasonable evaluation of the liveness of the target object, effectively improving the accuracy of liveness detection.

[0071] In one embodiment, the liveness detection model can be Figure 5 The steps shown are trained, including:

[0072] In step 301, a batch of sample images is input into the liveness detection model to be trained. The sample images include corresponding sample infrared images, sample speckle images, and sample frequency domain images. The sample infrared feature images and sample speckle feature images output by the first sub-model of the liveness detection model to be trained, as well as the sample frequency domain feature images output by the second sub-model, are obtained.

[0073] Specifically, when training a liveness detection model to be trained, the server first obtains training data, namely sample images. A set of sample images contains three images: a sample infrared image, a sample speckle pattern, and a sample frequency domain image. The objects (human faces) in the sample infrared image and the sample speckle pattern are the same. The sample frequency domain image can be obtained by frequency domain conversion based on the sample speckle pattern. The sample infrared image, sample speckle pattern, and sample frequency domain image are all annotated with labels to indicate whether the object in the sample image is live or prosthetic. The server inputs a batch of sample images into the liveness detection model to be trained and obtains the sample infrared feature map and sample speckle feature map output by the first sub-model of the liveness detection model to be trained, as well as the sample frequency domain feature map output by the second sub-model. The batch size can be set by those skilled in the art according to actual needs.

[0074] Step 302 : Obtain a sample infrared classification result and a sample speckle classification result according to the sample infrared feature map and the sample speckle feature map respectively, and obtain a sample fusion classification result according to the sample infrared feature map and the sample speckle feature map.

[0075] Specifically, after the server obtains the sample infrared feature map and the sample speckle feature map output by the first sub-model of the liveness detection model to be trained, it can obtain the sample infrared classification result and the sample speckle classification result based on the sample infrared feature map and the sample speckle feature map, respectively, and obtain the sample fusion classification result based on the sample infrared feature map and the sample speckle feature map.

[0076] In some examples, the feature extraction network of the first sub-model has three output heads, which respectively output sample infrared classification results, sample speckle classification results, and sample fusion classification results.

[0077] Step 303: Obtain the sample frequency domain classification result according to the sample frequency domain feature map.

[0078] Specifically, after the server obtains the sample frequency domain feature map output by the second sub-model of the living body detection model to be trained, it can obtain the sample frequency domain classification result according to the sample frequency domain feature map.

[0079] Step 304 : constructing a loss function based on the sample infrared classification results, the sample speckle classification results, the sample fusion classification results, and the sample frequency domain classification results.

[0080] Step 305: Train the liveness detection model to be trained according to the loss function until convergence, and obtain a trained liveness detection model.

[0081] Specifically, after obtaining the sample infrared classification results, sample speckle classification results, sample fusion classification results, and sample frequency domain classification results, the server can construct a loss function based on the sample infrared classification results, sample speckle classification results, sample fusion classification results, and sample frequency domain classification results. The server trains the liveness detection model to be trained using the constructed loss function until convergence, thereby obtaining a trained liveness detection model. The convergence conditions can be configured by those skilled in the art based on actual needs. During model training, using the infrared classification results, speckle classification results, fusion classification results, and frequency domain classification results as constraints can prevent the fused features from collapsing into a single modality, thereby training a liveness detection model with higher detection accuracy.

[0082] In some examples, the server can construct a loss function based on the cross entropy loss, that is, construct an infrared loss term based on the sample infrared classification result, construct a speckle loss term based on the sample speckle classification result, construct a fusion loss term based on the sample fusion classification result, and construct a frequency domain loss term based on the sample frequency domain classification result, and use the sum of these four loss terms as the loss function.

[0083] In some examples, the loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term, and a gradient harmonic loss term. The infrared loss term is a cross entropy loss term constructed based on the infrared classification results of the samples, the speckle loss term is a cross entropy loss term constructed based on the speckle classification results of the samples, the frequency domain loss term is a cross entropy loss term constructed based on the frequency domain classification results of the samples, and the gradient harmonic loss term can be obtained by Figure 6 The steps shown are constructed, including:

[0084] Step 401 : Calculate the gradient modulus of each sample image respectively, and determine the number of sample images whose gradient modulus is within a preset range.

[0085] Specifically, the server may calculate the gradient modulus of each sample image respectively, determine in turn whether the gradient modulus of each sample image is within a preset range, and determine the number of sample images whose gradient modulus is within the preset range.

[0086] Step 402 : determining the gradient density of the sample images of the batch according to the number of sample images whose gradient modulus is within a preset range and a preset interval length.

[0087] Specifically, after determining the number of sample images whose gradient modulus is within a preset range, the server can determine the gradient density of the sample images of the batch according to the number of sample images whose gradient modulus is within the preset range and the preset interval length.

[0088] In some examples, the gradient density of the sample images of the batch is determined based on the number of sample images whose gradient modulus is within a preset range and a preset interval length, which can be achieved by the following formula:

[0089]

[0090] Where, g k is the gradient modulus of the kth sample graph in the batch, N is the total number of sample graphs in the batch, ε is a preset constant, δ ε (g k ,g) is the distribution of gradient modulus in N sample graphs The number of sample graphs, l ε (g) is the preset interval length, GD(g k ) is the gradient density of the k-th sample graph in the batch.

[0091] Step 403 : constructing a gradient harmonic loss term based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch.

[0092] Specifically, after the server determines the gradient density of the sample graphs in the batch, it can construct a gradient harmonic loss term based on the gradient density, the sample fusion classification results, the sample graph labels, and the number of sample graphs in the batch. Considering that some low-quality outliers may affect the detection effect of the ultimately trained liveness detection model, when constructing the loss function, the gradient harmonic loss term can be constructed based on the characteristics of the sample fusion feature graph and the sample fusion classification results. This can significantly reduce the impact of outliers on model training and further improve the accuracy of the trained liveness detection model.

[0093] In some examples, a gradient harmonic loss term is constructed based on the gradient density, the sample fusion classification result, the label of the sample image, and the number of sample images in the batch, and is implemented by the following formula:

[0094]

[0095] Where p k is the fusion classification result of the k-th sample graph in the batch, is the label of the kth sample image in the batch, Based on p k and The constructed cross entropy loss function, L GHM is the constructed gradient harmonic loss term.

[0096] In some examples, the loss function constructed by the server is: L total =aL ir +bLspeck +cL GHM +dL Freq , where a is the preset first loss weight coefficient, b is the preset second loss weight coefficient, c is the preset third loss weight coefficient, d is the preset fourth loss weight coefficient, L ir is the infrared loss term, L speck is the speckle loss term, L GHM is the gradient harmonic loss term, L Freq is the frequency domain loss term.

[0097] The steps of the various methods above are divided only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0098] Another embodiment of the present application relates to a liveness detection system. The following is a detailed description of the implementation details of the liveness detection system of this embodiment. The following content is only for the convenience of understanding the implementation details and is not necessary for the implementation of this solution. The schematic diagram of the liveness detection system of this embodiment can be as follows: Figure 7 As shown, the system includes a camera module 501, a frequency domain calculation module 502, a storage module 503, an execution module 504, a scoring module 505, and an output module 506. The camera module 501 is connected to the frequency domain calculation module 502 and the execution module 504. The execution module 504 is also connected to the storage module 503 and the scoring module 505, and the scoring module 505 is also connected to the output module 506.

[0099] The camera module 501 is used to obtain an infrared image and a speckle pattern of a target object.

[0100] The frequency domain calculation module 502 is used to perform frequency domain conversion on the speckle pattern to obtain a frequency domain map.

[0101] The storage module 503 is used to store a pre-trained liveness detection model, where the liveness detection model includes a first sub-model and a second sub-model.

[0102] The execution module 504 is used to input the infrared image, speckle pattern, and frequency domain image into the liveness detection model to obtain the fusion detection score and frequency domain detection score of the target object output by the liveness detection model. The first sub-model is used to output the fusion detection score of the target object based on the infrared image and speckle pattern, and the second sub-model is used to output the frequency domain detection score of the target object based on the frequency domain image.

[0103] The scoring module 505 is configured to determine a liveness score of the target object according to the fusion detection score and the frequency domain detection score.

[0104] The output module 506 is configured to determine whether the liveness score is greater than a first preset threshold, and determine that the target object is live if the liveness score is greater than the first preset threshold.

[0105] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0106] Another embodiment of the present application relates to an electronic device, such as Figure 8 As shown, it includes: at least one processor 601; and a memory 602 that is communicatively connected to the at least one processor 601; wherein the memory 602 stores instructions that can be executed by the at least one processor 601, and the instructions are executed by the at least one processor 601 to enable the at least one processor 601 to execute the liveness detection method in the above-mentioned embodiments.

[0107] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0108] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0109] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0110] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0111] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for detecting a living body, characterized in that: include: Acquire an infrared image and a speckle image of the target object, and perform frequency domain conversion on the speckle image to obtain a frequency domain image; Inputting the infrared image, the speckle image and the frequency domain image into a pre-trained liveness detection model, and obtaining a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model; wherein the liveness detection model includes a first sub-model that outputs the fusion detection score based on the infrared image and the speckle image, and a second sub-model that outputs the frequency domain detection score based on the frequency domain image; A liveness score of the target object is determined according to the fusion detection score and the frequency domain detection score, and when the liveness score is greater than a first preset threshold, the target object is determined to be live.

2. The method for detecting living body according to claim 1, characterized in that: The step of inputting the infrared image, the speckle image, and the frequency domain image into a pre-trained liveness detection model to obtain a fusion detection score and a frequency domain detection score of the target object output by the liveness detection model includes: Extracting features from the infrared image and the speckle image by using the first sub-model to obtain an infrared feature image and a speckle feature image; Acquire a fusion classification result according to the infrared feature map and the speckle feature map, and determine a fusion detection score of the target object according to the fusion classification result; Performing feature extraction on the frequency domain graph by using the second sub-model to obtain a frequency domain feature graph; A frequency domain classification result is obtained according to the frequency domain feature map, and a frequency domain detection score of the target object is determined according to the frequency domain classification result.

3. The method for liveness detection according to claim 2, characterized in that: The liveness detection model is trained by the following steps: Inputting a batch of sample images into the liveness detection model to be trained, the sample images including corresponding sample infrared images, sample speckle images and sample frequency domain images, obtaining the sample infrared feature images and sample speckle feature images output by the first sub-model of the liveness detection model to be trained, and the sample frequency domain feature images output by the second sub-model; Obtaining a sample infrared classification result and a sample speckle classification result according to the sample infrared feature map and the sample speckle feature map respectively, and obtaining a sample fusion classification result according to the sample infrared feature map and the sample speckle feature map; Obtaining a sample frequency domain classification result according to the sample frequency domain feature map; Constructing a loss function according to the sample infrared classification result, the sample speckle classification result, the sample fusion classification result, and the sample frequency domain classification result; The liveness detection model to be trained is trained according to the loss function until convergence to obtain a trained liveness detection model.

4. The method for liveness detection according to claim 3, characterized in that: The loss function includes an infrared loss term, a speckle loss term, a frequency domain loss term and a gradient harmonic loss term, wherein the infrared loss term is a cross entropy loss term constructed based on the sample infrared classification result, the speckle loss term is a cross entropy loss term constructed based on the sample speckle classification result, and the frequency domain loss term is a cross entropy loss term constructed based on the sample frequency domain classification result; The gradient reconciliation loss term is constructed by the following steps: Calculating the gradient modulus of each sample image respectively, and determining the number of sample images whose gradient modulus is within a preset range; Determining the gradient density of the sample images of the batch according to the number of sample images whose gradient modulus is within a preset range and a preset interval length; The gradient harmonic loss term is constructed according to the gradient density, the sample fusion classification result, the label of the sample image and the number of sample images in the batch.

5. The method for detecting living body according to claim 4, characterized in that: The step of determining the gradient density of the sample images of the batch according to the number of sample images whose gradient modulus is within a preset range and a preset interval length is implemented by the following formula: Among them, g k is the gradient modulus of the kth sample graph in the batch, N is the total number of sample graphs in the batch, ε is a preset constant, δ ε (g k , g) is the distribution of gradient modulus length in the N sample graphs The number of sample graphs, l ε (g) is the preset interval length, GD(g k ) is the gradient density of the kth sample graph of the batch; The gradient reconciliation loss term is constructed according to the gradient density, the sample fusion classification result, the label of the sample graph and the number of sample graphs in the batch, and is implemented by the following formula: Among them, p k is the fusion classification result of the kth sample image in the batch, is the label of the kth sample image in the batch, Based on the p k and stated The constructed cross entropy loss function, L GHM is the gradient harmonization loss term.

6. The method for liveness detection according to claim 5, characterized in that: The constructed loss function is expressed by the following formula: L total =aL ir +bL speck +cL GHM +dL Freq Wherein, a is the preset first loss weight coefficient, b is the preset second loss weight coefficient, c is the preset third loss weight coefficient, d is the preset fourth loss weight coefficient, L ir is the infrared loss term, L speck is the speckle loss term, L GHM is the gradient reconciliation loss term, L Freq is the frequency domain loss term.

7. The living body detection method according to any one of claims 3 to 6, characterized in that: The first sub-model includes a first input interface, a second input interface, a backbone network with a group convolution group of 2 connected to the first input interface and the second input interface, and a first output head, a second output head and a third output head connected to the backbone network, the first input interface is used to receive the input infrared image, the second interface is used to receive the input speckle pattern, the backbone network is used to extract features of the infrared image and features of the speckle pattern to obtain an infrared feature map and a speckle feature map, the first output head is used to output an infrared classification result according to the infrared feature map, the second output head is used to output a speckle classification result according to the speckle feature map, and the third output head is used to output a fusion classification result according to the infrared feature map and the speckle feature map.

8. The living body detection method according to any one of claims 3 to 6, characterized in that: The second sub-model includes a third input interface, a first 3×3 convolutional layer connected to the third input interface, a second 3×3 convolutional layer connected to the first 3×3 convolutional layer, a 1×1 convolutional layer connected to the second 3×3 convolutional layer, a channel attention mechanism module connected to the 1×1 convolutional layer, a feature extraction module connected to the channel attention mechanism module, and a fourth output head connected to the feature extraction module and the 1×1 convolutional layer respectively. The third input interface is used to receive the input frequency domain map, the first 3×3 convolutional layer, the second 3×3 convolutional layer, the 1×1 convolutional layer, the channel attention mechanism module, and the feature extraction module are used to extract features of the frequency domain map to obtain a frequency domain feature map, and the fourth output head is used to output a frequency domain classification result according to the frequency domain feature map.

9. The living body detection method according to any one of claims 1 to 6, characterized in that: The step of performing frequency domain conversion on the speckle pattern to obtain a frequency domain graph comprises: compressing the speckle pattern according to a JPEG algorithm; The compressed speckle image is subjected to 2D-DCT transformation to calculate and obtain the frequency domain image.

10. The living body detection method according to any one of claims 1 to 6, characterized in that: After acquiring the infrared image and the speckle image of the target object and before performing frequency domain conversion on the speckle image to obtain the frequency domain image, the method further includes: Detecting key points of a human face in the acquired infrared image, and affine transforming the infrared image to a preset size according to the key points of the human face; Performing face detection on the acquired speckle image to obtain a face frame of the speckle image; Cropping the speckle pattern according to the face frame, and scaling the cropped speckle pattern to the preset size; The performing frequency domain conversion on the speckle pattern to obtain the frequency domain map specifically includes: performing frequency domain conversion on the speckle pattern after the scaling transformation to obtain the frequency domain map.

11. The living body detection method according to any one of claims 1 to 6, characterized in that: Determining the liveness score of the target object according to the fusion detection score and the frequency domain detection score is achieved by the following formula: S total =λS GHM +μS freq Among them, λ is the preset first scoring weight coefficient, μ is the preset second scoring weight coefficient, S GHM Score the fusion detection, S freq S is the frequency domain detection score, S total A liveness score is provided for the target object.

12. A living body detection system, characterized in that: It includes a camera module, a frequency domain calculation module, a storage module, an execution module, a scoring module and an output module; The camera module is used to obtain an infrared image and a speckle image of a target object; The frequency domain calculation module is used to perform frequency domain conversion on the speckle pattern to obtain a frequency domain graph; The storage module is used to store a pre-trained liveness detection model, where the liveness detection model includes a first sub-model and a second sub-model; The execution module is used to input the infrared image, the speckle image and the frequency domain image into the living body detection model, and obtain the fusion detection score and the frequency domain detection score of the target object output by the living body detection model, the first sub-model is used to output the fusion detection score based on the infrared image and the speckle image, and the second sub-model is used to output the frequency domain detection score based on the frequency domain image; The scoring module is used to determine the liveness score of the target object according to the fusion detection score and the frequency domain detection score; The output module is used to determine whether the liveness score is greater than a first preset threshold, and determine that the target object is a live object if the liveness score is greater than the first preset threshold.

13. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the living body detection method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the living body detection method according to any one of claims 1 to 11 is implemented.

Images