Liveness detection method, device and readable storage medium

Abstract

The present application discloses a liveness detection method, device and readable storage medium, the liveness detection method comprising: obtaining face data to be detected and shooting device motion information corresponding to the face data to be detected, and determining face region data and background region data corresponding to each time frame image in the face data to be detected, and then performing three-dimensional face reconstruction on the face region data to obtain three-dimensional face point cloud data, and then performing liveness detection on the face data to be detected based on the three-dimensional face point cloud data to obtain a first liveness detection result, and then calculating the optical flow information corresponding to the background region data, and performing liveness detection on the face data to be detected based on the shooting device motion information and the optical flow information to obtain a second liveness detection result, and then determining a target liveness detection result based on the first liveness detection result and the second liveness detection result. The present application solves the technical problem of low security of face recognition systems.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence in financial technology (Fintech), and in particular to a liveness detection method, device and readable storage medium. Background Art

[0002] With the continuous development of financial technology, especially Internet technology finance, more and more technologies (such as distributed, blockchain, artificial intelligence, etc.) are applied in the financial field, but the financial industry also puts forward higher requirements on technology, such as higher requirements on the distribution of corresponding to-do items in the financial industry.

[0003] With the continuous development of computer software and artificial intelligence, the application fields of artificial intelligence are becoming more and more extensive. For example, artificial intelligence is often used in face recognition. However, since face data is relatively easy to obtain, liveness detection is usually required when performing face recognition to ensure the accuracy of face recognition. At present, liveness detection is usually performed by analyzing the changes in face movements in face videos or image sequences (such as motion liveness), voice information and mouth movement changes (digital liveness) or three-dimensional structural information of the face, that is, liveness detection is performed based on the surface features of face image sequences or face videos. However, with the increasing development of video and image editing technology, if a malicious attacker attacks the face recognition system through a forged face image sequence or face video, the current face recognition system is increasingly difficult to resist the illegal attacks of malicious attackers, which in turn makes it difficult to ensure the security of the current face recognition system. Summary of the invention

[0004] The main purpose of this application is to provide a liveness detection method, device and readable storage medium, aiming to solve the technical problem of low security of face recognition systems in the prior art.

[0005] To achieve the above object, the present application provides a liveness detection method, which is applied to a liveness detection device, and the liveness detection method includes:

[0006] Acquire the face data to be detected and the motion information of the shooting device corresponding to the face data to be detected, and determine the face area data and the corresponding background area data corresponding to each time frame image in the face data to be detected;

[0007] Performing three-dimensional reconstruction of the face on the face region data to obtain three-dimensional point cloud data of the face;

[0008] Based on the three-dimensional face point cloud data, performing liveness detection on the face data to be detected to obtain a first liveness detection result;

[0009] Calculating optical flow information corresponding to the background area data, and performing liveness detection on the to-be-detected face data based on the shooting device motion information and the optical flow information to obtain a second liveness detection result;

[0010] A target liveness detection result is determined based on the first liveness detection result and the second liveness detection result.

[0011] The present application also provides a liveness detection device, which is a virtual device and is applied to a liveness detection device. The liveness detection device includes:

[0012] A first determination module is used to obtain face data to be detected and motion information of a camera corresponding to the face data to be detected, and determine face region data and background region data corresponding to each time frame image in the face data to be detected;

[0013] A three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the face area data to obtain three-dimensional point cloud data of the face;

[0014] A first liveness detection module, configured to perform liveness detection on the face data to be detected based on the three-dimensional face point cloud data, and obtain a first liveness detection result;

[0015] A second liveness detection module is used to calculate the optical flow information corresponding to the background area data, and perform liveness detection on the face data to be detected based on the motion information of the shooting device and the optical flow information to obtain a second liveness detection result;

[0016] The second determining module is used to determine a target liveness detection result based on the first liveness detection result and the second liveness detection result.

[0017] The present application also provides a liveness detection device, which is a physical device, and includes: a memory, a processor, and a program of the liveness detection method stored in the memory and executable on the processor. When the program of the liveness detection method is executed by the processor, the steps of the liveness detection method as described above can be implemented.

[0018] The present application also provides a readable storage medium, on which is stored a program for implementing the liveness detection method. When the program for the liveness detection method is executed by a processor, the steps of the liveness detection method as described above are implemented.

[0019] The present application provides a liveness detection method, device and readable storage medium. Compared with the technical means of performing liveness detection based on the surface features of a face image sequence or a face video adopted in the prior art, the present application determines the face area data and background area data corresponding to each time frame image in the face data to be detected after obtaining the face data to be detected and the motion information of the shooting device corresponding to the face data to be detected, and then performs three-dimensional face reconstruction on the face area data to obtain three-dimensional face point cloud data. Since it is difficult to extract regular three-dimensional face features from a forged face area image sequence, the purpose of accurately identifying the forged face image sequence can be achieved, and malicious attackers can be prevented from attacking the face recognition system by forging a face image sequence. Based on the three-dimensional face point cloud data, the face data to be detected can be subjected to liveness detection, and a first liveness detection result for identifying the forged face image sequence can be obtained, and then the optical flow information corresponding to the background area data is calculated. It is noted that if it is a real video, the optical flow information should match the motion information of the shooting device. If it is a forged face video, the optical flow information does not match the motion information of the shooting device. Then, based on the motion information of the shooting device and the optical flow information, the face data to be detected is subjected to liveness detection, thereby achieving the purpose of accurately identifying the forged face video, preventing malicious attackers from attacking the face recognition system through forged face videos, and obtaining a second liveness detection result for identifying the forged face video. Furthermore, by combining the first liveness detection result and the second liveness detection result, it is possible to determine whether the face data to be detected corresponds to a live face, obtain the target liveness detection result, and then determine whether the face recognition system is attacked. This overcomes the technical defect in the prior art that liveness detection is based on the surface features of face image sequences or face videos, and it is difficult to defend against malicious attackers attacking the face recognition system through forged face image sequences or face videos, thereby improving the security of the face recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 This is a flow chart of the first embodiment of the liveness detection method of the present application;

[0023] Figure 2 A schematic diagram of a user shooting a video of a face to be detected in the liveness detection method of this application;

[0024] Figure 3 A schematic diagram of comparing the semantic image and the time frame image in the living body detection method of the present application;

[0025] Figure 4 This is a flow chart of the second embodiment of the liveness detection method of the present application;

[0026] Figure 5 A schematic diagram of the second-order difference matrix described in the living body detection method of this application;

[0027] Figure 6 A schematic diagram of the edge difference matrix in the living body detection method of the present application;

[0028] Figure 7 A schematic diagram of the square difference matrix described in the living body detection method of the present application;

[0029] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The present application embodiment provides a liveness detection method. In the first embodiment of the liveness detection method of the present application, refer to Figure 1 , the living body detection method comprises:

[0033] Step S10, obtaining face data to be detected and camera motion information corresponding to the face data to be detected, and determining face region data and background region data corresponding to each time frame image in the face data to be detected;

[0034] In this embodiment, it should be noted that the face data to be detected is a face video to be detected, and the liveness detection method is applied to a face recognition system to perform liveness detection on the face video to be detected to determine whether the face video to be detected corresponds to a live face, wherein, if the face video to be detected corresponds to a live face, it proves that the face video to be detected is shot by a live face, and if the face video to be detected does not correspond to a live face, it proves that the face video to be detected is a forged face video or a face image sequence.

[0035] In addition, it should be noted that the face video to be detected is a face video shot in motion, and the shooting device motion information is the motion trajectory data of the preset shooting device, which is used to represent the motion trajectory of the preset shooting device during the mobile shooting process, such as Figure 2 The figure shows a schematic diagram of a user shooting a video of a face to be detected, wherein the dotted arrow indicates that the moving direction during mobile shooting is horizontally from left to right.

[0036] Acquire the face data to be detected and the motion information of the shooting device corresponding to the face data to be detected, and determine the face area data and the corresponding background area data corresponding to each time frame image in the face data to be detected. Specifically, shoot the face video to be detected by moving a preset shooting device, and synchronously collect the gyroscope data corresponding to the preset shooting device, wherein the gyroscope data is the acceleration data of the movement of the preset shooting device, and then integrate the gyroscope data to obtain the motion trajectory data of the preset shooting device, and use the motion trajectory data of the preset shooting device as the motion information of the shooting device, and then based on the preset image segmentation model, respectively segment each time frame image corresponding to the face video to be detected into the face area and the background area to obtain the face area data and the background area data.

[0037] The face region data at least includes a face region corresponding to the time frame image, and the background region data at least includes a background region corresponding to the time frame image.

[0038] The step of determining the face region data and the background region data corresponding to each time frame image in the face data to be detected comprises:

[0039] Step S11, based on a preset image segmentation model, semantic segmentation is performed on each of the time frame images to obtain the face area and the background area corresponding to each of the time frame images.

[0040] In this embodiment, it should be noted that the preset image segmentation model is a pre-trained neural network model for semantic segmentation, and the time frame image is a constituent image of a time frame in the face video to be detected.

[0041] Based on a preset image segmentation model, semantic segmentation is performed on each of the time frame images to obtain the face area and the background area corresponding to each of the time frame images. Specifically, the following steps are performed on each of the time frame images:

[0042] Output the time frame image to the preset image segmentation model, downsample the time frame image a preset number of times, wherein the downsampling method includes maximum pooling, average pooling, random pooling and summed area pooling, etc., to obtain a downsampled image, wherein the downsampled image includes high-level semantic information of the time frame image, wherein the high-level semantic information includes abstract feature information of the image subject, such as subject position information, subject contour information and other information, further, upsample the downsampled image, wherein the upsampling method includes bilinear interpolation, deconvolution, depooling and other methods, so as to gradually restore the spatial information of the downsampled image, that is, gradually restore the position information of the pixel points in the downsampled image, so that the upsampled image is consistent with the resolution of the input time frame image, and obtain a semantic image, wherein the semantic image includes the face area and the background area, such as Figure 3 The figure shows a comparison diagram of the semantic image and the time frame image, wherein the image on the left is the time frame image, the image on the right is the semantic image, area 1 is the face area, and area 2 is the background area.

[0043] Step S20, performing three-dimensional reconstruction of the face region data to obtain three-dimensional point cloud data of the face;

[0044] In this embodiment, it should be noted that the three-dimensional point cloud data of the face includes three-dimensional sparse point cloud data of the face and three-dimensional dense point cloud data of the face, wherein the three-dimensional sparse point cloud data of the face is obtained by three-dimensional reconstruction based on the SfM (Structure from Motion) reconstruction technology, and the three-dimensional dense point cloud of the face is obtained by three-dimensional reconstruction based on the MVS (Multi-View Stereo reconstruction) reconstruction technology.

[0045] In addition, it should be noted that each of the time frame images is a sequence of time frame images along the time axis, and each of the time frame images is a frame of the face video to be detected. Since the face video to be detected is a face video shot while moving, and each of the time frame images is a face image shot by a preset shooting device at different shooting angles, the two-dimensional face feature points of each time frame image can be mapped to the same three-dimensional face feature point in three-dimensional space, and the three-dimensional point cloud data of the face reconstructed based on each time image should conform to the three-dimensional contour features of the face in the face video to be detected. As for the forged image sequence, since each face image in the forged image sequence is a face image collected by a malicious attacker through various channels, there is no correlation between the face images in the forged image sequence, and the three-dimensional point cloud data of the face obtained by three-dimensional reconstruction based on the forged image sequence usually does not conform to the three-dimensional contour features of the face.

[0046] Step S30, performing liveness detection on the face data to be detected based on the three-dimensional face point cloud data to obtain a first liveness detection result;

[0047] In this embodiment, based on the three-dimensional face point cloud data, liveness detection is performed on the face data to be detected to obtain a first liveness detection result. Specifically, the three-dimensional face point cloud data is input into a preset point cloud data classification model, the three-dimensional face point cloud data is classified to obtain a point cloud data classification label, and then based on the point cloud data classification label, it is determined whether the face video to be detected corresponds to a live face to obtain a first liveness detection result.

[0048] The step of performing liveness detection on the face data to be detected based on the face three-dimensional point cloud data to obtain a first liveness detection result includes:

[0049] Step S31, classifying the three-dimensional point cloud data of the face based on a preset point cloud data classification model to obtain a point cloud data classification result;

[0050] In this embodiment, based on a preset point cloud data classification model, the three-dimensional point cloud data of the face is classified to obtain a point cloud data classification result. Specifically, the point cloud data feature representation matrix corresponding to the three-dimensional point cloud data of the face is input into the preset point cloud data classification model, and features are extracted from the point cloud data feature representation matrix to obtain a point cloud data category feature representation matrix corresponding to the point cloud data feature representation matrix, wherein the point cloud data feature representation matrix is ​​an encoding matrix corresponding to the three-dimensional point cloud data of the face, and is used to represent the three-dimensional point cloud data of the face by encoding, and then the point cloud data category feature representation matrix is ​​fully connected to obtain a point cloud data classification label vector, and then the point cloud data classification label vector is used as the point cloud data classification result.

[0051] Step S32: Based on the classification result of the point cloud data, liveness detection is performed on the face data to be detected to obtain the first liveness detection result.

[0052] In this embodiment, based on the classification result of the point cloud data, liveness detection is performed on the face data to be detected to obtain the first liveness detection result. Specifically, the classification label code and the first classification probability in the classification label vector of the point cloud data are extracted, wherein the classification label code is the identifier of the three-dimensional point cloud data of the face, and the first classification probability is the probability that the three-dimensional point cloud data of the face belongs to the data category corresponding to the classification label vector. Then, based on the classification label code and the first classification probability, liveness detection is performed on the face video to be detected to generate a first liveness score corresponding to the face video to be detected, and the first liveness score is used as the first liveness detection result, wherein the first liveness classification is a score used to represent the probability that the face video to be detected corresponds to the liveness of the face. For example, assuming that the classification label code is 1, indicating that the three-dimensional point cloud data of the face belongs to point cloud data of the forged video data type, the first classification probability is 40%, and the first liveness score is 60 points.

[0053] Step S40, calculating the optical flow information corresponding to the background area data, and performing liveness detection on the face data to be detected based on the shooting device motion information and the optical flow information to obtain a second liveness detection result;

[0054] In the present embodiment, it should be noted that the optical flow information is the position movement information of the pixel points in each of the time frame images, and the optical flow information includes at least one optical flow value, wherein the optical flow value is the position information of a specific pixel value moving from one time frame image to another time frame image in the x direction and the y direction respectively, and the optical flow information includes sparse optical flow and dense optical flow, wherein the sparse optical flow includes the optical flow values ​​of a preset number of pixel points, and the dense optical flow includes the optical flow value of each pixel point in the time frame image, wherein the method for calculating the sparse optical flow includes the Lucas-Kanade algorithm, etc., and the method for calculating the dense optical flow includes the Farneback algorithm, etc.

[0055] In addition, it should be noted that since the face video to be detected is a face video shot in motion, the preset shooting device will form a motion trajectory, and in the face video to be detected, the background area pixels in each time frame image can be considered to be formed by the movement of the background area pixels of the time frame image of the previous time frame, and the movement trajectory of the background area pixels should be highly similar to the movement trajectory of the preset shooting device, that is, the shooting device motion information and the optical flow information should be highly similar, and for the forged face video, due to the editing of the video, the movement trajectory of the pixels of each time frame image in the forged face video will deviate from the movement trajectory of the shooting device, that is, the optical flow information corresponding to the forged face video is not highly similar to the device motion information of the shooting device.

[0056] Calculate the optical flow information corresponding to the background area data, and perform liveness detection on the face data to be detected based on the motion information of the shooting device and the optical flow information to obtain a second liveness detection result. Specifically, based on a preset optical flow information calculation method, calculate the optical flow information corresponding to the background area of ​​each time frame image, wherein the preset optical flow information calculation method includes a Lucas-Kanade algorithm and a Farneback algorithm, etc., and then compare the optical flow information with the motion information of the shooting device to verify whether the motion trajectory information of the pixel point corresponding to the optical flow information is consistent with the motion trajectory information of the preset shooting device, and obtain a comparison result. Then, based on the comparison result, perform liveness detection on the face video to be detected to obtain a second liveness detection result.

[0057] The step of performing liveness detection on the face data to be detected based on the motion information of the shooting device and the optical flow information to obtain a second liveness detection result includes:

[0058] Step S41, calculating the information matching degree between the motion information of the shooting device and the optical flow information;

[0059] In this embodiment, the information matching degree between the motion information of the shooting device and the optical flow information is calculated. Specifically, a first motion trajectory corresponding to the motion information of the shooting device and a second motion trajectory corresponding to the optical flow information are generated, and then the trajectory similarity between the first motion trajectory and the second motion trajectory is calculated, and the trajectory similarity is used as the information matching degree.

[0060] Step S42: compare the information matching degree with a preset matching degree threshold to perform liveness detection on the face data to be detected, and obtain the second liveness detection result.

[0061] In this embodiment, the information matching degree is compared with a preset matching degree threshold to perform liveness detection on the face data to be detected to obtain the second liveness detection result. Specifically, the information matching degree is compared with the preset matching degree threshold. If the information matching degree is greater than or equal to the preset matching degree threshold, it is determined that the face video to be detected corresponds to a live face, and a second liveness score corresponding to the information matching degree is generated, and the second liveness score is used as the second liveness detection result. If the information matching degree is less than the preset matching degree threshold, it is determined that the face video to be detected does not correspond to a live face, that is, the second liveness score is set to 0, and the second liveness score is used as the second liveness detection result.

[0062] Step S50: determining a target liveness detection result based on the first liveness detection result and the second liveness detection result.

[0063] In this embodiment, based on the first liveness detection result and the second liveness detection result, a target liveness detection result is determined. Specifically, a weighted average of the first liveness score and the second liveness score is performed to obtain a target liveness score, and based on the target liveness score, it is determined whether the face video to be detected corresponds to a live face to obtain the target liveness detection result.

[0064] The step of determining a target liveness detection result based on the first liveness detection result and the second liveness detection result includes:

[0065] Step S51, performing image recognition on each of the time frame images based on a preset image recognition model to obtain an image recognition result corresponding to each of the time frame images;

[0066] In this embodiment, it should be noted that since the forged face video or face image sequence requires editing of the image or video, the image of each frame in the forged face video can be obtained as the first type of training data and assigned a first type of label, and the image of each frame in the non-forged face video can be obtained as the second type of training data and assigned a second type of label. Furthermore, based on the first type of label, the first type of training data, the second type of label and the second type of training data, a preset image recognition model for identifying each frame image corresponding to the forged face video can be trained.

[0067] Based on a preset image recognition model, image recognition is performed on each of the time frame images to obtain an image recognition result corresponding to each of the time frame images. Specifically, each of the time frame images is input into a preset image recognition model, and each of the time frame images is classified to determine whether the time frame image is an image in an image sequence corresponding to a forged face video, and an image classification result of each of the time frame images is obtained, and each of the image classification results is used as the image recognition result.

[0068] Step S52, performing liveness detection on the face data to be detected based on the image recognition result to obtain a third liveness detection result;

[0069] In this embodiment, based on the image recognition result, liveness detection is performed on the face data to be detected to obtain a third liveness detection result. Specifically, based on each of the image classification results, the image ratio of the time frame images of the image sequence corresponding to the face video that is not a forged face to all the time frame images is calculated, and a third liveness score corresponding to the image ratio is generated, and the third liveness score is used as the third liveness result.

[0070] Step S53: Generate a second target liveness detection result based on the first liveness detection result, the second liveness detection result and the third liveness detection result.

[0071] In this embodiment, based on the first liveness detection result, the second liveness detection result and the third liveness detection result, a second target liveness detection result is generated. Specifically, a weighted average is performed on the first liveness score, the second liveness score and the third liveness score to obtain a second target liveness score, and based on the second target liveness score, it is determined whether the face video to be detected corresponds to a live face to obtain the target liveness detection result.

[0072] The present embodiment provides a liveness detection method. Compared with the technical means used in the prior art for liveness detection based on the surface features of a face image sequence or a face video, the present embodiment determines the face area data and background area data corresponding to each time frame image in the face data to be detected after obtaining the face data to be detected and the motion information of the shooting device corresponding to the face data to be detected, and then performs three-dimensional face reconstruction on the face area data to obtain three-dimensional face point cloud data. Since it is difficult to extract regular three-dimensional face features from a forged face area image sequence, the purpose of accurately identifying the forged face image sequence can be achieved, and malicious attackers can be prevented from attacking the face recognition system by forging a face image sequence. Based on the three-dimensional face point cloud data, the face data to be detected can be liveness detected to obtain a first liveness detection result for identifying the forged face image sequence, and then the optical flow information corresponding to the background area data is calculated. It should be noted that If it is a real video, the optical flow information should match the motion information of the shooting device. If it is a forged face video, the optical flow information does not match the motion information of the shooting device. Then, based on the motion information of the shooting device and the optical flow information, the face data to be detected is subjected to liveness detection, so as to achieve the purpose of accurately identifying the forged face video, and prevent malicious attackers from attacking the face recognition system through forged face videos, and obtain a second liveness detection result for identifying the forged face video. Furthermore, by combining the first liveness detection result and the second liveness detection result, it is possible to determine whether the face data to be detected corresponds to a live face, obtain the target liveness detection result, and then determine whether the face recognition system is attacked, which overcomes the technical defect of the prior art that liveness detection is based on the surface features of face image sequences or face videos, and it is difficult to defend against the attack of malicious attackers on the face recognition system through forged face image sequences or face videos, thereby improving the security of the face recognition system.

[0073] Further, refer to Figure 4Based on the first embodiment of the present application, in another embodiment of the present application, the step of determining the target liveness detection result based on the first liveness detection result and the second liveness detection result includes:

[0074] Step A10, obtaining frame difference noise characteristic data corresponding to each of the time frame images;

[0075] In this embodiment, it should be noted that the frame difference noise feature data is feature data of the noise information in the frame difference data, which is used to distinguish between artificially edited videos and naturally shot videos, wherein the artificially edited videos are forged face videos, and the naturally shot videos are non-forged face videos. It should be noted that since the artificially edited videos need to be obtained by editing, the original frame difference noise features will be changed during editing, and the frame difference noise features of the artificially edited videos will be different from the frame difference noise features of the naturally shot videos. For example, the frame difference noise corresponding to the naturally shot videos should conform to the Gaussian distribution or the Poisson distribution, while when the video is edited, the distribution of the frame difference noise will be destroyed, and the frame difference noise corresponding to the artificially edited videos will not conform to the Gaussian distribution or the Poisson distribution.

[0076] Frame difference noise feature data corresponding to each of the time frame images is obtained. Specifically, the frame difference between the time frame images corresponding to adjacent time frames is calculated to obtain frame difference data, and then feature extraction is performed on the frame difference data to obtain frame difference noise feature data.

[0077] The frame difference noise feature data includes spatial domain frame difference noise feature data and frequency domain frame difference noise feature data.

[0078] The step of obtaining frame difference noise characteristic data corresponding to each of the time frame images comprises:

[0079] Step A11, calculating the adjacent frame differences between the time frame images to obtain each frame difference map;

[0080] In this embodiment, adjacent frame differences between the time frame images are calculated to obtain each frame difference map. Specifically, based on the time sequence of the time frames corresponding to the time frame images, the frame difference between the time frame images corresponding to each two adjacent time frames is calculated to obtain each frame difference map. For example, assuming that each of the time frame images is (f1, f2, ..., f N ), then the frame difference d N-1 =f N -f N-1 , and then the frame difference map is (d1, d2, ..., d N ).

[0081] Step A12, filtering each of the frame difference images respectively to amplify the noise signal in the frame difference data to obtain the spatial domain frame difference noise feature data;

[0082] In the present embodiment, it should be noted that if there is no noise signal in each time frame image, most of the pixel values ​​in the frame difference image matrix corresponding to the frame difference image should be 0, because there must be noise signals in naturally shot videos, and the noise signals should be distributed everywhere in the frame difference image. However, in artificially edited images, due to manual editing of some areas of the image, the noise signal in the edited area will disappear, and then in the frame difference image it will be manifested as the noise signal is not distributed in some areas of the frame difference image, and in the frame difference image matrix it is manifested as the pixel values ​​in some areas are 0.

[0083] The frame difference images are filtered respectively to amplify the noise signal in the frame difference data and obtain the spatial frame difference noise feature data. Specifically, based on a preset filter kernel, a convolution process is performed on the frame difference image matrix corresponding to each frame difference image to amplify the pixel value in the frame difference image, thereby amplifying the noise signal, and obtaining a convolution process matrix corresponding to each frame difference image matrix, and then using each convolution process matrix as the spatial frame difference noise feature data, wherein the preset filter kernel is a convolution kernel for convolution processing, wherein the preset filter kernel includes a second-order difference matrix, an edge difference matrix, a square difference matrix, etc., wherein, as Figure 5 The schematic diagram of the second-order difference matrix is ​​shown in FIG. Figure 6 The schematic diagram of the edge difference matrix is ​​shown as Figure 7 Shown is a schematic diagram of the square difference matrix.

[0084] Step A13, performing Fourier transform on each of the frame difference images respectively, so as to transform the frame difference images from the spatial domain to the frequency domain, and obtain the frequency domain frame difference noise characteristic data.

[0085] In this embodiment, it should be noted that the frequency domain frame difference noise feature data includes a frame difference spectrum diagram, which is an image in the spatial domain. The frame difference diagram includes spatial information of the image, wherein the spatial information reflects features such as the position, shape, and size of objects in the image. The frame difference spectrum diagram is an image corresponding to the frame difference diagram in the frequency domain, and is used to represent the image frequency of the frame difference diagram, wherein the image frequency is the degree of change in pixel values ​​of the image, that is, the gradient of the image.

[0086] In addition, it should be noted that the main component of the image is the low-frequency information, which forms the basic grayscale level of the image and has a smaller determining effect on the image structure. The medium-frequency information determines the basic structure of the image and forms the main edge structure of the image. The high-frequency information forms the edges and details of the image, and the noise signal in the image is usually medium and high-frequency information.

[0087] In addition, it should be noted that the noise signal and low-frequency signal in the frame difference spectrum graph corresponding to the naturally shot video should be evenly distributed in each area of ​​the frame difference spectrum graph, while for the frame difference spectrum graph corresponding to the artificially edited video, the noise signal is eliminated due to artificial editing, which is manifested in the frame difference spectrum graph as the disappearance of high-frequency information in some areas.

[0088] Fourier transform is performed on each of the frame difference images respectively, so as to transform the frame difference images from the spatial domain to the frequency domain, and obtain the frequency domain frame difference noise characteristic data. Specifically, Fourier transform is performed on each of the frame difference images respectively, so as to transform each of the frame difference images from the spatial domain to the frequency domain, and obtain the frame difference spectrum image corresponding to each of the frame difference images, and then each of the frame difference spectrum images is used as the frequency domain frame difference noise characteristic data.

[0089] Step A20, performing liveness detection on the face data to be detected based on the frame difference noise feature data to obtain a fourth liveness detection result;

[0090] In this embodiment, based on the frame difference noise feature data, liveness detection is performed on the face data to be detected to obtain a fourth liveness detection result. Specifically, based on the frame difference noise feature data, liveness detection is performed on each of the frame difference images to obtain a liveness detection sub-result corresponding to each of the frame difference images, and then based on each of the liveness detection sub-results, a fourth liveness score is calculated, and the fourth liveness score is used as the fourth liveness detection result. In an implementable scheme, the number of liveness judgments of the liveness detection sub-results judged to be live is determined, and then the ratio of the number of liveness judgments to the number of the frame difference images is calculated to obtain the fourth liveness score.

[0091] In addition, it should be noted that since the embodiment of the present application performs liveness detection based on frame difference noise feature data, it can distinguish between artificially edited videos and naturally shot videos. Among them, the liveness detection sub-results corresponding to each frame difference noise feature data corresponding to the artificially edited video should be judged as non-live with a high probability. Therefore, even if a malicious attacker uses forged video data to attack the face recognition system, the forged video data can be identified and the faces in the forged video data can be judged as non-live, which improves the accuracy of liveness detection and thus improves the accuracy of face recognition.

[0092] The step of performing liveness detection on the face data to be detected based on the frame difference noise feature data to obtain a fourth liveness detection result comprises:

[0093] Step A21, inputting the frame difference noise feature data into a preset frame difference noise feature classification model, classifying each frame difference noise feature in the frame difference noise feature data, and obtaining a frame difference noise feature classification result;

[0094] In this embodiment, it should be noted that the frame difference noise feature data includes at least a frame difference noise feature corresponding to a frame difference image, the frame difference noise feature includes the convolution processing matrix and the frame difference spectrum image, and the preset frame difference noise feature classification model is a trained neural network model for classifying the frame difference noise feature, wherein the type of the frame difference noise feature includes a living body frame difference noise feature type and a non-living body frame difference noise feature type.

[0095] The frame difference noise feature data is input into a preset frame difference noise feature classification model, and each frame difference noise feature in the frame difference noise feature data is classified to obtain a frame difference noise feature classification result. Specifically, the frame difference noise feature representation matrix corresponding to each frame difference image is input into the preset frame difference noise feature classification model, and feature extraction is performed on each frame difference noise feature representation matrix to obtain a feature extraction matrix corresponding to each frame difference noise feature representation matrix, wherein the frame difference noise feature representation matrix is ​​an encoding matrix of the frame difference noise feature, and then each feature extraction matrix is ​​fully connected to obtain a classification label vector corresponding to each frame difference noise feature, and then based on the classification label in each classification label vector, it is determined whether the type of each frame difference noise feature is a living body frame difference noise feature type or a non-living body frame difference noise feature type to obtain a frame difference noise feature classification result, wherein the living body frame difference noise feature type indicates that the frame difference image corresponding to the frame difference noise feature corresponds to a living face, and the non-living body frame difference noise feature type indicates that the frame difference image corresponding to the frame difference noise feature corresponds to a non-living face.

[0096] In addition, it should be noted that when training the preset frame difference noise feature classification model, naturally shot face video data and artificially edited face video data can be obtained respectively, and the frame difference noise features corresponding to the naturally shot face video data are used as the first type of training data, and the first type of training data is assigned a first classification label corresponding to the living frame difference feature type, and the frame difference noise features corresponding to the artificially edited face video data are used as the second type of training data, and the second type of training data is assigned a second classification label corresponding to the non-living frame difference feature type, and then based on the first type of training data, the first classification label, the second type of training data and the second classification label, the preset frame difference noise feature classification model can be trained.

[0097] Step A22: Based on the frame difference noise feature classification result, liveness detection is performed on the face data to be detected to obtain the fourth liveness detection result.

[0098] In this embodiment, based on the frame difference noise feature classification result, liveness detection is performed on the face data to be detected to obtain the fourth liveness detection result. Specifically, based on the frame difference noise feature classification result, the number of liveness frame difference features of the frame difference noise features belonging to the liveness frame difference feature type is counted, and based on the number of liveness frame difference features and the number of frame difference images, a fourth liveness score is calculated, and the fourth liveness score is used as the fourth liveness detection result.

[0099] Step A30: Generate a third target liveness detection result based on the first liveness detection result, the second liveness detection result and the fourth liveness detection result.

[0100] In this embodiment, based on the first liveness detection result, the second liveness detection result and the fourth liveness detection result, a third target liveness detection result is generated. Specifically, a weighted average is performed on the first liveness score, the second liveness score and the fourth liveness score to obtain a third target liveness score, and based on the third target liveness score, it is determined whether the face video to be detected corresponds to a live face to obtain the third target liveness detection result.

[0101] The present embodiment provides a liveness detection method. Compared with the technical means of performing liveness detection based on the surface features of a face image sequence or a face video adopted in the prior art, the present embodiment obtains the frame difference noise feature data corresponding to each of the time frame images, wherein it should be noted that the frame difference noise feature data of the artificially edited forged video data is different from the frame difference noise feature data of the naturally shot video data, and then performs liveness detection on the face data to be detected based on the frame difference noise feature data, so as to identify whether the face video to be detected corresponding to the frame difference noise feature data is forged video data, and then obtain a more accurate fourth liveness detection result. Furthermore, based on the first liveness detection result, the second liveness detection result and the fourth liveness detection result, it can be comprehensively judged whether the face data to be detected corresponds to a live face, and a third target liveness detection result is generated, and then it can be determined whether the face recognition system is attacked, thereby overcoming the technical defect of performing liveness detection based on the surface features of a face image sequence or a face video in the prior art, which is difficult to defend against the attack of a malicious attacker on the face recognition system by forging a face image sequence or a face video, and thus improving the security of the face recognition system.

[0102] Reference Figure 8 , Figure 8 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0103] like Figure 8 As shown, the liveness detection device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0104] Optionally, the liveness detection device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0105] Those skilled in the art will understand that Figure 8 The structure of the liveness detection device shown in the figure does not constitute a limitation on the liveness detection device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0106] like Figure 8 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module and a liveness detection program. The operating system is a program that manages and controls the hardware and software resources of the liveness detection device, and supports the operation of the liveness detection program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the liveness detection system.

[0107] exist Figure 8 In the liveness detection device shown, the processor 1001 is used to execute the liveness detection program stored in the memory 1005 to implement the steps of any of the liveness detection methods described above.

[0108] The specific implementation of the liveness detection device of the present application is basically the same as the above-mentioned embodiments of the liveness detection method, and will not be repeated here.

[0109] The embodiment of the present application further provides a liveness detection device, which is applied to a liveness detection device, and includes:

[0110] A first determination module is used to obtain face data to be detected and motion information of a camera corresponding to the face data to be detected, and determine face region data and background region data corresponding to each time frame image in the face data to be detected;

[0111] A three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the face area data to obtain three-dimensional point cloud data of the face;

[0112] A first liveness detection module, configured to perform liveness detection on the face data to be detected based on the three-dimensional face point cloud data, and obtain a first liveness detection result;

[0113] A second liveness detection module is used to calculate the optical flow information corresponding to the background area data, and perform liveness detection on the face data to be detected based on the motion information of the shooting device and the optical flow information to obtain a second liveness detection result;

[0114] The second determining module is used to determine a target liveness detection result based on the first liveness detection result and the second liveness detection result.

[0115] Optionally, the first living body detection module includes:

[0116] A classification unit, used to classify the three-dimensional point cloud data of the face based on a preset point cloud data classification model to obtain a point cloud data classification result;

[0117] The first liveness detection unit is used to perform liveness detection on the face data to be detected based on the point cloud data classification result to obtain the first liveness detection result.

[0118] Optionally, the second living body detection module includes:

[0119] A calculation unit, used to calculate the information matching degree between the motion information of the shooting device and the optical flow information;

[0120] The second liveness detection unit is used to compare the information matching degree with a preset matching degree threshold to perform liveness detection on the face data to be detected and obtain the second liveness detection result.

[0121] Optionally, the first determining module includes:

[0122] The semantic segmentation unit is used to perform semantic segmentation on each of the time frame images based on a preset image segmentation model to obtain the face area and the background area corresponding to each of the time frame images.

[0123] Optionally, the second determining module includes:

[0124] An image recognition unit, configured to perform image recognition on each of the time frame images based on a preset image recognition model, and obtain an image recognition result corresponding to each of the time frame images;

[0125] A third liveness detection unit, configured to perform liveness detection on the face data to be detected based on the image recognition result, to obtain a third liveness detection result;

[0126] The first determining unit is configured to generate a second target liveness detection result based on the first liveness detection result, the second liveness detection result, and the third liveness detection result.

[0127] Optionally, the second determining module further includes:

[0128] An acquisition unit, used for acquiring frame difference noise characteristic data corresponding to each of the time frame images;

[0129] a fourth liveness detection unit, configured to perform liveness detection on the to-be-detected face data based on the frame difference noise feature data, to obtain a fourth liveness detection result;

[0130] The second determining unit is configured to generate a third target liveness detection result based on the first liveness detection result, the second liveness detection result, and the fourth liveness detection result.

[0131] Optionally, the acquiring unit includes:

[0132] A calculation subunit, used for calculating the adjacent frame differences between the time frame images to obtain each frame difference map;

[0133] A filtering subunit, used for filtering each of the frame difference images respectively to amplify the noise signal in the frame difference data and obtain the spatial domain frame difference noise feature data;

[0134] The Fourier transform subunit is used to perform Fourier transform on each of the frame difference images respectively, so as to transform the frame difference images from the spatial domain to the frequency domain, and obtain the frequency domain frame difference noise characteristic data.

[0135] Optionally, the fourth living body detection unit includes:

[0136] A classification subunit, configured to input the frame difference noise feature data into a preset frame difference noise feature classification model, classify each frame difference noise feature in the frame difference noise feature data, and obtain a frame difference noise feature classification result;

[0137] The liveness detection subunit is used to perform liveness detection on the face data to be detected based on the frame difference noise feature classification result to obtain the fourth liveness detection result.

[0138] The specific implementation of the liveness detection device of the present application is basically the same as the above-mentioned embodiments of the liveness detection method, and will not be repeated here.

[0139] An embodiment of the present application provides a readable storage medium, and the readable storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned living body detection methods.

[0140] The specific implementation of the readable storage medium of the present application is basically the same as the above-mentioned embodiments of the living body detection method, and will not be repeated here.

[0141] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A method for detecting a living body, characterized in that: The living body detection method comprises: Acquire the face data to be detected and the motion information of the shooting device corresponding to the face data to be detected, and determine the face area data and the corresponding background area data corresponding to each time frame image in the face data to be detected; the shooting device motion information is the motion trajectory data of the preset shooting device, which is used to represent the motion trajectory of the preset shooting device during the mobile shooting process; Performing three-dimensional reconstruction of the face on the face region data to obtain three-dimensional point cloud data of the face; Based on the three-dimensional face point cloud data, performing liveness detection on the face data to be detected to obtain a first liveness detection result; Calculating optical flow information corresponding to the background area data, and performing liveness detection on the to-be-detected face data based on the shooting device motion information and the optical flow information to obtain a second liveness detection result; Determine a target liveness detection result based on the first liveness detection result and the second liveness detection result; The step of performing liveness detection on the face data to be detected based on the motion information of the shooting device and the optical flow information to obtain a second liveness detection result comprises: Calculating the information matching degree between the motion information of the shooting device and the optical flow information; The information matching degree is compared with a preset matching degree threshold to perform liveness detection on the face data to be detected, and obtain the second liveness detection result.

2. The method for detecting a living body according to claim 1, wherein: The step of performing liveness detection on the face data to be detected based on the face three-dimensional point cloud data to obtain a first liveness detection result comprises: Based on a preset point cloud data classification model, classify the three-dimensional point cloud data of the face to obtain a point cloud data classification result; Based on the point cloud data classification result, liveness detection is performed on the face data to be detected to obtain the first liveness detection result.

3. The method for detecting a living body according to claim 1, wherein: The face region data at least includes a face region corresponding to the time frame image, and the background region data at least includes a background region corresponding to the time frame image. The step of determining the face region data and the background region data corresponding to each time frame image in the face data to be detected comprises: Based on a preset image segmentation model, semantic segmentation is performed on each of the time frame images to obtain the face area and the background area corresponding to each of the time frame images.

4. The method for detecting a living body according to claim 1, wherein: The step of determining a target liveness detection result based on the first liveness detection result and the second liveness detection result comprises: Based on a preset image recognition model, performing image recognition on each of the time frame images respectively to obtain an image recognition result corresponding to each of the time frame images; Based on the image recognition result, performing liveness detection on the face data to be detected to obtain a third liveness detection result; Generate a second target liveness detection result based on the first liveness detection result, the second liveness detection result, and the third liveness detection result; The step of generating a second target liveness detection result based on the first liveness detection result, the second liveness detection result and the third liveness detection result comprises: A weighted average is performed on the first liveness score, the second liveness score, and the third liveness score to obtain a second target liveness score, and based on the second target liveness score, it is determined whether the face video to be detected corresponds to a live face, and the second target liveness detection result is obtained.

5. The method for detecting a living body according to claim 1, wherein: The step of determining a target liveness detection result based on the first liveness detection result and the second liveness detection result comprises: Acquire frame difference noise characteristic data corresponding to each of the time frame images; Based on the frame difference noise feature data, performing liveness detection on the to-be-detected face data to obtain a fourth liveness detection result; Generate a third target liveness detection result based on the first liveness detection result, the second liveness detection result, and the fourth liveness detection result; The step of generating a third target liveness detection result based on the first liveness detection result, the second liveness detection result and the fourth liveness detection result comprises: A weighted average is performed on the first liveness score, the second liveness score, and the fourth liveness score to obtain a third target liveness score, and based on the third target liveness score, it is determined whether the face video to be detected corresponds to a live face, to obtain the third target liveness detection result.

6. The method for detecting a living body according to claim 5, characterized in that: The frame difference noise feature data includes spatial domain frame difference noise feature data and frequency domain frame difference noise feature data. The step of obtaining frame difference noise characteristic data corresponding to each of the time frame images comprises: Calculating adjacent frame differences between the time frame images to obtain frame difference maps; Performing filtering processing on each of the frame difference images respectively to amplify the noise signal in the frame difference data to obtain the spatial domain frame difference noise feature data; Fourier transform is performed on each of the frame difference images respectively to transform the frame difference images from the spatial domain to the frequency domain to obtain the frequency domain frame difference noise characteristic data.

7. The method for detecting a living body according to claim 5, wherein: The step of performing liveness detection on the face data to be detected based on the frame difference noise feature data to obtain a fourth liveness detection result comprises: Inputting the frame difference noise feature data into a preset frame difference noise feature classification model, classifying each frame difference noise feature in the frame difference noise feature data, and obtaining a frame difference noise feature classification result; Based on the frame difference noise feature classification result, liveness detection is performed on the face data to be detected to obtain the fourth liveness detection result.

8. A living body detection device, characterized in that: The liveness detection device comprises: a memory, a processor, and a program stored in the memory for implementing the liveness detection method. The memory is used to store a program for implementing a living body detection method; The processor is used to execute a program for implementing the liveness detection method, so as to implement the steps of the liveness detection method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores a program for implementing the liveness detection method, and the program for implementing the liveness detection method is executed by a processor to implement the steps of the liveness detection method according to any one of claims 1 to 7.

Images