Face spoofing attack detection, face recognition methods, apparatuses, and devices

Abstract

The embodiment of the specification discloses a face forgery attack detection method, a face recognition method, a device and equipment. The face forgery attack detection method can perform frame disassembly processing on a video containing a face to be detected to generate a video frame sequence. For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain a first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain a second feature of the video frame. The second features of the video frames in the video frame sequence are used to determine a fusion feature of the video to be detected. The fusion feature is input into a classification model to obtain a detection result of whether the video to be detected exists face forgery attack.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method, apparatus and device for detecting face spoofing attacks and for face recognition. Background Technology

[0002] In recent years, AI face-swapping has become a popular entertainment activity, and some applications (APPs) that can provide AI face-swapping services have emerged on the market, lowering the barrier to entry for AI face-swapping.

[0003] However, while AI face-swapping technology provides entertainment and convenience, it also poses risks to facial recognition-based cybersecurity systems. For example, AI face-swapping technology can be used to forge faces and attack facial recognition systems for profit. Therefore, there is an urgent need to develop a face forgery attack detection technology to accurately detect such attacks. Summary of the Invention

[0004] This specification provides a method, apparatus, and device for detecting face spoofing attacks and for face recognition, in order to accurately detect whether a face spoofing attack exists in a video.

[0005] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0006] Firstly, a method for detecting face spoofing attacks is proposed, including:

[0007] The video to be detected, which contains faces, is split into frames to generate a video frame sequence;

[0008] For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0009] The fusion features of the video to be detected are determined based on the second features of the video frames in the video frame sequence;

[0010] The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0011] Secondly, a face recognition method is proposed, including:

[0012] Obtain facial video of the user to be identified;

[0013] The method for detecting face spoofing attacks as described in the first aspect is used to detect whether the face video contains a spoofed face.

[0014] If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

[0015] Thirdly, a face spoofing attack detection device is proposed, including:

[0016] The preprocessing module splits the video to be detected, which contains faces, into frames to generate a video frame sequence;

[0017] The feature embedding module, for any video frame in the video frame sequence, inputs the video frame into a first feature embedding model to obtain the first feature of the video frame, and inputs the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0018] The feature fusion module determines the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence;

[0019] The classification module inputs the fused features into the classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0020] Fourthly, a facial recognition device is proposed, comprising:

[0021] The video acquisition module acquires facial videos of the user to be identified.

[0022] The face forgery detection module detects whether the face video contains a forged face based on the face forgery attack detection method described in the first aspect.

[0023] The determination module determines that if the face video contains a forged face, the face recognition for the user to be identified will fail.

[0024] Fifthly, an electronic device is proposed, comprising:

[0025] Processor; and

[0026] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations:

[0027] The video to be detected, which contains faces, is split into frames to generate a video frame sequence;

[0028] For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0029] The fusion features of the video to be detected are determined based on the second features of the video frames in the video frame sequence;

[0030] The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0031] In a sixth aspect, a computer-readable storage medium is provided that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations:

[0032] The video to be detected, which contains faces, is split into frames to generate a video frame sequence;

[0033] For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0034] The fusion features of the video to be detected are determined based on the second features of the video frames in the video frame sequence;

[0035] The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0036] Seventhly, an electronic device is proposed, comprising:

[0037] Processor; and

[0038] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations:

[0039] Obtain facial video of the user to be identified;

[0040] The method for detecting face spoofing attacks as described in the first aspect is used to detect whether the face video contains a spoofed face.

[0041] If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

[0042] Eighthly, a computer-readable storage medium is provided that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations:

[0043] Obtain facial video of the user to be identified;

[0044] The method for detecting face spoofing attacks as described in the first aspect is used to detect whether the face video contains a spoofed face.

[0045] If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

[0046] The at least one technical solution provided in the embodiments of this specification, after splitting the video to be detected into a video frame sequence, firstly extracts the first feature of each video frame in the video frame sequence through a first feature embedding model, then inputs the first feature into a second feature embedding model with a self-attention mechanism to extract the second feature that can reflect the texture details in the video frame, and then fuses the second features of each video frame in the video frame sequence to obtain a fused feature that is more conducive to distinguishing between real faces and face forgeries, thereby achieving an accurate detection result of whether face forgery attacks exist in the video to be detected, and thus preventing face forgery attacks. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart illustrating a face spoofing attack detection method provided in the embodiments of this specification.

[0049] Figure 2 This is one of the schematic diagrams illustrating the principle of a face spoofing attack detection method provided in the embodiments of this specification.

[0050] Figure 3 This is the second schematic diagram illustrating the principle of a face spoofing attack detection method provided in the embodiments of this specification.

[0051] Figure 4 This is a schematic diagram of the input features of the second feature embedding model provided in the embodiments of this specification.

[0052] Figure 5 This is a flowchart illustrating a face recognition method provided in an embodiment of this specification.

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification.

[0054] Figure 7 This is a schematic diagram of the structure of a face spoofing attack detection device provided in one embodiment of this specification.

[0055] Figure 8 This is a schematic diagram of the structure of a face recognition device provided in one embodiment of this specification. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this document.

[0057] To accurately detect the presence of spoofed faces in videos and thus prevent spoofing attacks, this specification provides a method and apparatus for detecting spoofing attacks. This method and apparatus can be executed by an electronic device or by software or hardware installed in an electronic device. The electronic device includes, but is not limited to, terminal devices and servers. The terminal devices include, but are not limited to, any one of the following smart terminal devices: smartphones, personal computers (PCs), laptops, tablets, e-readers, smart TVs, wearable devices, etc. The servers include, but are not limited to, any one of the following: a single server, multiple servers, a server cluster, and a cloud server.

[0058] The following section introduces a face spoofing attack detection method provided in the embodiments of this specification.

[0059] like Figure 1 As shown in the embodiments of this specification, a face spoofing attack detection method may include:

[0060] Step 102: Perform frame splitting on the video to be detected that contains faces to generate a video frame sequence.

[0061] For example, for a video to be detected containing n+1 frames, the video frame sequence obtained by splitting it into frames can be represented as: T0, T1, T2, ..., Tn, where 0 to n represent the temporal order of the video frames in the video to be detected.

[0062] Step 104: For any video frame in the video frame sequence, input the video frame into the first feature embedding model to obtain the first feature of the video frame, and input the first feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0063] In this embodiment, the first feature embedding model and the second feature embedding model are image feature extraction models. In this specification, the first feature embedding model can be considered as a backbone network, and the second feature embedding model can be considered as a feature enhancement network. Specifically, the first feature embedding model can be any of the following networks: ResNet, Inception, and Siamese Network. Since the residual network contains residual structures, and residual structures can effectively enhance the expressive power of features, the first feature embedding model can be a residual network, specifically ResNet18. The second feature embedding model can be a feature embedding model with a self-attention mechanism, such as Transformer.

[0064] The first and second feature embedding models described above are pre-trained. Optionally, when training these models, in addition to using general loss functions such as cross-entropy, a triplet loss function can also be used to determine whether the model has converged. This is because triplet loss introduces the idea of ​​contrastive learning, which can better increase the feature distance between the two categories, i.e., increase the inter-class distance between real faces and fake faces, thereby better constraining the learning process of the first and second feature embedding models.

[0065] Step 104 in the embodiments of this specification aims to first extract the basic features (first features) in the video frame using a first feature embedding model (such as ResNet18), and then further optimize the feature representation of the video frame using a second feature embedding model with a self-attention mechanism (such as Transformer) to extract a second feature that can reflect the texture details (intrafacial details and facial contour details) in the video frame, so as to better distinguish between real faces and fake faces.

[0066] Figure 2 and Figure 3 Two schematic diagrams illustrating the implementation principles of a face spoofing attack detection method provided in the embodiments of this specification are shown respectively. From Figure 2 and Figure 3 It can be seen that after splitting the video to be detected into frames to obtain a video frame sequence, each video frame in the video frame sequence can be input into the first feature embedding model to obtain the first feature corresponding to each video frame; as for the extraction of the second feature of each video frame, there are at least two methods:

[0067] The first method

[0068] like Figure 2As shown, the first feature of each video frame in the video frame sequence is directly input into the second feature embedding model to obtain the second feature of each video frame in the video frame sequence. That is, the step of inputting the first feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: inputting the first feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0069] Furthermore, in the first method described above, before inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame, the method may further include: acquiring manually extracted features of the video frame; correspondingly, inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame may include: fusing the first feature of the video frame and the manually extracted features and then inputting the fused features into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The manually extracted features corresponding to a video frame may include, but are not limited to, handcrafted features, such as histograms, HOG features, etc.

[0070] The second method

[0071] like Figure 3 As shown, before step 104, the video to be detected is first input into the video description generation module to obtain the content description text of each video frame in the video to be detected; the content description text of each video frame in the video to be detected is then input into the text feature embedding model to obtain the text features of each video frame in the video to be detected; then, the first feature and text feature of each video frame in the video frame sequence are respectively input into the second feature embedding model to obtain the second feature of each video frame in the video frame sequence. That is, inputting the first feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: inputting the first feature and the text feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0072] The video description generation module can be a video caption, and the text feature embedding model can be a Generative Pre-trained Transformer 3 (GPT3). Of course, the text feature embedding model can also be other natural language processing (NLP) models.

[0073] Furthermore, when the second feature embedding model is a Transformer, the step of inputting the first feature and the text feature of the video frame into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: inputting the first feature of the video frame into the second feature embedding model, and inputting the text feature of the video frame into the second feature embedding model in the form of a query to obtain the second feature of the video frame. It should be noted that for information on the structure of the Transformer and the meaning of a query, please refer to relevant technologies; this specification will not elaborate further.

[0074] Figure 4 A schematic diagram of the input features of the second feature embedding model provided in the embodiments of this specification is shown. (Reference) Figure 4 As can be seen, the input of the second feature embedding model includes the fusion (concat) of the first feature of the video frame and the text feature. For example, for video frame T0, its first feature can be represented as I0 and its text feature can be represented as T0_t. Then the input of the second feature embedding model corresponding to video frame T0 includes the fusion of I0 and T0_t, which can be specifically represented as I0-T0_t. For other video frames, the same principle applies, and will not be elaborated further.

[0075] As can be seen, in the second method mentioned above, in determining the second feature of the video frame, in addition to the image features (first feature) of the video frame, the text features of the video frame are also introduced, realizing the fusion of image and text features. Therefore, the detection effect of face forgery attacks can be improved from the perspective of video content understanding.

[0076] Optionally, in the second method described above, before inputting the first feature of the video frame into the second feature embedding model and inputting the text feature of the video frame into the second feature embedding model in a query manner to obtain the second feature of the video frame, the method may further include: obtaining manually extracted features of the video frame; correspondingly, inputting the first feature of the video frame into the second feature embedding model and inputting the text feature of the video frame into the second feature embedding model in a query manner to obtain the second feature of the video frame may include: fusing the first feature and the manually extracted feature of the video frame and inputting the fused feature into the second feature embedding model, and inputting the text feature of the video frame into the second feature embedding model in a query manner to obtain the second feature of the video frame. Similarly, the manually extracted feature corresponding to a video frame may include, but is not limited to, handcrafted features, such as histograms, HOG features, etc.

[0077] It is understandable that, in both the first and second methods mentioned above, the introduction of manual feature extraction can further enhance the differences in image texture, thereby further improving the detection effect of face forgery attacks.

[0078] Step 106: Determine the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence.

[0079] There are many ways to determine the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence. Three of them are listed below.

[0080] The first method

[0081] The second features of the video frames in the video frame sequence are concatenated to obtain the fused features of the video to be detected. It can be understood that since the second features of the video frames can reflect the texture details (such as details within and around the face) of the video frame, the fused features obtained by concatenating the second features of each video frame in the video frame sequence can distinguish between real and fake faces based on facial details.

[0082] The second method

[0083] In video face spoofing attack detection tasks, the defective areas in face spoofing may manifest not only in abrupt changes in details and contour details within the face (the second feature itself), but also in abrupt changes in certain sensitive areas between video frames. Therefore, if these two types of features can be extracted, face spoofing attacks can be detected more accurately.

[0084] Specifically, such as Figure 2 or Figure 3 As shown, the second features of the video frames in the video frame sequence are concatenated to obtain the basic spatial features of the video to be detected; based on the second features of two adjacent video frames in the video frame sequence, the inter-frame difference features between the two adjacent video frames are determined (which can be obtained by subtracting the second features of the two adjacent video frames), and the inter-frame difference features between the two adjacent video frames in the video frame sequence are concatenated in sequence to obtain the temporal spatial features (temporal texture enhancement information) of the video to be detected; the basic spatial features and the temporal spatial features are fused (e.g., Concat) to obtain the fused features of the video to be detected.

[0085] It is easy to see that the second method described above extracts texture details (basic spatial features) from a single frame image through the self-attention mechanism of the second feature embedding model. On the other hand, it obtains temporal texture enhancement information (temporal spatial features) of the video to be detected by determining the inter-frame differences. Finally, these two features are fused to obtain features that are more conducive to distinguishing between real face videos and fake face videos, thus improving the detection effect of face forgery attacks.

[0086] The third method

[0087] In addition to the basic spatial features that reflect texture details and the temporal spatial features that reflect temporal texture enhancement information, the text features of the video to be detected can be introduced to further achieve the fusion of image and text features, thereby further improving the detection effect of face spoofing attacks from the perspective of video content understanding.

[0088] Specifically, such as Figure 2 or Figure 3 As shown, the second features of the video frames in the video frame sequence can be concatenated first (specifically, the second features of two adjacent video frames in the video frame sequence can be concatenated in sequence) to obtain the basic spatial features of the video to be detected; then, based on the second features of two adjacent video frames in the video frame sequence, the inter-frame difference features between the two adjacent video frames can be determined (which can be obtained by subtracting the second features of the two adjacent video frames), and the inter-frame difference features between two adjacent video frames in the video frame sequence can be concatenated in sequence to obtain the temporal spatial features of the video to be detected; finally, the basic spatial features, the temporal spatial features, and the text features of the video to be detected (or the text features of each video frame in the video to be detected, i.e., the text features of the entire video to be detected) are fused to obtain the fused features of the video to be detected.

[0089] Similarly, such as Figure 3 As shown, the text features of the video to be detected can be obtained before step 108 by the following method: inputting the video to be detected into the video description generation module to obtain the content description text of the video to be detected; inputting the content description text into the text feature embedding model to obtain the text features of the video to be detected. It can be understood that here the text features of the entire video to be detected are used as global features and fused with the basic spatial features and the temporal spatial features.

[0090] It should be noted that in the embodiments of this specification, there is a fusion of image and text features twice (e.g., Figure 3(As indicated by the dashed arrow in the diagram), the first step involves using the text features of each video frame in the video to be detected as input to the second feature embedding model. The second step involves fusing the text features of the video to be detected as global features with the basic spatial features and the temporal spatial features. In specific implementation, either of these two fusions can be incorporated into the face spoofing attack detection method provided in the embodiments of this specification, or both of these fusions can be incorporated into the face spoofing attack detection method provided in the embodiments of this specification.

[0091] Step 108: Input the fused features into the classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0092] The classification model is also pre-trained and can be any classification model, such as the logistic regression model softmax. Figure 2 or Figure 3 As shown, the output of the classification model is whether the face in the video to be detected is a real face or a fake face, that is, the detection result of whether the video to be detected is a face spoofing attack.

[0093] The embodiment of this specification provides a method for detecting face spoofing attacks. After splitting the video to be detected into a video frame sequence, the method first extracts the first feature of each video frame in the video frame sequence through a first feature embedding model. Then, the first feature is input into a second feature embedding model with a self-attention mechanism to extract the second feature that can reflect the texture details in the video frame. The second features of each video frame in the video frame sequence are then fused to obtain a fused feature that is more conducive to distinguishing between real faces and face spoofing. This enables accurate detection of whether a face spoofing attack exists in the video to be detected, thereby preventing face spoofing attacks.

[0094] Optionally, the face spoofing attack detection method provided in the embodiments of this specification extracts texture detail information (basic spatial features) in a single frame image through the self-attention mechanism of the second feature embedding model. On the other hand, it obtains temporal texture enhancement information (temporal spatial features) of the video to be detected by determining the inter-frame differences. Finally, the two parts of features are fused to obtain features that are more conducive to distinguishing between real face videos and face spoofing videos, thus improving the detection effect of face spoofing attacks.

[0095] Optionally, the face spoofing attack detection method provided in the embodiments of this specification introduces text features of video frames in addition to image features (first features) of video frames, realizing the fusion of image and text features. Therefore, it can improve the detection effect of face spoofing attacks from the perspective of video content understanding.

[0096] Optionally, the face spoofing attack detection method provided in the embodiments of this specification can further enhance the difference in image texture by introducing manual feature extraction, thereby further improving the detection effect of face spoofing attacks.

[0097] Based on the aforementioned face spoofing attack detection method, this specification also proposes a face recognition method to prevent face recognition-based authentication systems from being subjected to face spoofing attacks, which will be briefly described below.

[0098] like Figure 5 As shown in the embodiments described herein, a face recognition method may include:

[0099] Step 502: Obtain the facial video of the user to be identified.

[0100] Step 504: Based on the face spoofing attack detection method provided in the embodiments of this specification, detect whether the face video contains face spoofing; if yes, proceed to step 506.

[0101] In other words, the face video is used as the video to be detected in order to confirm whether the face video contains a fake face.

[0102] Step 506: Determine that the face recognition for the user to be identified fails.

[0103] Optionally, if the detection result of step 504 is that the face video does not contain a fake face, then proceed to the next face comparison step to determine whether the identity authentication has passed.

[0104] It is understandable that if a forged face is detected in the facial video used for facial recognition, it indicates that an unauthorized user may be attempting to deceive the facial recognition system by using a forged face to pass authentication and achieve illegal purposes. However, the facial recognition method provided in the embodiments of this specification can not only accurately detect facial forgery attacks, but also provide a recognition result indicating that the facial recognition has failed after detecting the attack, thereby blocking further authentication and ultimately preventing successful facial forgery attacks, thus protecting the security of business systems using the facial recognition system.

[0105] The methods provided in this manual have been described above. The electronic equipment provided in this manual will now be introduced below.

[0106] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Please refer to it. Figure 6At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0107] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0108] Memory stores programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0109] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a face spoofing attack detection device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0110] The video to be detected, which contains faces, is split into frames to generate a video frame sequence;

[0111] For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0112] The fusion features of the video to be detected are determined based on the second features of the video frames in the video frame sequence;

[0113] The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0114] Alternatively, the processor reads the corresponding computer program from non-volatile memory into main memory and runs it, forming a facial recognition device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0115] Obtain facial video of the user to be identified;

[0116] A face spoofing attack detection method provided in the embodiments of this specification detects whether the face video contains a spoofed face;

[0117] If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

[0118] The above is as described in this instruction manual. Figure 1 or Figure 5 The methods disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in one or more embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in one or more embodiments of this specification can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0119] The electronic device can also perform Figure 1 or Figure 5 The methods provided in the illustrated embodiments will not be described again in this specification.

[0120] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] This specification also provides an embodiment of a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the embodiment is shown, and is specifically used to perform the following operations:

[0122] The video to be detected, which contains faces, is split into frames to generate a video frame sequence;

[0123] For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0124] The fusion features of the video to be detected are determined based on the second features of the video frames in the video frame sequence;

[0125] The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0126] This specification also provides an embodiment of a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 5 The method of the illustrated embodiment is specifically used to perform the following operations:

[0127] Obtain facial video of the user to be identified;

[0128] A face spoofing attack detection method provided in the embodiments of this specification detects whether the face video contains a spoofed face;

[0129] If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

[0130] The apparatus provided in the embodiments of this specification will now be described.

[0131] like Figure 7As shown, one embodiment of this specification provides a face forgery attack detection device 700. In one software implementation, the device 700 may include: a preprocessing module 701, a feature embedding module 702, a feature fusion module 703, and a classification module 704.

[0132] The preprocessing module 701 performs frame splitting on the video to be detected, which contains faces, to generate a video frame sequence.

[0133] For example, for a video to be detected containing n+1 frames, the video frame sequence obtained by splitting it into frames can be represented as: T0, T1, T2, ..., Tn, where 0 to n represent the temporal order of the video frames in the video to be detected.

[0134] The feature embedding module 702, for any video frame in the video frame sequence, inputs the video frame into a first feature embedding model to obtain the first feature of the video frame, and inputs the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0135] In this embodiment, the first feature embedding model and the second feature embedding model are image feature extraction models. In this specification, the first feature embedding model can be considered as a backbone network, and the second feature embedding model can be considered as a feature enhancement network. Specifically, the first feature embedding model can be any of the following networks: ResNet, Inception, and Siamese Network. Since the residual network contains residual structures, and residual structures can effectively enhance the expressive power of features, the first feature embedding model can be a residual network, specifically ResNet18. The second feature embedding model can be a feature embedding model with a self-attention mechanism, such as Transformer.

[0136] The first and second feature embedding models described above are pre-trained. Optionally, when training these models, in addition to using general loss functions such as cross-entropy, a triplet loss function can also be used to determine whether the model has converged. This is because triplet loss introduces the idea of ​​contrastive learning, which can better increase the feature distance between the two categories, i.e., increase the inter-class distance between real faces and fake faces, thereby better constraining the learning process of the first and second feature embedding models.

[0137] The feature embedding module 702 in the embodiments of this specification aims to first extract the basic features (first features) in the video frame using a first feature embedding model (such as ResNet18), and then further optimize the feature representation of the video frame using a second feature embedding model with a self-attention mechanism (such as Transformer) to extract a second feature that can reflect the texture details (intrafacial details and facial contour details) in the video frame, so as to better distinguish between real faces and fake faces.

[0138] Specifically, the feature embedding module 702 can input each video frame in the video frame sequence into the first feature embedding model to obtain the first feature corresponding to each video frame; as for the extraction of the second feature of each video frame, there are at least two methods:

[0139] The first method

[0140] The feature embedding module 702 can input the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

[0141] Furthermore, in the first method described above, the device 700 may further include: a manual feature acquisition module, which acquires manually extracted features of the video frame before inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame; correspondingly, the feature embedding module 702 can fuse the first feature and the manually extracted features of the video frame and input them into the second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The manually extracted features corresponding to a video frame may include, but are not limited to, handcrafted features, such as histograms, HOG features, etc.

[0142] The second method

[0143] The device 700 may also include a text generation module and a text feature extraction module.

[0144] The text generation module inputs the video to be detected into the video description generation module to obtain the content description text of each video frame in the video to be detected.

[0145] The text feature extraction module inputs the content description text of each video frame in the video to be detected into the text feature embedding model to obtain the text features of each video frame in the video to be detected.

[0146] Correspondingly, the feature embedding module 702 can input the first feature and text feature of each video frame in the video frame sequence into the second feature embedding model to obtain the second feature of each video frame in the video frame sequence.

[0147] The video description generation module can be a video caption, and the text feature embedding model can be a Generative Pre-trained Transformer 3 (GPT3). Of course, the text feature embedding model can also be other natural language processing (NLP) models.

[0148] Furthermore, when the second feature embedding model is a Transformer, the feature embedding module 702 can input the first feature of the video frame into the second feature embedding model, and input the text feature of the video frame as a query into the second feature embedding model to obtain the second feature of the video frame. It should be noted that for information on the structure of the Transformer and the meaning of a query, please refer to relevant technologies; this specification will not elaborate further.

[0149] As can be seen, in the second method mentioned above, in determining the second feature of the video frame, in addition to the image features (first feature) of the video frame, the text features of the video frame are also introduced, realizing the fusion of image and text features. Therefore, the detection effect of face forgery attacks can be improved from the perspective of video content understanding.

[0150] Optionally, in the second method described above, the device 700 may further include: a manual feature acquisition module to acquire manually extracted features of the video frame; correspondingly, the feature embedding module 702 can fuse the first feature and the manually extracted features of the video frame and input them into the second feature embedding model, and input the text features of the video frame into the second feature embedding model in the form of a query to obtain the second feature of the video frame. Similarly, the manually extracted features corresponding to a video frame may include, but are not limited to, handcrafted features, such as histograms, HOG features, etc.

[0151] It is understandable that, in both the first and second methods mentioned above, the introduction of manual feature extraction can further enhance the differences in image texture, thereby further improving the detection effect of face forgery attacks.

[0152] The feature fusion module 703 determines the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence.

[0153] Specifically, the feature fusion module 703 can determine the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence in many ways. Three of them are listed below.

[0154] The first method

[0155] The second features of the video frames in the video frame sequence are concatenated to obtain the fused features of the video to be detected. It can be understood that since the second features of the video frames can reflect the texture details (such as details within and around the face) of the video frame, the fused features obtained by concatenating the second features of each video frame in the video frame sequence can distinguish between real and fake faces based on facial details.

[0156] The second method

[0157] In video face spoofing attack detection tasks, the defective areas in face spoofing may manifest not only in abrupt changes in details and contour details within the face (the second feature itself), but also in abrupt changes in certain sensitive areas between video frames. Therefore, if these two types of features can be extracted, face spoofing attacks can be detected more accurately.

[0158] Specifically, the second features of the video frames in the video frame sequence are concatenated to obtain the basic spatial features of the video to be detected; based on the second features of two adjacent video frames in the video frame sequence, the inter-frame difference features between the two adjacent video frames are determined (which can be obtained by subtracting the second features of the two adjacent video frames), and the inter-frame difference features between the two adjacent video frames in the video frame sequence are concatenated in sequence to obtain the temporal spatial features (temporal texture enhancement information) of the video to be detected; the basic spatial features and the temporal spatial features are fused (e.g., Concat) to obtain the fused features of the video to be detected.

[0159] It is easy to see that the second method described above extracts texture details (basic spatial features) from a single frame image through the self-attention mechanism of the second feature embedding model. On the other hand, it obtains temporal texture enhancement information (temporal spatial features) of the video to be detected by determining the inter-frame differences. Finally, these two features are fused to obtain features that are more conducive to distinguishing between real face videos and fake face videos, thus improving the detection effect of face forgery attacks.

[0160] The third method

[0161] In addition to the basic spatial features that reflect texture details and the temporal spatial features that reflect temporal texture enhancement information, the text features of the video to be detected can be introduced to further achieve the fusion of image and text features, thereby further improving the detection effect of face spoofing attacks from the perspective of video content understanding.

[0162] Specifically, the second features of the video frames in the video frame sequence can be concatenated first (specifically, the second features of two adjacent video frames in the video frame sequence can be concatenated in sequence) to obtain the basic spatial features of the video to be detected; then, based on the second features of two adjacent video frames in the video frame sequence, the inter-frame difference features between the two adjacent video frames can be determined (which can be obtained by subtracting the second features of the two adjacent video frames), and the inter-frame difference features between two adjacent video frames in the video frame sequence can be concatenated in sequence to obtain the temporal spatial features of the video to be detected; finally, the basic spatial features, the temporal spatial features, and the text features of the video to be detected (or the text features of each video frame in the video to be detected, i.e., the text features of the entire video to be detected) are fused to obtain the fused features of the video to be detected.

[0163] Similarly, the device 700 may also include a text generation module and a text feature extraction module.

[0164] The text generation module inputs the video to be detected into the video description generation module to obtain the content description text of the video to be detected.

[0165] The text feature extraction module inputs the content description text into the text feature embedding model to obtain the text features of the video to be detected.

[0166] It is understandable that here the text features of the entire video to be detected are used as global features and fused with the basic spatial features and the temporal spatial features.

[0167] It should be noted that in the embodiments of this specification, there is a fusion of image and text features twice (e.g., Figure 3 (As indicated by the dashed arrow in the diagram), the first step involves using the text features of each video frame in the video to be detected as input to the second feature embedding model. The second step involves fusing the text features of the video to be detected as global features with the basic spatial features and the temporal spatial features. In specific implementation, either of these two fusions can be incorporated into the face spoofing attack detection method provided in the embodiments of this specification, or both of these fusions can be incorporated into the face spoofing attack detection method provided in the embodiments of this specification.

[0168] The classification module 704 inputs the fused features into the classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

[0169] The classification model is also pre-trained and can be any classification model, such as the logistic regression model softmax.

[0170] It should be noted that the 700 face spoofing attack detection device is capable of achieving... Figure 1 A method for detecting face spoofing attacks is provided, which can achieve the same technical effect. For details, please refer to the description of the method implementation section above, which will not be repeated here.

[0171] like Figure 8 As shown, one embodiment of this specification provides a face recognition device 800. In one software implementation, the device 800 may include: a video acquisition module 801, a face forgery detection module 802, and a determination module 803.

[0172] The video acquisition module 801 acquires the facial video of the user to be identified.

[0173] The face forgery detection module 802 detects whether the face video contains a forged face, based on a face forgery attack detection method provided in the embodiments of this specification.

[0174] The determination module 803 determines that if the face video contains a forged face, the face recognition for the user to be identified fails.

[0175] It should be noted that the 800 face spoofing attack detection device is capable of achieving... Figure 5 A face recognition method is provided, which can achieve the same technical effect. For details, please refer to the description of the method implementation section above, which will not be repeated here.

[0176] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0177] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0178] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

[0179] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0180] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0181] It should be noted that the terms "first," "second," etc., used in this specification and claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited; for example, the first character can be one or more.

[0182] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for detecting face spoofing attacks, comprising: The video to be detected, which contains faces, is split into frames to generate a video frame sequence; For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The video to be detected is input into the video description generation module to obtain the content description text of the video to be detected. The content description text is input into the text feature embedding model to obtain the text features of the video to be detected; Based on the second features of the video frames in the video frame sequence and the text features of the video to be detected, the fusion features of the video to be detected are determined. The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

2. The method according to claim 1, before inputting the fused features into the classification model to obtain the detection result of whether the video to be detected contains a face spoofing attack, the method further includes: The video to be detected is input into the video description generation module to obtain the content description text of each video frame in the video to be detected. The content description text of each video frame in the video to be detected is input into the text feature embedding model to obtain the text features of each video frame in the video to be detected. The step of inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: The first feature and the text feature of the video frame are input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

3. The method according to claim 2, wherein when the second feature embedding model is a Transformer, the step of inputting the first feature of the video frame and the text feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: The first feature of the video frame is input into the second feature embedding model, and the text feature of the video frame is input into the second feature embedding model in the form of a query to obtain the second feature of the video frame.

4. The method according to claim 1, wherein determining the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence and the text features of the video to be detected comprises: The second features of the video frames in the video frame sequence are concatenated to obtain the basic spatial features of the video to be detected. Based on the second feature of two adjacent video frames in the video frame sequence, the inter-frame difference feature between the two adjacent video frames is determined, and the inter-frame difference feature between two adjacent video frames in the video frame sequence is sequentially concatenated to obtain the temporal spatial feature of the video to be detected. The basic spatial features, the temporal spatial features, and the text features of the video to be detected are fused to obtain the fused features of the video to be detected.

5. The method according to claim 1, before inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame, the method further includes: Obtain manually extracted features from this video frame; The step of inputting the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame includes: The first feature of the video frame is fused with the manually extracted features and then input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame.

6. The method according to any one of claims 1-5, The loss functions used when training the first feature embedding model and the second feature embedding model include: Triple loss function.

7. The method according to any one of claims 1-5, wherein the first feature embedding model comprises one of the following: Residual networks; Inception; Twin networks.

8. The method according to any one of claims 1-2 and 4-5, The second feature embedding model is Transformer.

9. A face recognition method, comprising: Obtain facial video of the user to be identified; The method for detecting face spoofing attacks as described in any one of claims 1-8 is used to detect whether the face video contains a spoofed face. If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

10. A face spoofing attack detection device, comprising: The preprocessing module splits the video to be detected, which contains faces, into frames to generate a video frame sequence; The feature embedding module, for any video frame in the video frame sequence, inputs the video frame into a first feature embedding model to obtain the first feature of the video frame, and inputs the first feature of the video frame into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The text generation module inputs the video to be detected into the video description generation module to obtain the content description text of the video to be detected. The text feature extraction module inputs the content description text into the text feature embedding model to obtain the text features of the video to be detected; The feature fusion module determines the fusion features of the video to be detected based on the second features of the video frames in the video frame sequence and the text features of the video to be detected. The classification module inputs the fused features into the classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

11. A facial recognition device, comprising: The video acquisition module acquires facial videos of the user to be identified. A face forgery detection module detects whether the face video contains a forged face based on the face forgery attack detection method as described in any one of claims 1-8; The determination module determines that if the face video contains a forged face, the face recognition for the user to be identified will fail.

12. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: The video to be detected, which contains faces, is split into frames to generate a video frame sequence; For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The video to be detected is input into the video description generation module to obtain the content description text of the video to be detected. The content description text is input into the text feature embedding model to obtain the text features of the video to be detected; Based on the second features of the video frames in the video frame sequence and the text features of the video to be detected, the fusion features of the video to be detected are determined. The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

13. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: The video to be detected, which contains faces, is split into frames to generate a video frame sequence; For any video frame in the video frame sequence, the video frame is input into a first feature embedding model to obtain the first feature of the video frame, and the first feature of the video frame is input into a second feature embedding model with a self-attention mechanism to obtain the second feature of the video frame. The video to be detected is input into the video description generation module to obtain the content description text of the video to be detected. The content description text is input into the text feature embedding model to obtain the text features of the video to be detected; Based on the second features of the video frames in the video frame sequence and the text features of the video to be detected, the fusion features of the video to be detected are determined. The fused features are input into a classification model to obtain the detection result of whether the video to be detected is subjected to a face spoofing attack.

14. An electronic device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: Obtain facial video of the user to be identified; The method for detecting face spoofing attacks as described in any one of claims 1-8 is used to detect whether the face video contains a spoofed face. If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

15. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: Obtain facial video of the user to be identified; The method for detecting face spoofing attacks as described in any one of claims 1-8 is used to detect whether the face video contains a spoofed face. If included, then the facial recognition for the user to be identified is determined to be unsuccessful.

Images